Not overwriting unchanged source code files when tangling

Not overwriting unchanged source code files when tangling

[Holger Hoefling]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

Hi,

I have a problem/request for org-mode and was looking for help. I am using
org-mode to write source code files and tangle them out. I want to compile
them using make. My problem now is that org-mode overwrites the old files
every time I tangle them out, therefore also updating the time stamp - even
if nothing has changed. Subsequently, when I run make, everything gets
recompiled, not just the changed source code files as all time stamps have
changed.

Is there an option for org-mode to only overwrite source code files that
get tangled out if they have truly changed?

Thank you very much for your help!

Holger

[-- Attachment #2: Type: text/html, Size: 653 bytes --]

Re: Not overwriting unchanged source code files when tangling

[Brian Wightman]

On Fri, Nov 18, 2011 at 7:17 AM, Holger Hoefling <hhoeflin@gmail.com> wrote:
> I have a problem/request for org-mode and was looking for help. I am using
> org-mode to write source code files and tangle them out. I want to compile
> them using make. My problem now is that org-mode overwrites the old files
> every time I tangle them out, therefore also updating the time stamp - even
> if nothing has changed. Subsequently, when I run make, everything gets
> recompiled, not just the changed source code files as all time stamps have
> changed.
>
> Is there an option for org-mode to only overwrite source code files that get
> tangled out if they have truly changed?

I believe that to do this, you would need to have a dependency tree of
the nodes contributing to the output (perhaps already exists), and
recursively mark any node that refers to a node that changed as dirty.
 You would also have to store last update times on each node so that
they could be compared to each output file, contributing to the
determination of needing a regeneration or not.

From a make standpoint, if you were to have each node in a file (I am
not recommending this), make already has the smarts to handle this.
It just becomes unwieldy to manage from an editing perspective.

Perhaps a way to deal with this would be to tangle to a different
directory, and then sync any changes into your compilation source
directory.  If you would update the compilation directory only when
something differs from the tangle directory, then make could handle it
from that point on.

Brian

Re: Not overwriting unchanged source code files when tangling

[Tom Prince]

On Fri, 18 Nov 2011 08:23:18 -0600, Brian Wightman <MidLifeXis@wightmanfam.org> wrote:
> Perhaps a way to deal with this would be to tangle to a different
> directory, and then sync any changes into your compilation source
> directory.  If you would update the compilation directory only when
> something differs from the tangle directory, then make could handle it
> from that point on.

The tangle mechanism could probably handle this autoatically. i.e. not
saving a file if the contents are identical.

  Tom

Re: Not overwriting unchanged source code files when tangling

[Brian Wightman]

On Fri, Nov 18, 2011 at 10:46 AM, Tom Prince <tom.prince@ualberta.net> wrote:
> On Fri, 18 Nov 2011 08:23:18 -0600, Brian Wightman <MidLifeXis@wightmanfam.org> wrote:
>> Perhaps a way to deal with this would be to tangle to a different
>> directory, and then sync any changes into your compilation source
>> directory.  If you would update the compilation directory only when
>> something differs from the tangle directory, then make could handle it
>> from that point on.
>
> The tangle mechanism could probably handle this autoatically. i.e. not
> saving a file if the contents are identical.

If there is not a lot of extra memory / time overhead associated with
this, I could see this being a valid approach.  I would request that,
if implemented, this be placed behind an on/off switch.

The makefile could also handle this with something along these lines
(correct the leading space -> tab conversion as well as proper macro
definitions):

tangleflag: totangle.org
   $(TANGLECOMMAND) totangle.org
   $(TOUCH) tangleflag

syncflag: tangleflag
   $(SYNCCOMMAND) sourcedir tangledir
   $(TOUCH) syncflag

--Brian

Re: Not overwriting unchanged source code files when tangling

[Sebastien Vauban]

Hi Holger,

Holger Hoefling wrote:
> I have a problem/request for org-mode and was looking for help. I am using
> org-mode to write source code files and tangle them out. I want to compile
> them using make. My problem now is that org-mode overwrites the old files
> every time I tangle them out, therefore also updating the time stamp - even
> if nothing has changed. Subsequently, when I run make, everything gets
> recompiled, not just the changed source code files as all time stamps have
> changed.
>
> Is there an option for org-mode to only overwrite source code files that
> get tangled out if they have truly changed?
>
> Thank you very much for your help!

I have absolutely no idea on how hard it could be, but this makes me think at
some "cache" mechanism, like what currently exists for skipping evaluation of
unchanged code blocks.

The key could be to use the cache feature for tangling as well?

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Not overwriting unchanged source code files when tangling

[Carsten Dominik]

On 18.11.2011, at 14:17, Holger Hoefling wrote:

> Hi,
> 
> I have a problem/request for org-mode and was looking for help. I am using org-mode to write source code files and tangle them out. I want to compile them using make. My problem now is that org-mode overwrites the old files every time I tangle them out, therefore also updating the time stamp - even if nothing has changed. Subsequently, when I run make, everything gets recompiled, not just the changed source code files as all time stamps have changed.
> 
> Is there an option for org-mode to only overwrite source code files that get tangled out if they have truly changed?

How about changing the make file so that the dependence is on the Org file, not on the source file?
You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it?

Something along the lines of.... (untested, and probably wrong in this way...)

file.o: somefile.org
   emacs -batch --eval '(org-babel-tangle-file "somefile.org")'
   cc file.o ....

- Carsten

Re: Not overwriting unchanged source code files when tangling

[Brian Wightman]

On Fri, Nov 18, 2011 at 11:02 AM, Carsten Dominik
<carsten.dominik@gmail.com> wrote:
> How about changing the make file so that the dependence is on the Org file, not on the source file?
> You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it?

The original question was trying to avoid recompiling everything
generated from a tangle if the content didn't actually change.
Because retangling the source rewrites /all/ of the files, and resets
the dates, even if nothing has changed, make will then rebuild
everything that was tangled, not just the partial set of tangled files
that actually changed.

Brian

Re: Not overwriting unchanged source code files when tangling

[Eric Schulte]

Brian Wightman <brian@wightmanfam.org> writes:

> On Fri, Nov 18, 2011 at 11:02 AM, Carsten Dominik
> <carsten.dominik@gmail.com> wrote:
>> How about changing the make file so that the dependence is on the Org file, not on the source file?
>> You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it?
>
> The original question was trying to avoid recompiling everything
> generated from a tangle if the content didn't actually change.
> Because retangling the source rewrites /all/ of the files, and resets
> the dates, even if nothing has changed, make will then rebuild
> everything that was tangled, not just the partial set of tangled files
> that actually changed.
>

I think the best approach in this case would be to tangle each file out
to a temporary buffer, and then just before exiting the tangle function
the content of these temporary buffers could be checked against the
files on disk, and only those buffers which differ from disk would be
written.  See ob-tangle.el around line 240 for the relevant code.
Unfortunately this would not be trivial, as currently content is written
to the target files incrementally block by block.

If the code in the .org file is grouped by subtree it may be possible to
place calls to org-narrow-to-subtree in the Makefile before tangling, so
that only part of the file is tangled.

Finally, it may be easiest simply to play make's game as it were and
break up the Org-mode file into multiple files.  These multiple files
could still be combined during export using #+INCLUDE lines from a
single master Org-mode file.

Best -- Eric

>
> Brian
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Not overwriting unchanged source code files when tangling

[Achim Gratz]

Eric Schulte <schulte.eric@gmail.com> writes:
> I think the best approach in this case would be to tangle each file out
> to a temporary buffer, and then just before exiting the tangle function
> the content of these temporary buffers could be checked against the
> files on disk, and only those buffers which differ from disk would be
> written.  See ob-tangle.el around line 240 for the relevant code.
> Unfortunately this would not be trivial, as currently content is written
> to the target files incrementally block by block.

It would be wise to follow an age-old tradition and not clobber an
existing file with before it is known that the process will finish
without error.  A temporary file is easily trashed when something goes
wrong and the previous result still available.  If all went well, the
old file can be deleted (or renamed to a backup file) and the temporary
file can be moved to where the old one was if the two SHA1 differ.

> If the code in the .org file is grouped by subtree it may be possible to
> place calls to org-narrow-to-subtree in the Makefile before tangling, so
> that only part of the file is tangled.

I don't think that would work very well, see my other post.  It is the
content of the resulting file that needs to be compared, not the steps
in which that result is produced.

> Finally, it may be easiest simply to play make's game as it were and
> break up the Org-mode file into multiple files.  These multiple files
> could still be combined during export using #+INCLUDE lines from a
> single master Org-mode file.

Well, I've been wondering about this for some time: can one make sort of
an "indirect buffer" from the contents of multiple other buffers in
Emacs?  That would make such #+INCLUDEs much more seamless.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Not overwriting unchanged source code files when tangling

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>> I think the best approach in this case would be to tangle each file out
>> to a temporary buffer, and then just before exiting the tangle function
>> the content of these temporary buffers could be checked against the
>> files on disk, and only those buffers which differ from disk would be
>> written.  See ob-tangle.el around line 240 for the relevant code.
>> Unfortunately this would not be trivial, as currently content is written
>> to the target files incrementally block by block.
>
> It would be wise to follow an age-old tradition and not clobber an
> existing file with before it is known that the process will finish
> without error.  A temporary file is easily trashed when something goes
> wrong and the previous result still available.  If all went well, the
> old file can be deleted (or renamed to a backup file) and the temporary
> file can be moved to where the old one was if the two SHA1 differ.
>

Agreed, it would be preferable to build up the tangled contents in
memory and not write to the file system until the tangling process is
complete.

>> Finally, it may be easiest simply to play make's game as it were and
>> break up the Org-mode file into multiple files.  These multiple files
>> could still be combined during export using #+INCLUDE lines from a
>> single master Org-mode file.
>
> Well, I've been wondering about this for some time: can one make sort of
> an "indirect buffer" from the contents of multiple other buffers in
> Emacs?  That would make such #+INCLUDEs much more seamless.
>

Hmm, this could be useful both inside and outside of Org-mode.  Emacs
does have indirect buffers [1], however they are exact copies of another
buffer, I don't know if it would be possible to combine multiple
indirect buffers into a single buffer.

Thanks -- Eric

>
>
> Regards,
> Achim.


Footnotes: 
[1]  (info "(elisp) Indirect Buffers")

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Not overwriting unchanged source code files when tangling

[Holger Hoefling]

[-- Attachment #1: Type: text/plain, Size: 3499 bytes --]

Hi Eric,

sounds like the problem may after all not be that simple.Could the code
blocks be written incrementally to the buffer (or a temporary file on disk)
and only after everything has been tangled out all temporary buffers or
files checked against the ones on disk?

Unfortunately, I do not think that breaking up the org-file into smaller
org-files is a solution to my problem. I am running longer analyses (20
script files or more) each doing various things and files in parts
depending on each other. If I break up the org-file into so many small
parts, a large part of the advantage of using org (i.e. complete analysis
in a single file with extensive documentation) will be lost. True, I can
still include the subfiles during export - but i hardly ever export code
blocks at all (I instead hide the code from export and produce a beamer
presentation or latex article using the output from my earlier code). So
overall, I would really like to keep everything together in one file.

Someone else earlier suggested just tangling out to a temporary directory
myself and synching in the makefile. This suggestion is close to what I
would like to do and I may use it. However, it is still not quite
satisfactory as this brings other problems. Source files sometimes call
each other -> when they are in a different directory after tangling than
they are during the make file execution (due to synching), then either
execution from within org may break, or the execution from the make file.

If I knew more about e-lisp, I would just hack up a solution myself. I
think org-mode is a really awesome tool.

Thanks for your help and suggestions

Holger

On Fri, Nov 18, 2011 at 8:42 PM, Eric Schulte <schulte.eric@gmail.com>wrote:

> Brian Wightman <brian@wightmanfam.org> writes:
>
> > On Fri, Nov 18, 2011 at 11:02 AM, Carsten Dominik
> > <carsten.dominik@gmail.com> wrote:
> >> How about changing the make file so that the dependence is on the Org
> file, not on the source file?
> >> You could then arrange for make to call emacs in batch-mode to tangle
> the source file and then compile it?
> >
> > The original question was trying to avoid recompiling everything
> > generated from a tangle if the content didn't actually change.
> > Because retangling the source rewrites /all/ of the files, and resets
> > the dates, even if nothing has changed, make will then rebuild
> > everything that was tangled, not just the partial set of tangled files
> > that actually changed.
> >
>
> I think the best approach in this case would be to tangle each file out
> to a temporary buffer, and then just before exiting the tangle function
> the content of these temporary buffers could be checked against the
> files on disk, and only those buffers which differ from disk would be
> written.  See ob-tangle.el around line 240 for the relevant code.
> Unfortunately this would not be trivial, as currently content is written
> to the target files incrementally block by block.
>
> If the code in the .org file is grouped by subtree it may be possible to
> place calls to org-narrow-to-subtree in the Makefile before tangling, so
> that only part of the file is tangled.
>
> Finally, it may be easiest simply to play make's game as it were and
> break up the Org-mode file into multiple files.  These multiple files
> could still be combined during export using #+INCLUDE lines from a
> single master Org-mode file.
>
> Best -- Eric
>
> >
> > Brian
> >
>
> --
> Eric Schulte
> http://cs.unm.edu/~eschulte/
>

[-- Attachment #2: Type: text/html, Size: 4203 bytes --]

Re: Not overwriting unchanged source code files when tangling

[Holger Hoefling]

[-- Attachment #1: Type: text/plain, Size: 2505 bytes --]

Hi Carsten,

thanks for the suggestion, but as I agree with Brian. If there is more than
one source file in the org-file, then the whole project would still be
recompiled, not just the updated file.

To be more exact, I actually don't want to compile things, but run R
scripts using make. So the waiting time if a computationally intensive step
is repeated although it is not necessary can be substantial.

I wonder how difficult the following change would be (no emacs lisp
experience, also do not know the org source code):

- would it be possible to write out the source files when tangling into a
temporary directory, then compare to the actual target files and overwrite
only if something has changed? Then the time stamps would stay fixed.
Hopefully, this would not involve too much work:
- creating temporary files and remembering the mapping to true files
- tangling out as usual into temporary files (so probably little change
there)
- compare temporary file to true file (does emacs already have a diff
utility that could be used?)
- overwrite true file if any changes
- delete temporary files

Especially, with this method no dependencies would be necessary and it
would not be necessary to keep track in the org files which source blocks
have been changed since the last tangling

Thank you for your suggestion

Holger

On Fri, Nov 18, 2011 at 6:02 PM, Carsten Dominik
<carsten.dominik@gmail.com>wrote:

>
> On 18.11.2011, at 14:17, Holger Hoefling wrote:
>
> > Hi,
> >
> > I have a problem/request for org-mode and was looking for help. I am
> using org-mode to write source code files and tangle them out. I want to
> compile them using make. My problem now is that org-mode overwrites the old
> files every time I tangle them out, therefore also updating the time stamp
> - even if nothing has changed. Subsequently, when I run make, everything
> gets recompiled, not just the changed source code files as all time stamps
> have changed.
> >
> > Is there an option for org-mode to only overwrite source code files that
> get tangled out if they have truly changed?
>
> How about changing the make file so that the dependence is on the Org
> file, not on the source file?
> You could then arrange for make to call emacs in batch-mode to tangle the
> source file and then compile it?
>
> Something along the lines of.... (untested, and probably wrong in this
> way...)
>
> file.o: somefile.org
>   emacs -batch --eval '(org-babel-tangle-file "somefile.org")'
>   cc file.o ....
>
> - Carsten

[-- Attachment #2: Type: text/html, Size: 3062 bytes --]

Re: Not overwriting unchanged source code files when tangling

[Nick Dokos]

Holger Hoefling <hhoeflin@gmail.com> wrote:

> Hi Carsten,
> 
> thanks for the suggestion, but as I agree with Brian. If there is more
> than one source file in the org-file, then the whole project would
> still be recompiled, not just the updated file.
> 
> To be more exact, I actually don't want to compile things, but run R
> scripts using make. So the waiting time if a computationally intensive
> step is repeated although it is not necessary can be substantial.
> 
> I wonder how difficult the following change would be (no emacs lisp experience, also do not know the org source code):
> 
> - would it be possible to write out the source files when tangling
> - into a temporary directory, then compare to the actual target files
> - and overwrite only if something has changed? Then the time stamps
> - would stay fixed. Hopefully, this would not involve too much work:

You've lost right there unless there is a method to select *which* source
blocks to tangle. IOW, the problem is not the *comparison* of the temp and actual
target files, it is the *production* of the temp files themselves: that's
the computationally expensive step and this method does nothing to alleviate
that. Unless I'm missing something.

Nick

> - creating temporary files and remembering the mapping to true files

> - tangling out as usual into temporary files (so probably little
> - change there)

> - compare temporary file to true file (does emacs already have a diff
> - utility that could be used?)

> - overwrite true file if any changes

> - delete temporary files

Re: Not overwriting unchanged source code files when tangling

[Holger Hoefling]

[-- Attachment #1: Type: text/plain, Size: 1975 bytes --]

Hi Nick,

I think you misunderstood me there - I am actually not worried about how
computationally intensive the tangling process is. This always works very
quickly, so even if they have to be copied around and take a bit longer, I
would not mind.

Thanks

Holger

On Fri, Nov 18, 2011 at 8:32 PM, Nick Dokos <nicholas.dokos@hp.com> wrote:

> Holger Hoefling <hhoeflin@gmail.com> wrote:
>
> > Hi Carsten,
> >
> > thanks for the suggestion, but as I agree with Brian. If there is more
> > than one source file in the org-file, then the whole project would
> > still be recompiled, not just the updated file.
> >
> > To be more exact, I actually don't want to compile things, but run R
> > scripts using make. So the waiting time if a computationally intensive
> > step is repeated although it is not necessary can be substantial.
> >
> > I wonder how difficult the following change would be (no emacs lisp
> experience, also do not know the org source code):
> >
> > - would it be possible to write out the source files when tangling
> > - into a temporary directory, then compare to the actual target files
> > - and overwrite only if something has changed? Then the time stamps
> > - would stay fixed. Hopefully, this would not involve too much work:
>
> You've lost right there unless there is a method to select *which* source
> blocks to tangle. IOW, the problem is not the *comparison* of the temp and
> actual
> target files, it is the *production* of the temp files themselves: that's
> the computationally expensive step and this method does nothing to
> alleviate
> that. Unless I'm missing something.
>
> Nick
>
> > - creating temporary files and remembering the mapping to true files
>
> > - tangling out as usual into temporary files (so probably little
> > - change there)
>
> > - compare temporary file to true file (does emacs already have a diff
> > - utility that could be used?)
>
> > - overwrite true file if any changes
>
> > - delete temporary files
>
>

[-- Attachment #2: Type: text/html, Size: 2574 bytes --]

Re: Not overwriting unchanged source code files when tangling

[Nick Dokos]

Holger Hoefling <hhoeflin@gmail.com> wrote:

> I think you misunderstood me there - I am actually not worried about how
> computationally intensive the tangling process is. This always works very
> quickly, so even if they have to be copied around and take a bit longer, I
> would not mind.
> 

Ah, ok - so you are talking about

       tangle                    compile
   org -------> bunch of files -------------> output

The tangling step produces a bunch of files that are (re)compiled (or in
any case require some sort of lengthy processing) to produce some output file.


IMO, the best way to deal with it is still make: let's say

     foo.org  ----> a.x b.x c.x ------> foo.out

where the first arrow is the tangle and the second arrow is some processor, call it X.
The standard way to set up a makefile is schematically:

--8<---------------cut here---------------start------------->8---
foo.out: a.x b.x c.x
	 X a.x b.c c.x -o foo.out

a.x b.x c.x: foo.org
         tangle foo.org

--8<---------------cut here---------------end--------------->8---


Rewrite the make file as follows:


--8<---------------cut here---------------start------------->8---
foo.out: a.y b.y c.y
     X a.y b.y c.y -o foo.out

a.y: a.x
     cmp --silent a.x a.y || cp a.x a.y

b.y: b.x
     cmp --silent b.x b.y || cp b.x b.y

c.y: c.x
     cmp --silent c.x c.y || cp c.x c.y

a.x b.x c.x: foo.org
     tangle foo.org
--8<---------------cut here---------------end--------------->8---


So if the *contents* of (say) a.x have not changed by the tangling, it compares
equal to a.y and the copy is skipped. That leaves a.y untouched.

OTOH, if the contents of a.x change (or a.y does not exist in the first
place), the comparison fails and we copy a.x to a.y.  That updates a.y
and forces further updates on anything that depends on it.

Using some make fu (works for GNU make, but not necessarily for other makes),
you can write it more compactly:


--8<---------------cut here---------------start------------->8---
foo.out: a.y b.y c.y
     X a.y b.y c.y -o foo.out

%.y: %.x
     -cmp --silent $< $@ || cp $< $@

a.x b.x c.x: foo.org
     tangle foo.org
--8<---------------cut here---------------end--------------->8---

HTH,
Nick

Re: Not overwriting unchanged source code files when tangling

[Holger Hoefling]

[-- Attachment #1: Type: text/plain, Size: 2576 bytes --]

Hey Nick,

thank you very much. That sounds like a very good solution to my problem
that does not require changes to org-mode.

Best

Holger

On Sat, Nov 19, 2011 at 5:00 AM, Nick Dokos <nicholas.dokos@hp.com> wrote:

> Holger Hoefling <hhoeflin@gmail.com> wrote:
>
> > I think you misunderstood me there - I am actually not worried about how
> > computationally intensive the tangling process is. This always works very
> > quickly, so even if they have to be copied around and take a bit longer,
> I
> > would not mind.
> >
>
> Ah, ok - so you are talking about
>
>       tangle                    compile
>   org -------> bunch of files -------------> output
>
> The tangling step produces a bunch of files that are (re)compiled (or in
> any case require some sort of lengthy processing) to produce some output
> file.
>
>
> IMO, the best way to deal with it is still make: let's say
>
>     foo.org  ----> a.x b.x c.x ------> foo.out
>
> where the first arrow is the tangle and the second arrow is some
> processor, call it X.
> The standard way to set up a makefile is schematically:
>
> --8<---------------cut here---------------start------------->8---
> foo.out: a.x b.x c.x
>         X a.x b.c c.x -o foo.out
>
> a.x b.x c.x: foo.org
>         tangle foo.org
>
> --8<---------------cut here---------------end--------------->8---
>
>
> Rewrite the make file as follows:
>
>
> --8<---------------cut here---------------start------------->8---
> foo.out: a.y b.y c.y
>     X a.y b.y c.y -o foo.out
>
> a.y: a.x
>     cmp --silent a.x a.y || cp a.x a.y
>
> b.y: b.x
>     cmp --silent b.x b.y || cp b.x b.y
>
> c.y: c.x
>     cmp --silent c.x c.y || cp c.x c.y
>
> a.x b.x c.x: foo.org
>     tangle foo.org
> --8<---------------cut here---------------end--------------->8---
>
>
> So if the *contents* of (say) a.x have not changed by the tangling, it
> compares
> equal to a.y and the copy is skipped. That leaves a.y untouched.
>
> OTOH, if the contents of a.x change (or a.y does not exist in the first
> place), the comparison fails and we copy a.x to a.y.  That updates a.y
> and forces further updates on anything that depends on it.
>
> Using some make fu (works for GNU make, but not necessarily for other
> makes),
> you can write it more compactly:
>
>
> --8<---------------cut here---------------start------------->8---
> foo.out: a.y b.y c.y
>     X a.y b.y c.y -o foo.out
>
> %.y: %.x
>     -cmp --silent $< $@ || cp $< $@
>
> a.x b.x c.x: foo.org
>     tangle foo.org
> --8<---------------cut here---------------end--------------->8---
>
> HTH,
> Nick
>
>
>

[-- Attachment #2: Type: text/html, Size: 3589 bytes --]

Re: Not overwriting unchanged source code files when tangling

[Holger Hoefling]

[-- Attachment #1: Type: text/plain, Size: 3279 bytes --]

Hi everyone,

I wanted to thank everyone for their helpful suggestions and wanted to
share the best solutions I heard of and found.

One solution is to include a rule in the makefile for every sourcecode file
that that copies it and only updates the copy if something has changed (see
Nick's email below).

Another one is the use of a non-standard make program like makepp, that
allows for using md5 checksums for files instead of timestamp in order to
derive which files have to be rebuild.

Thanks again!

Holger

On Sat, Nov 19, 2011 at 7:58 AM, Holger Hoefling <hhoeflin@gmail.com> wrote:

> Hey Nick,
>
> thank you very much. That sounds like a very good solution to my problem
> that does not require changes to org-mode.
>
> Best
>
> Holger
>
> On Sat, Nov 19, 2011 at 5:00 AM, Nick Dokos <nicholas.dokos@hp.com> wrote:
>
>> Holger Hoefling <hhoeflin@gmail.com> wrote:
>>
>> > I think you misunderstood me there - I am actually not worried about how
>> > computationally intensive the tangling process is. This always works
>> very
>> > quickly, so even if they have to be copied around and take a bit
>> longer, I
>> > would not mind.
>> >
>>
>> Ah, ok - so you are talking about
>>
>>       tangle                    compile
>>   org -------> bunch of files -------------> output
>>
>> The tangling step produces a bunch of files that are (re)compiled (or in
>> any case require some sort of lengthy processing) to produce some output
>> file.
>>
>>
>> IMO, the best way to deal with it is still make: let's say
>>
>>     foo.org  ----> a.x b.x c.x ------> foo.out
>>
>> where the first arrow is the tangle and the second arrow is some
>> processor, call it X.
>> The standard way to set up a makefile is schematically:
>>
>> --8<---------------cut here---------------start------------->8---
>> foo.out: a.x b.x c.x
>>         X a.x b.c c.x -o foo.out
>>
>> a.x b.x c.x: foo.org
>>         tangle foo.org
>>
>> --8<---------------cut here---------------end--------------->8---
>>
>>
>> Rewrite the make file as follows:
>>
>>
>> --8<---------------cut here---------------start------------->8---
>> foo.out: a.y b.y c.y
>>     X a.y b.y c.y -o foo.out
>>
>> a.y: a.x
>>     cmp --silent a.x a.y || cp a.x a.y
>>
>> b.y: b.x
>>     cmp --silent b.x b.y || cp b.x b.y
>>
>> c.y: c.x
>>     cmp --silent c.x c.y || cp c.x c.y
>>
>> a.x b.x c.x: foo.org
>>     tangle foo.org
>> --8<---------------cut here---------------end--------------->8---
>>
>>
>> So if the *contents* of (say) a.x have not changed by the tangling, it
>> compares
>> equal to a.y and the copy is skipped. That leaves a.y untouched.
>>
>> OTOH, if the contents of a.x change (or a.y does not exist in the first
>> place), the comparison fails and we copy a.x to a.y.  That updates a.y
>> and forces further updates on anything that depends on it.
>>
>> Using some make fu (works for GNU make, but not necessarily for other
>> makes),
>> you can write it more compactly:
>>
>>
>> --8<---------------cut here---------------start------------->8---
>> foo.out: a.y b.y c.y
>>     X a.y b.y c.y -o foo.out
>>
>> %.y: %.x
>>     -cmp --silent $< $@ || cp $< $@
>>
>> a.x b.x c.x: foo.org
>>     tangle foo.org
>> --8<---------------cut here---------------end--------------->8---
>>
>> HTH,
>> Nick
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4549 bytes --]

Re: Not overwriting unchanged source code files when tangling

[Allen S. Rout]

On 11/19/2011 01:32 PM, Holger Hoefling wrote:
> Hi everyone,
>
> I wanted to thank everyone for their helpful suggestions and wanted to
> share the best solutions I heard of and found.
>
> One solution is to include a rule in the makefile for every sourcecode
> file that that copies it and only updates the copy if something has
> changed (see Nick's email below).
>

Or tangle to one directory, and then copy to another;  then there's one 
rule for all of the transitions.


work/%: tangle/%
	@cmp --silent  $< $@ || ( echo "Updating $@" ; cp $< $@ )

I liked Nick's "cmp" invocation; I started out with diff. :)


- Allen S. Rout

Re: Not overwriting unchanged source code files when tangling

[Nick Dokos]

Allen S. Rout <asr@ufl.edu> wrote:

> On 11/19/2011 01:32 PM, Holger Hoefling wrote:
> > Hi everyone,
> >
> > I wanted to thank everyone for their helpful suggestions and wanted to
> > share the best solutions I heard of and found.
> >
> > One solution is to include a rule in the makefile for every sourcecode
> > file that that copies it and only updates the copy if something has
> > changed (see Nick's email below).
> >
> 
> Or tangle to one directory, and then copy to another;  then there's
> one rule for all of the transitions.
> 
> 
> work/%: tangle/%
> 	@cmp --silent  $< $@ || ( echo "Updating $@" ; cp $< $@ )
> 
> I liked Nick's "cmp" invocation; I started out with diff. :)
> 

I first saw it in Kernighan and Pike's "Unix Programming Environment"
(1984!).  I'm sure the trick went back even further.

Nick

Re: Not overwriting unchanged source code files when tangling

[cberry]

Holger Hoefling <hhoeflin@gmail.com> writes:

> Hi Carsten, thanks for the suggestion, but as I agree with Brian. If
> there is more than one source file in the org-file, then the whole
> project would still be recompiled, not just the updated file.To be
> more exact, I actually don&#39;t want to compile things, but run R
> scripts using make. 

There are caching tools in R that might handle this. packages weaver and
cacheSweave come to mind.

For example, tangling your *.org file to produce a trivial Sweave
document consisting of a single code chunk with the 'cache=T' argument,
then Sweaving it using the 'weaver' driver from the weaver package will
cache all the computations. Subsequent changes to the code followed by a
new weave will result in cached values being used when possible and
fresh computations being done as needed. I believe this is done on an
expression by expression basis so adding a bit of whitespace or a
comment will not trigger recomputation.

So you can overwrite the document and making trivial changes to
expressions without having to recompute expressions that have not
changed (and have no dependencies that are changed).


HTH,

Chuck



> So the waiting time if a computationally intensive step is repeated
> although it is not necessary can be substantial.

> I wonder how difficult the following change would be (no emacs lisp
> experience, also do not know the org source code):- would it be
> possible to write out the source files when tangling into a temporary
> directory, then compare to the actual target files and overwrite only
> if something has changed? Then the time stamps would stay
> fixed. Hopefully, this would not involve too much work:


> - creating temporary files and remembering the mapping to true files-
> tangling out as usual into temporary files (so probably little change
> there)- compare temporary file to true file (does emacs already have a
> diff utility that could be used?)

> - overwrite true file if any changes- delete temporary
> filesEspecially, with this method no dependencies would be necessary
> and it would not be necessary to keep track in the org files which
> source blocks have been changed since the last tangling

> Thank you for your suggestionHolger

> On Fri, Nov 18, 2011 at 6:02 PM, Carsten Dominik <span dir="ltr"><mailto:carsten.dominik@gmail.com></span> wrote:

>
> On 18.11.2011, at 14:17, Holger Hoefling wrote:
>
>> Hi,
>>

>> I have a problem/request for org-mode and was looking for help. I am
>> using org-mode to write source code files and tangle them out. I want
>> to compile them using make. My problem now is that org-mode
>> overwrites the old files every time I tangle them out, therefore also
>> updating the time stamp - even if nothing has changed. Subsequently,
>> when I run make, everything gets recompiled, not just the changed
>> source code files as all time stamps have changed.
>
>>
>> Is there an option for org-mode to only overwrite source code files
>> that get tangled out if they have truly changed?
>
> How about changing the make file so that the dependence is on the Org file, not on the source file?
> You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it?
>
> Something along the lines of.... (untested, and probably wrong in this way...)
>
> file.o: http://somefile.org
>    emacs -batch --eval &#39;(org-babel-tangle-file "http://somefile.org")&#39;
>    cc file.o ....
>
> - Carsten

-- 
Charles C. Berry                            Dept of Family/Preventive Medicine
cberry@tajo.ucsd.edu			    UC San Diego
http://famprevmed.ucsd.edu/faculty/cberry/  La Jolla, San Diego 92093-0901

Re: Not overwriting unchanged source code files when tangling

[Achim Gratz]

Holger Hoefling <hhoeflin@gmail.com> writes:
> I have a problem/request for org-mode and was looking for help. I am
> using org-mode to write source code files and tangle them out. I want
> to compile them using make. My problem now is that org-mode overwrites
> the old files every time I tangle them out, therefore also updating
> the time stamp - even if nothing has changed. Subsequently, when I run
> make, everything gets recompiled, not just the changed source code
> files as all time stamps have changed.

Make really doesn't have the smarts to deal with builds that it doesn't
fully control.  In your case, the actual source aren't the tangled
files, but really (parts of) an orgmode file.  With a bit of clever
organization it should be possible to employ git to keep track of the
"sources" for make.  In a nutshell, you'd check in the sources into a
repository and only compile those that are different from the last
commit.  After the successful compile you autocommit the tangled files.
Since you will probably don't need version control of the tangled files
(although that might be useful for debugging of the tangling process
itself), you could either amend each commit or just periodically chop
the repository and do a garbage collect.

Putting the smarts into orgmode to keep track of each tangle block looks
complicated in the general case; while possible is a bit of a stretch
IMHO.  It would also produce false positives for a lot of cases that I'd
consider useful (splitting or fusioning tangle blocks without changing
the result).  But the solution outlined above could be used if the
content of each file to be tangled was always available in full in a
buffer before writing to the tangled file.  The buffer could just be
dropped if the SHA1 of the file on disk and the buffer content is
identical.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Not overwriting unchanged source code files when tangling

[Rustom Mody]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

Such tools are for example scons is a make alternative that rethinks make's
"older timestamp => rebuild"

From www.scons.org/architecture/index.html

*(S)Cons decides if a file was out-of-date by using MD5 checksums of the
> contents of files, not timestamps. *
>


SCons also comes to mind because you are using make for R not (traditional)
programming


*Most of the build tools just mentioned were written by programmers and for
> programmers. The fact that most programmer-friendly utilities do a poor job
> of fulfilling the needs of non-programmers prompted Greg Wilson to organize
> the Software Carpentry competition in January 2000. Software Carpentry was
> an open design contest with the express goal of producing a set of
> next-generation utilities, including a build tool, that would be accessible
> not only to programmers but also to computer **users** such as physical
> scientists. *



[Disclaimer: Ive not tried scons myself]

[-- Attachment #2: Type: text/html, Size: 1526 bytes --]

Re: Not overwriting unchanged source code files when tangling

[Rustom Mody]

Sorry for the earlier mail: Send got pressed on a half-cooked mail :-)

Heres a cleanup.
--------------------------------------------------------------------------

Build-tools like scons rethink make's "older timestamp => rebuild"
model.  You may want to look at one such.

From http://www.scons.org/architecture/index.html

> (S)Cons decides if a file was out-of-date by using MD5 checksums of the contents of
> files, not timestamps.

SCons also comes to mind because you are using make for R not
(traditional) programming

[Again from the above link]
> Most of the build tools just mentioned were written by programmers and for
> programmers.

....

> (SCons arose from a)
> design contest with the express goal of producing a set of next-generation utilities,
> including a build tool, that  would be accessible not only to programmers but also to
> computer users such as physical scientists.


[Disclaimer: Ive not tried Scons myself]