Re: [PATCH] * lisp/ob-core.el (org-babel-execute-src-block): insert hash for silent results

[Eric Schulte <schulte.eric@gmail.com>]

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Achim,
>
> 2013ko martxoak 10an, Achim Gratz-ek idatzi zuen:
>> But back to my earlier remark about the hash value actually being a
>> signature of the source block and not the result.  If I use noweb
>> references, the reference text is cached, not its expansion.  See the
>> example below where after the first invocation I change the source block
>> referenced to deliver a different result.  That invalidates the cache
>> for direct invocation of that block, but fails to do so for the indirect
>> invocation.  If you look at the two result blocks, you see that the same
>> hash is added to two different blocks.
>
> I think this points in the direction of having the notion of
> dependencies among source blocks.  This is an idea that knitr
> (http://yihui.name/knitr/) implements.  The idea would be to include in
> the hash of a source block X (in addition to all the pieces that are
> already in the hash) the hash of the blocks that X depends on.  So in
> your example, the data that generated the hashes beginning 0bd... would
> be made distinct, because they would include in one case the hash
> 6bd... and in the other d8d... .
>
> As in knitr, I think that manual dependency specification (e.g. in the
> header args of the block) should be possible.  But it would also be
> possible to automatically infer that a block depends on any block that
> it references via a :var header or noweb reference – which would in turn
> automatically fix the case you discussed.
>

This is what is already taking place.  The :var header arguments are
automatically expanded into dependencies between code blocks, and the
results of previous code blocks are included in the hash calculation of
the current code block.

From re-looking at Achim's previous noweb example, it seems that we
currently do *not* include the values of noweb expansions in code block
hash calculations, I think this is a bug which should be fixed.

>
> And when evaluating a block, the dependencies should be (recursively)
> evaluated first, in case any of them has changed.
>

This is exactly what happens currently with previous blocks referenced
through :var header arguments.

>
> Is it clear what I am describing, and do you have thoughts on it?
>

Very, thank you for spelling it out.  I believe that given the bug fix
just mentioned, the current model indeed does support automatic
inference of dependencies between blocks.

>
>> 
>>> If one did want to move hashes to code blocks it would be a major
>>> refactoring which would (in my opinion) require significant
>>> justification.
>> 
>> I'm not disputing that it requires significant effort.  The benefits
>> would be that we might have a chance to clear up some confusion over the
>> code execution model of Babel and better support different ones.
>
> FWIW, I think that hashes shouldn’t be stored in the buffer text at
> all.

To echo Achim's response, you've accidentally uttered Org-mode heresy.
A core design principle is that everything be represented as plain text
in the buffer.  That said, the hashes should be largely hidden by
default, and the degree of hiding can be controlled by the
`org-babel-hash-show' variable.

> 
> They’re not really part of the document data or metadata.  Rather,
> they are information about how the content of the document (code and
> its results) was instantiated/computed in a particular
> environment/occasion.  I’d rather see them stored in a lisp data
> structure.  They could be written out to an invisible file when the
> org buffer is saved, and re-read on load.
>
>> Oh yes, there's a whole set of _other_ problems that are waiting to be
>> solved.  :-)
>
> There always is.  :-)

I think Org-mode already provides the bulk of what is desired.  If we
agree to treat ":cache yes :results none" as obviously taking place for
side effects, and then sticking a hash behind the :cache header argument
with the code block, then what functionality would be missing?

Thanks,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte