Re: [PATCH] * lisp/ob-core.el (org-babel-execute-src-block): insert hash for silent results

[Eric Schulte <schulte.eric@gmail.com>]

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Eric,
>
> I’m jointly replying to 2 of your emails.
>
> 2013ko martxoak 13an, Eric Schulte-ek idatzi zuen:
>> This is what is already taking place.  The :var header arguments are
>> automatically expanded into dependencies between code blocks, and the
>> results of previous code blocks are included in the hash calculation of
>> the current code block.
>
> Wow, I did not realize that the :var handling was so sophisticated.
> Would it be possible to introduce a :depends code-block-name header
> argument, which recycles the same dependency calculation but does not
> bind a variable in the code block?  Many of the variables that I rely on
> between blocks are large data frames, and I worry that dumping them into
> the org buffer and then reloading them into R[fn:1] will result in a
> slowdown and/or loss of some structure in the data.
>

Unless you actually try :var and find it lacking in some way, I'd prefer
to stick with simply using :var to identify dependencies between code
blocks.  We've seen in other places how providing multiple alias for
header arguments increases rather than reduces confusion.

>
> [fn:1] My understanding of the :var-handling code is that this is how it
> acquires the values to assign to the variables, as opposed to re-using a
> variable that is present in a session.  But the code is complex, so
> maybe I’m wrong (again).
>
> I also think this would make the feature more discoverable: a :var is
> just a sub-type of :depends, with extra functionality.  Coming from a
> Sweave/knitr background, I expected something like :depends, and thus
> didn’t notice :var
>

Maybe the documentation of :var should be improved to enhance
discoverability.  I would be happy to apply a patch to this effect.

>
>> 
>> From re-looking at Achim's previous noweb example, it seems that we
>> currently do *not* include the values of noweb expansions in code block
>> hash calculations, I think this is a bug which should be fixed.
>
> +1
>
>> To echo Achim's response, you've accidentally uttered Org-mode heresy.
>
> Oh no.  The good news is that thanks to your and Achim’s explanation, I
> think I now understand this principle better.
>
>>>> Oh yes, there's a whole set of _other_ problems that are waiting to be
>>>> solved.  :-)
>>> 
>>> There always is.  :-)
>> 
>> I think Org-mode already provides the bulk of what is desired.  If we
>> agree to treat ":cache yes :results none" as obviously taking place for
>> side effects, and then sticking a hash behind the :cache header argument
>> with the code block, then what functionality would be missing?
>
> This was more of a joke on my part: life gets boring when you run out of
> problems to work on.  In this specific case, though:
> 1) a :depends header argument
> 2) including the session PID in results hashes by default (because it is
>    the only sensible thing to do)
>

I personally think automatically invalidating all hashes simply because
a new session has been started is surprising and counter-intuitive.
There is a "library of Babel", in which common code blocks (such as one
returning session ID for hashing) may be placed so that they can be
easily re-used across files and users.

>
> 2013ko martxoak 13an, Eric Schulte-ek idatzi zuen:
>> Well, I suppose one man's dirty kludge is another's beautiful hack.  The
>> question here is whether the complexity lies in the implementation (and
>> thus the interface) or in the code block itself.  While I generally
>> prefer the later, in this case of ":results none :cache yes" I would be
>> open to placing some custom logic in the backend, which stores the hash
>> value with the code block, possibly changing
>> 
>>      #+begin_src R :cache yes
>>        # code to perform side effect
>>      #+end_src
>> 
>> to
>>   
>>      #+begin_src R :cache 9f4e5b4b07e93c680ab37fc4ba1f75e1bfc0ee0a
>>        # code to perform side effect
>>      #+end_src
>> 
>> keeping in mind that the actual hash value should be hidden after the
>> first couple of characters.
>
> If you like this solution, may I try once more to convince you of the
> empty #+RESULTS solution I originally proposed?  I looked at the code
> for inserting/hiding/finding hash values, and it looks like it would be
> complicated to change.  Empty #+RESULTS is easy, although perhaps less
> conceptually pure.
>

Why not just return a dummy value at the end of the code block?

    #+begin_src R :cache yes
      # code to perform side effect
      "done"
    #+end_src

>
> If you want the cache in the header, I think I can try to work on a
> patch, but it does look tricky.  So I am not sure I will have time to
> work on it until next week.  (If anyone else wants to beat me to the
> punch, please feel free!)
>
> One question: should we have the cache in the header only for :results
> none blocks, or for all blocks?
>

I'm just as happy raising an error or warning when the :cache and
":results none" options are found together, and doing no caching in that
case.  Users can always just return a dummy value and remove ":results
none".

>
>> I was actually very proud of this solution.  It is what would be done by
>> the framework if we did implement custom support, but by doing it with
>> code blocks the exact mechanics are visible to the user.
>
> Agreed.  But if it is the only “right” thing to do, or one of a very
> small set of “right” things, I think it’s a win in terms of
> conciseness/ease of use to do it automatically.  And I think this is the
> case here: the presence of :session yes is a clear signal that there is
> out-of-band (from org’s perspective) communication happening between
> code blocks, and that the invariance of a result can’t be relied on in a
> different session process.  So when the session PID changes, the hash
> value should change as well, to trigger reevaluation.
>
>> How should session startup be determined if not through inclusion of the
>> session PID in the code block hash?  Perhaps the above could be made
>> more elegant through the addition of an elisp function which returns the
>> pid of the current R session, allowing the above to be truncated to
>> something like the following.
>> 
>>      #+begin_src R :cache yes :session foo :var pid=(R-pid "foo")
>>        # code to perform side effect
>>        x <- 'side effect'
>>        'done'
>>      #+end_src
>> 
>> I don't suppose ESS provides such a function?
>
> You can get the value with
> (process-id (get-process ess-current-process-name)), which you have to
> evaluate in the current session buffer (the one that C-c C-v C-z takes
> you to).  Generally speaking, I guess each ob-foo should provide a
> function to retrieve this value, since it will be different for
> different languages.

It sounds like such an (R-pid "foo") function would be easy to
implement.  I'd vote for that solution (implementing this function in
your .emacs, and then sharing it if necessary) for now.  If this need to
associate PIDs with results becomes more wide spread (in a couple of
years of Org-mode code blocks this is the first time I've seen it
arise), then a built-in solution becomes more appealing.

Hope this is helpful,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte