problems with export and :cache

problems with export and :cache

[Andreas Leha]

Hi all,

babel's :cache seems to be ignored during export.  At least on #+call
lines.

In the example below the caching works fine for interactive evaluation,
i.e. C-c C-c on the #+call line returns immediately.

If I export the subtree with the #+call line, however, the code block
gets executed and the export is slow.

I'd expect no evaluation even during export.

Is this a bug or am I missing something?

Regards,
Andreas

PS: The example:
--8<---------------cut here---------------start------------->8---
* Test Cached Export
** A long running code block.
#+name: foo
#+begin_src emacs-lisp :var bar="baz"
  (sit-for 15)
  (message "bar=%S" bar)
#+end_src

#+RESULTS: foo
: bar="baz"

** Calling

Exporting this subtree will demonstrate my problem.  I expect the call
line below to not execute anything.  This works for interactive
execution (C-c C-c).  But if I export this subtree only, the code is
executed.

This returns immediately thanks to the cached result.
#+call: foo("qux") :cache yes

#+results[f2b650eb5296f72a1f7237c2a65b7fb3443acf5f]:
: bar="qux"
--8<---------------cut here---------------end--------------->8---

Re: problems with export and :cache

[Andreas Leha]

Hi all,

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
> Hi all,
>
> babel's :cache seems to be ignored during export.  At least on #+call
> lines.
>
> In the example below the caching works fine for interactive evaluation,
> i.e. C-c C-c on the #+call line returns immediately.
>
> If I export the subtree with the #+call line, however, the code block
> gets executed and the export is slow.
>
> I'd expect no evaluation even during export.
>
> Is this a bug or am I missing something?
>
> Regards,
> Andreas
>
> PS: The example:
>
> * Test Cached Export
> ** A long running code block.
> #+name: foo
> #+begin_src emacs-lisp :var bar="baz"
>   (sit-for 15)
>   (message "bar=%S" bar)
> #+end_src
>
> #+RESULTS: foo
> : bar="baz"
>
> ** Calling
>
> Exporting this subtree will demonstrate my problem.  I expect the call
> line below to not execute anything.  This works for interactive
> execution (C-c C-c).  But if I export this subtree only, the code is
> executed.
>
> This returns immediately thanks to the cached result.
> #+call: foo("qux") :cache yes
>
> #+results[f2b650eb5296f72a1f7237c2a65b7fb3443acf5f]:
> : bar="qux"


I should have added that adding :eval no-export to the #+call line does
not help either.

To me this seems to be a bug.

Best,
Andreas

Re: problems with export and :cache

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 2696 bytes --]

Hi Andreas,

2015ko urriak 28an, Andreas Leha-ek idatzi zuen:
> 
> Hi all,
> 
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>> Hi all,
>> 
>> babel's :cache seems to be ignored during export.  At least on #+call
>> lines.
>> 
>> In the example below the caching works fine for interactive evaluation,
>> i.e. C-c C-c on the #+call line returns immediately.
>> 
>> If I export the subtree with the #+call line, however, the code block
>> gets executed and the export is slow.

Export works by making a copy of the org buffer, modifying it in some
ways while executing babel code, then exporting the modified buffer
copy.  The modifications can under some circumstances change the
relevant features for the cache, leading to spurious re-evaluation.

I tried to write a patch that would do the buffer modifications in such
a way that they would not affect the cache.  But I was never happy with
the patch, and also could not find a nice simple test case for the
re-evaluation bug.  So this work never went anywhere.  (It was also 2
years ago, so things could have changed quite a bit.)  The simple patch
attached to this message fixes a bug that my testing indicated was
responsible for erroneous re-evaluations at least some of the time.

I personally regard the babel cache as dangerous and unpredictable in
its present form.  You’re much better off using language-specific
caching/memoization features and/or a disciplined regime of manual
reevaluation.

>> 
>> I'd expect no evaluation even during export.
>> 
>> Is this a bug or am I missing something?
>> 
>> Regards,
>> Andreas
>> 
>> PS: The example:
>> 
>> * Test Cached Export
>> ** A long running code block.
>> #+name: foo
>> #+begin_src emacs-lisp :var bar="baz"
>> (sit-for 15)
>> (message "bar=%S" bar)
>> #+end_src
>> 
>> #+RESULTS: foo
>> : bar="baz"
>> 
>> ** Calling
>> 
>> Exporting this subtree will demonstrate my problem.  I expect the call
>> line below to not execute anything.  This works for interactive
>> execution (C-c C-c).  But if I export this subtree only, the code is
>> executed.
>> 
>> This returns immediately thanks to the cached result.
>> #+call: foo("qux") :cache yes
>> 
>> #+results[f2b650eb5296f72a1f7237c2a65b7fb3443acf5f]:
>> : bar="qux"
> 
> 
> I should have added that adding :eval no-export to the #+call line does
> not help either.

There are two places you could add the :eval option.  See
(info "(org) Evaluating code blocks") .  You may have to experiment with
putting it in one or both places in order to get the desired result.
(If none of the possible combinations works, then it is a bug IMO.)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-babel-fix-header-arg-duplication.patch --]
[-- Type: text/x-diff, Size: 1950 bytes --]

From d7355798edc643bbbca7ab1ead2a2e11f37aa64c Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Thu, 29 Oct 2015 13:31:28 +0000
Subject: [PATCH] babel: fix header arg duplication

* lisp/ob-core.el (org-babel-process-params): Make idempotent.
* testing/lisp/test-ob.el (ob/process-params-no-duplicates): New test.
---
 lisp/ob-core.el         |  8 +++++++-
 testing/lisp/test-ob.el | 12 ++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 61753ce..fd6d785 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -1597,7 +1597,13 @@ shown below.
       (cons :result-type  (cond ((member "output" result-params) 'output)
 				((member "value" result-params) 'value)
 				(t 'value))))
-     (org-babel-get-header params :var 'other))))
+     (loop for item in params
+	   unless (memq (car item) '(:colname-names
+				     :rowname-names
+				     :result-params
+				     :result-type
+				     :var))
+	   collect item))))
 
 ;; row and column names
 (defun org-babel-del-hlines (table)
diff --git a/testing/lisp/test-ob.el b/testing/lisp/test-ob.el
index 508a3ed..c2feb39 100644
--- a/testing/lisp/test-ob.el
+++ b/testing/lisp/test-ob.el
@@ -1477,6 +1477,18 @@ echo \"$data\"
   (should (equal "foo\\\"bar"
 		 (org-babel-script-escape "\"foo\\\\\\\"bar\""))))
 
+(ert-deftest ob/process-params-no-duplicates ()
+    (should (equal (org-babel-process-params '((:colname-names . 1)
+                                               (:rowname-names . 1)
+                                               (:result-params . 1)
+                                               (:result-type . 1)
+                                               (:var . "\"foo\"")))
+                   '((:var)
+		     (:colname-names . 1)
+		     (:rowname-names . 1)
+		     (:result-params . 1)
+		     (:result-type . value)))))
+
 (provide 'test-ob)
 
 ;;; test-ob ends here
-- 
2.6.2


[-- Attachment #3: Type: text/plain, Size: 16 bytes --]


-- 
Aaron Ecay

Re: problems with export and :cache

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> The simple patch attached to this message fixes a bug that my testing
> indicated was responsible for erroneous re-evaluations at least some
> of the time.

Thank you. A minor comment follows.

>        (cons :result-type  (cond ((member "output" result-params) 'output)
>  				((member "value" result-params) 'value)
>  				(t 'value))))
> -     (org-babel-get-header params :var 'other))))
> +     (loop for item in params
> +	   unless (memq (car item) '(:colname-names
> +				     :rowname-names
> +				     :result-params
> +				     :result-type
> +				     :var))
> +	   collect item))))

Using `loop' here is, IMO, over-doing it. You can use instead, e.g.,

  (org-remove-if
   (lambda (item)
     (memq (car item)
           '(:colname-names :rowname-names :result-params :result-type :var)))
   params)

Another option is to extend `org-babel-get-header' to accept a list of
keys instead of a single key.

In any case, this patch, or a variant thereof, should be applied to
maint.


Regards,

-- 
Nicolas Goaziou

Re: problems with export and :cache

[Aaron Ecay]

Hi Nicolas,

2015ko urriak 29an, Nicolas Goaziou-ek idatzi zuen:
> 
> Hello,
> 
> Aaron Ecay <aaronecay@gmail.com> writes:
> 
>> The simple patch attached to this message fixes a bug that my testing
>> indicated was responsible for erroneous re-evaluations at least some
>> of the time.
> 
> Thank you. A minor comment follows.
> 
>> (cons :result-type  (cond ((member "output" result-params) 'output)
>> ((member "value" result-params) 'value)
>> (t 'value))))
>> -     (org-babel-get-header params :var 'other))))
>> +     (loop for item in params
>> +	   unless (memq (car item) '(:colname-names
>> +				     :rowname-names
>> +				     :result-params
>> +				     :result-type
>> +				     :var))
>> +	   collect item))))
> 
> Using `loop' here is, IMO, over-doing it. You can use instead, e.g.,
> 
>   (org-remove-if
>    (lambda (item)
>      (memq (car item)
>            '(:colname-names :rowname-names :result-params :result-type :var)))
>    params)

OK.  I’ll push this to maint later today.

> 
> Another option is to extend `org-babel-get-header' to accept a list of
> keys instead of a single key.

I’d like to remove this function, actually.  I hate the org functions
that do X most of the time, but the opposite of X when you pass them
an optional argument (in this case the third arg ‘other’).  Almost
all (a couple dozen in total) of the calls to o-b-get-header are in
the context (mapcar #'cdr (org-babel-get-header params :var)).  I’d
like to introduce a function org-babel--get-var-values to cover these
cases, and replace the remaining 2 calls in the codebase (which both
extract :column-names) with assq (since :column-names should appear
maximally once, unlike :var).  The sole use of the evil 'other arg is
refactored away by my patch.

Thoughts?

-- 
Aaron Ecay

Re: problems with export and :cache

[Nicolas Goaziou]

Aaron Ecay <aaronecay@gmail.com> writes:

> I’d like to remove this function, actually.  I hate the org functions
> that do X most of the time, but the opposite of X when you pass them
> an optional argument (in this case the third arg ‘other’).

> Almost all (a couple dozen in total) of the calls to o-b-get-header
> are in the context (mapcar #'cdr (org-babel-get-header params :var)).
> I’d like to introduce a function org-babel--get-var-values to cover
> these cases, and replace the remaining 2 calls in the codebase (which
> both extract :column-names) with assq (since :column-names should
> appear maximally once, unlike :var). The sole use of the evil 'other
> arg is refactored away by my patch.
>
> Thoughts?

Could these 2 use cases be refactored?

  (mapcar #'cdr (org-babel-get-header params :var))

is not very far (but stlightly different) from

  (cadr (org-babel-get-header params :colname-names))

when there is a single association in PARAMS. Otherwise, it is up to the
user to know that there can be multiple :var keys but only
one :column-names.

IOW, aren't you after `org-babel-get-header' without the OTHER argument?

Regards,

Re: problems with export and :cache

[Aaron Ecay]

Hi Nicolas,

2015ko urriak 29an, Nicolas Goaziou-ek idatzi zuen:
> Could these 2 use cases be refactored?
> 
>   (mapcar #'cdr (org-babel-get-header params :var))
> 
> is not very far (but stlightly different) from
> 
>   (cadr (org-babel-get-header params :colname-names))
> 
> when there is a single association in PARAMS. Otherwise, it is up to the
> user to know that there can be multiple :var keys but only
> one :column-names.

But users know that anyway, since it’s implied by the semantics of
:colname-names.  Actually the most typical way of accessing the value
of :colname-names in the code is (cdr (assoc :colname-names params))
(~20 uses, which ought to use assq instead); using o-b-get-header for
:colname-names is an exception.

The issue is that o-b-g-h gives an API for accessing key-values in a
1:many mapping.  But :var is the only key that can actually be 1:many.
All other keys are singletons, and can be handled through the usual
emacs API for 1:1 mappings (assq and friends) – and in the existing
codebase they often are.  If we provide an API within babel, we should
make it as specific as possible.

(OTOH if we provide 1:many mapping access, we should name it
org-assq-multiple or something like that.  But it’s unnecessary in this
case.)

-- 
Aaron Ecay

Re: problems with export and :cache

[Nicolas Goaziou]

Aaron Ecay <aaronecay@gmail.com> writes:

> But users know that anyway, since it’s implied by the semantics of
> :colname-names.  Actually the most typical way of accessing the value
> of :colname-names in the code is (cdr (assoc :colname-names params))
> (~20 uses, which ought to use assq instead); using o-b-get-header for
> :colname-names is an exception.
>
> The issue is that o-b-g-h gives an API for accessing key-values in a
> 1:many mapping.  But :var is the only key that can actually be 1:many.
> All other keys are singletons, and can be handled through the usual
> emacs API for 1:1 mappings (assq and friends) – and in the existing
> codebase they often are.  If we provide an API within babel, we should
> make it as specific as possible.

Ideally, the API should be transparent to the internal representation of
parameters used in Babel. Using `assq' means 1) you know that Babel
internally uses alists for parameters 2) you hope it will do it forever.

In any case, your suggestion is already better than the current
situation in Babel. So I'm fine with it. It might not be optimal
however.

Regards,

Re: problems with export and :cache

[Aaron Ecay]

Hi Nicolas,

2015ko urriak 29an, Aaron Ecay-ek idatzi zuen:
>> Using `loop' here is, IMO, over-doing it. You can use instead, e.g.,
>> 
>> (org-remove-if
>> (lambda (item)
>> (memq (car item)
>> '(:colname-names :rowname-names :result-params :result-type :var)))
>> params)
> 
> OK.  I’ll push this to maint later today.

Pushed as 54d3032

-- 
Aaron Ecay

Re: problems with export and :cache

[Andreas Leha]

Hi all,

As a follow-up.  The caching produces un-expected results for call lines
'in the other direction' as well: Changes in the arguments to the called
code block do not invalidate the cache.

--8<---------------cut here---------------start------------->8---
* Test Cached Export

** A long running code block.
#+name: foo
#+begin_src emacs-lisp :var bar="baz"
  (sit-for 15)
  (message "bar=%S" bar)
#+end_src


** Cached Calls
Execute these calls one after the other.

#+name: cached_call
#+call: foo("qux") :cache yes

#+results[f2b650eb5296f72a1f7237c2a65b7fb3443acf5f]: cached_call
: bar="qux"


#+name: cached_call
#+call: foo("qux") :cache yes


#+name: cached_call
#+call: foo("quxxxxxxx") :cache yes


For me, the last one does not call foo, but I expect it to do so.
--8<---------------cut here---------------end--------------->8---

Regards,
Andreas


PS: I have not updated org yet.

Re: problems with export and :cache

[Achim Gratz]

Aaron Ecay writes:
>> Another option is to extend `org-babel-get-header' to accept a list of
>> keys instead of a single key.
>
> I’d like to remove this function, actually.  I hate the org functions
> that do X most of the time, but the opposite of X when you pass them
> an optional argument (in this case the third arg ‘other’).  Almost
> all (a couple dozen in total) of the calls to o-b-get-header are in
> the context (mapcar #'cdr (org-babel-get-header params :var)).  I’d
> like to introduce a function org-babel--get-var-values to cover these
> cases, and replace the remaining 2 calls in the codebase (which both
> extract :column-names) with assq (since :column-names should appear
> maximally once, unlike :var).  The sole use of the evil 'other arg is
> refactored away by my patch.

Your commit 0d000f5680 implementing this (I can't find the push
announced on the list, so I hope it's OK to piggy-back on here) breaks
the test ob-awk/tabular-input.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: problems with export and :cache

[Aaron Ecay]

Hi Achim,

2015ko azaroak 7an, Achim Gratz-ek idatzi zuen:
> Your commit 0d000f5680 implementing this (I can't find the push
> announced on the list, so I hope it's OK to piggy-back on here) breaks
> the test ob-awk/tabular-input.

Thanks for the report.  Fixed in 4773baf.

-- 
Aaron Ecay

Re: problems with export and :cache

[Achim Gratz]

Aaron Ecay writes:
>> Your commit 0d000f5680 implementing this (I can't find the push
>> announced on the list, so I hope it's OK to piggy-back on here) breaks
>> the test ob-awk/tabular-input.
>
> Thanks for the report.  Fixed in 4773baf.

Fix confirmed, thank you.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: problems with export and :cache

[Andreas Leha]

Hi Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:
> Hi Andreas,
>
> 2015ko urriak 28an, Andreas Leha-ek idatzi zuen:
>> 
>> Hi all,
>> 
>> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>>> Hi all,
>>> 
>>> babel's :cache seems to be ignored during export.  At least on #+call
>>> lines.
>>> 
>>> In the example below the caching works fine for interactive evaluation,
>>> i.e. C-c C-c on the #+call line returns immediately.
>>> 
>>> If I export the subtree with the #+call line, however, the code block
>>> gets executed and the export is slow.
>
> Export works by making a copy of the org buffer, modifying it in some
> ways while executing babel code, then exporting the modified buffer
> copy.  The modifications can under some circumstances change the
> relevant features for the cache, leading to spurious re-evaluation.
>
> I tried to write a patch that would do the buffer modifications in such
> a way that they would not affect the cache.  But I was never happy with
> the patch, and also could not find a nice simple test case for the
> re-evaluation bug.  So this work never went anywhere.  (It was also 2
> years ago, so things could have changed quite a bit.)  The simple patch
> attached to this message fixes a bug that my testing indicated was
> responsible for erroneous re-evaluations at least some of the time.
>

Thanks a lot.  I applied your patch but it seems that your patch does
not solve the problem in this specific situation.  Or do I miss
something?


> I personally regard the babel cache as dangerous and unpredictable in
> its present form.  You’re much better off using language-specific
> caching/memoization features and/or a disciplined regime of manual
> reevaluation.

I agree and avoid the caching mechanism generally.  But all alternative
approaches do involve more work, so I am regularly tempted to use it
again.

Generally, I think that caching is a sensitive area.  And if we think
that it is unpredictable and advise people to stay off of it, we are
better off removing it than offering it in this state.  At least until
it behaves (more) predictable.


>
>>> 
>>> I'd expect no evaluation even during export.
>>> 
>>> Is this a bug or am I missing something?
>>> 
>>> Regards,
>>> Andreas
>>> 
>>> PS: The example:
>>> 
>>> * Test Cached Export
>>> ** A long running code block.
>>> #+name: foo
>>> #+begin_src emacs-lisp :var bar="baz"
>>> (sit-for 15)
>>> (message "bar=%S" bar)
>>> #+end_src
>>> 
>>> #+RESULTS: foo
>>> : bar="baz"
>>> 
>>> ** Calling
>>> 
>>> Exporting this subtree will demonstrate my problem.  I expect the call
>>> line below to not execute anything.  This works for interactive
>>> execution (C-c C-c).  But if I export this subtree only, the code is
>>> executed.
>>> 
>>> This returns immediately thanks to the cached result.
>>> #+call: foo("qux") :cache yes
>>> 
>>> #+results[f2b650eb5296f72a1f7237c2a65b7fb3443acf5f]:
>>> : bar="qux"
>> 
>> 
>> I should have added that adding :eval no-export to the #+call line does
>> not help either.
>
> There are two places you could add the :eval option.  See
> (info "(org) Evaluating code blocks") .  You may have to experiment with
> putting it in one or both places in order to get the desired result.
> (If none of the possible combinations works, then it is a bug IMO.)

True.  But can you explain to me why it is not possible to disable the
evaluation of a call line?  If the evaluation of a code block is
disabled (:eval never) there is no evaluation and no results.  If I
specify ':eval never' at the end of the #+call line I do not get any
results either but the called source block is unnecessarily and
unexpectedly evaluated.

So, basically I would argue that

  #+call: foo("qux") :eval never

should do what it says an evaluate never, i.e. it should behave as

  #+call: foo[:eval never]("qux") :eval never

Regards,
Andreas




>
> From d7355798edc643bbbca7ab1ead2a2e11f37aa64c Mon Sep 17 00:00:00 2001
> From: Aaron Ecay <aaronecay@gmail.com>
> Date: Thu, 29 Oct 2015 13:31:28 +0000
> Subject: [PATCH] babel: fix header arg duplication
>
> * lisp/ob-core.el (org-babel-process-params): Make idempotent.
> * testing/lisp/test-ob.el (ob/process-params-no-duplicates): New test.
> ---
>  lisp/ob-core.el         |  8 +++++++-
>  testing/lisp/test-ob.el | 12 ++++++++++++
>  2 files changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/lisp/ob-core.el b/lisp/ob-core.el
> index 61753ce..fd6d785 100644
> --- a/lisp/ob-core.el
> +++ b/lisp/ob-core.el
> @@ -1597,7 +1597,13 @@ shown below.
>        (cons :result-type  (cond ((member "output" result-params) 'output)
>  				((member "value" result-params) 'value)
>  				(t 'value))))
> -     (org-babel-get-header params :var 'other))))
> +     (loop for item in params
> +	   unless (memq (car item) '(:colname-names
> +				     :rowname-names
> +				     :result-params
> +				     :result-type
> +				     :var))
> +	   collect item))))
>  
>  ;; row and column names
>  (defun org-babel-del-hlines (table)
> diff --git a/testing/lisp/test-ob.el b/testing/lisp/test-ob.el
> index 508a3ed..c2feb39 100644
> --- a/testing/lisp/test-ob.el
> +++ b/testing/lisp/test-ob.el
> @@ -1477,6 +1477,18 @@ echo \"$data\"
>    (should (equal "foo\\\"bar"
>  		 (org-babel-script-escape "\"foo\\\\\\\"bar\""))))
>  
> +(ert-deftest ob/process-params-no-duplicates ()
> +    (should (equal (org-babel-process-params '((:colname-names . 1)
> +                                               (:rowname-names . 1)
> +                                               (:result-params . 1)
> +                                               (:result-type . 1)
> +                                               (:var . "\"foo\"")))
> +                   '((:var)
> +		     (:colname-names . 1)
> +		     (:rowname-names . 1)
> +		     (:result-params . 1)
> +		     (:result-type . value)))))
> +
>  (provide 'test-ob)
>  
>  ;;; test-ob ends here
> -- 
> 2.6.2

Re: problems with export and :cache

[Aaron Ecay]

Hi Andreas,

2015ko urriak 29an, Andreas Leha-ek idatzi zuen:
> Thanks a lot.  I applied your patch but it seems that your patch does
> not solve the problem in this specific situation.  Or do I miss
> something?

That patch only solves one possible source of the problem.  There are
others, which are likely affecting you.

> 
> 
>> I personally regard the babel cache as dangerous and unpredictable in
>> its present form.  You’re much better off using language-specific
>> caching/memoization features and/or a disciplined regime of manual
>> reevaluation.
> 
> I agree and avoid the caching mechanism generally.  But all alternative
> approaches do involve more work, so I am regularly tempted to use it
> again.
> 
> Generally, I think that caching is a sensitive area.  And if we think
> that it is unpredictable and advise people to stay off of it, we are
> better off removing it than offering it in this state.  At least until
> it behaves (more) predictable.

I agree.  I would be in favor of putting strong warnings in the
documentation that caching is not to be relied on.  (We shouldn’t
remove it from the codebase entirely for backwards compatibility
reasons.)  But we should hear a consensus from the community before
proceeding down this road.

-- 
Aaron Ecay

Re: problems with export and :cache

[Nicolas Goaziou]

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

> Generally, I think that caching is a sensitive area.  And if we think
> that it is unpredictable and advise people to stay off of it, we are
> better off removing it than offering it in this state.  At least until
> it behaves (more) predictable.

Is it necessarily broken?

I mean, if the hash contains the current parameters and contents of the
current block, along with all hashes contents and parameters of related
code blocks (i.e., called or referenced in the block), shouldn't it be
predictable?

Regards,

Re: problems with export and :cache

[Aaron Ecay]

Hi Nicolas,

2015ko urriak 29an, Nicolas Goaziou-ek idatzi zuen:
> 
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
> 
>> Generally, I think that caching is a sensitive area.  And if we think
>> that it is unpredictable and advise people to stay off of it, we are
>> better off removing it than offering it in this state.  At least until
>> it behaves (more) predictable.
> 
> Is it necessarily broken?

If this means “can it ever work?” then I think the answer is “yes it
can”.  But I think the current implementation is broken and likely to
remain so for the foreseeable future.  The issues are:

1. :cache only works for code which is a pure function of its header args
2. When combined with :session, the environment that the code is evaluated
   in is not created anew each time it is run.  This makes it much easier
   to leak references to (e.g.) variables defined in other blocks
3. The proper notion of purity is not easily defined when the code does
   things like modifying the emacs environment, touching the filesystem,
   or accessing the language’s RNG.
4. We (org devs) don’t actually understand how the semantics of cache
   interacts with other babel features.  See:
   <http://mid.gmane.org/86fvqqc8jb.fsf@somewhere.org>.

1-3 are likely to be extremely confusing for users, especially less
technically sophisticated ones (what’s a “pure function” anyway)?  The
inability to give a clear introductory explanation of the feature in
combination with 4 indicating we don’t actually understand it ourselves
makes me feel like we should not be advertising, let alone recommending,
it.

The only other literate programming environment that I know of that
implements such a feature is knitr (for R).  They address these issues
by providing (optional) free-variable analysis to construct a dependency
graph between code blocks.  There is also some handling of RNG seed
values.  The documentation <http://yihui.name/knitr/demo/cache/> is much
more comprehensive, including a prominent statement about the dangers of
side effect-ful code and a nuanced discussion of several issues,
including the RNG.

Aaron

PS I’ve looked through my old notes on the interaction of cache with
babel export.  I can’t reproduce any of my old test cases, so maybe that
bug has been fixed as a side effect of some other change.
 
-- 
Aaron Ecay

Re: problems with export and :cache

[Thomas S. Dye]

Aloha all,

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Nicolas,
>
> 2015ko urriak 29an, Nicolas Goaziou-ek idatzi zuen:
>> 
>> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>> 
>>> Generally, I think that caching is a sensitive area.  And if we think
>>> that it is unpredictable and advise people to stay off of it, we are
>>> better off removing it than offering it in this state.  At least until
>>> it behaves (more) predictable.
>> 
>> Is it necessarily broken?
>
> If this means “can it ever work?” then I think the answer is “yes it
> can”.  But I think the current implementation is broken and likely to
> remain so for the foreseeable future.  The issues are:
>
> 1. :cache only works for code which is a pure function of its header args
> 2. When combined with :session, the environment that the code is evaluated
>    in is not created anew each time it is run.  This makes it much easier
>    to leak references to (e.g.) variables defined in other blocks
> 3. The proper notion of purity is not easily defined when the code does
>    things like modifying the emacs environment, touching the filesystem,
>    or accessing the language’s RNG.
> 4. We (org devs) don’t actually understand how the semantics of cache
>    interacts with other babel features.  See:
>    <http://mid.gmane.org/86fvqqc8jb.fsf@somewhere.org>.
>
> 1-3 are likely to be extremely confusing for users, especially less
> technically sophisticated ones (what’s a “pure function” anyway)?  The
> inability to give a clear introductory explanation of the feature in
> combination with 4 indicating we don’t actually understand it ourselves
> makes me feel like we should not be advertising, let alone recommending,
> it.
>
> The only other literate programming environment that I know of that
> implements such a feature is knitr (for R).  They address these issues
> by providing (optional) free-variable analysis to construct a dependency
> graph between code blocks.  There is also some handling of RNG seed
> values.  The documentation <http://yihui.name/knitr/demo/cache/> is much
> more comprehensive, including a prominent statement about the dangers of
> side effect-ful code and a nuanced discussion of several issues,
> including the RNG.
>
> Aaron
>
> PS I’ve looked through my old notes on the interaction of cache with
> babel export.  I can’t reproduce any of my old test cases, so maybe that
> bug has been fixed as a side effect of some other change.

FWIW, I think relying on cache to speed up export is the wrong
approach.  Having all code run during each export is, to me, a feature
that makes a document "live."  I'm willing to be patient during export
to get this feature.

If speed is important and a live document isn't desired, then one
solution is to rename the results and refer to this name in the
document, rather than to the name of the code block that produced the
results.  I do this manually, which is OK, but I've often wanted
something like :persist-as my-result so I can be certain to have a good
link from the results back to the code that produced them.

That said, points 2 and 4 seem to me serious issues that users must
understand if they choose to use :cache.  So, at the least, the
documentation needs revision.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: problems with export and :cache

[Andreas Leha]

Hi Thomas,

Thomas S. Dye <tsd@tsdye.com> writes:
> Aloha all,
>
> Aaron Ecay <aaronecay@gmail.com> writes:
>
>> Hi Nicolas,
>>
>> 2015ko urriak 29an, Nicolas Goaziou-ek idatzi zuen:
>>> 
>>> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>>> 
>>>> Generally, I think that caching is a sensitive area.  And if we think
>>>> that it is unpredictable and advise people to stay off of it, we are
>>>> better off removing it than offering it in this state.  At least until
>>>> it behaves (more) predictable.
>>> 
>>> Is it necessarily broken?
>>
>> If this means “can it ever work?” then I think the answer is “yes it
>> can”.  But I think the current implementation is broken and likely to
>> remain so for the foreseeable future.  The issues are:
>>
>> 1. :cache only works for code which is a pure function of its header args
>> 2. When combined with :session, the environment that the code is evaluated
>>    in is not created anew each time it is run.  This makes it much easier
>>    to leak references to (e.g.) variables defined in other blocks
>> 3. The proper notion of purity is not easily defined when the code does
>>    things like modifying the emacs environment, touching the filesystem,
>>    or accessing the language’s RNG.
>> 4. We (org devs) don’t actually understand how the semantics of cache
>>    interacts with other babel features.  See:
>>    <http://mid.gmane.org/86fvqqc8jb.fsf@somewhere.org>.
>>
>> 1-3 are likely to be extremely confusing for users, especially less
>> technically sophisticated ones (what’s a “pure function” anyway)?  The
>> inability to give a clear introductory explanation of the feature in
>> combination with 4 indicating we don’t actually understand it ourselves
>> makes me feel like we should not be advertising, let alone recommending,
>> it.
>>
>> The only other literate programming environment that I know of that
>> implements such a feature is knitr (for R).  They address these issues
>> by providing (optional) free-variable analysis to construct a dependency
>> graph between code blocks.  There is also some handling of RNG seed
>> values.  The documentation <http://yihui.name/knitr/demo/cache/> is much
>> more comprehensive, including a prominent statement about the dangers of
>> side effect-ful code and a nuanced discussion of several issues,
>> including the RNG.
>>
>> Aaron
>>
>> PS I’ve looked through my old notes on the interaction of cache with
>> babel export.  I can’t reproduce any of my old test cases, so maybe that
>> bug has been fixed as a side effect of some other change.
>
> FWIW, I think relying on cache to speed up export is the wrong
> approach.  Having all code run during each export is, to me, a feature
> that makes a document "live."  I'm willing to be patient during export
> to get this feature.
>
> If speed is important and a live document isn't desired, then one
> solution is to rename the results and refer to this name in the
> document, rather than to the name of the code block that produced the
> results.  I do this manually, which is OK, but I've often wanted
> something like :persist-as my-result so I can be certain to have a good
> link from the results back to the code that produced them.

There we go again.  Another advice not to use the :cache.

>
> That said, points 2 and 4 seem to me serious issues that users must
> understand if they choose to use :cache.  So, at the least, the
> documentation needs revision.
>

Just to add (maybe that belongs to point 4 anyway): And the caching
should work predictably -- even during export.

Regards,
Andreas

Re: problems with export and :cache

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> If this means “can it ever work?” then I think the answer is “yes it
> can”.  But I think the current implementation is broken and likely to
> remain so for the foreseeable future.  The issues are:
>
> 1. :cache only works for code which is a pure function of its header args
> 2. When combined with :session, the environment that the code is evaluated
>    in is not created anew each time it is run.  This makes it much easier
>    to leak references to (e.g.) variables defined in other blocks
> 3. The proper notion of purity is not easily defined when the code does
>    things like modifying the emacs environment, touching the filesystem,
>    or accessing the language’s RNG.
> 4. We (org devs) don’t actually understand how the semantics of cache
>    interacts with other babel features.  See:
>    <http://mid.gmane.org/86fvqqc8jb.fsf@somewhere.org>.
>
> 1-3 are likely to be extremely confusing for users, especially less
> technically sophisticated ones (what’s a “pure function” anyway)?  The
> inability to give a clear introductory explanation of the feature in
> combination with 4 indicating we don’t actually understand it ourselves
> makes me feel like we should not be advertising, let alone recommending,
> it.
>
> The only other literate programming environment that I know of that
> implements such a feature is knitr (for R).  They address these issues
> by providing (optional) free-variable analysis to construct a dependency
> graph between code blocks.  There is also some handling of RNG seed
> values.  The documentation <http://yihui.name/knitr/demo/cache/> is much
> more comprehensive, including a prominent statement about the dangers of
> side effect-ful code and a nuanced discussion of several issues,
> including the RNG.

Thank you for the explanations.

Assuming the user knows what s?he is doing (so I'm not bothered by
issues 2 and 3), :cache is still a somewhat useful feature. I don't mind
advertising it. Though, I agree we could include big fat warnings about
it in the manual.

Also, :cache might be a bit misleading as it implies more than what this
feature offer, i.e., a dumb "don't update results if contents didn't
change". I cannot think of a better name, tho.


Regards,

-- 
Nicolas Goaziou

Re: problems with export and :cache

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 2743 bytes --]

Hi Nicolas,

2015ko azaroak 1an, Nicolas Goaziou-ek idatzi zuen:
> 
> Hello,
> 
> Aaron Ecay <aaronecay@gmail.com> writes:
> 
>> If this means “can it ever work?” then I think the answer is “yes it
>> can”.  But I think the current implementation is broken and likely to
>> remain so for the foreseeable future.  The issues are:
>> 
>> 1. :cache only works for code which is a pure function of its header args
>> 2. When combined with :session, the environment that the code is evaluated
>> in is not created anew each time it is run.  This makes it much easier
>> to leak references to (e.g.) variables defined in other blocks
>> 3. The proper notion of purity is not easily defined when the code does
>> things like modifying the emacs environment, touching the filesystem,
>> or accessing the language’s RNG.
>> 4. We (org devs) don’t actually understand how the semantics of cache
>> interacts with other babel features.  See:
>> <http://mid.gmane.org/86fvqqc8jb.fsf@somewhere.org>.
>> 
>> 1-3 are likely to be extremely confusing for users, especially less
>> technically sophisticated ones (what’s a “pure function” anyway)?  The
>> inability to give a clear introductory explanation of the feature in
>> combination with 4 indicating we don’t actually understand it ourselves
>> makes me feel like we should not be advertising, let alone recommending,
>> it.
>> 
>> The only other literate programming environment that I know of that
>> implements such a feature is knitr (for R).  They address these issues
>> by providing (optional) free-variable analysis to construct a dependency
>> graph between code blocks.  There is also some handling of RNG seed
>> values.  The documentation <http://yihui.name/knitr/demo/cache/> is much
>> more comprehensive, including a prominent statement about the dangers of
>> side effect-ful code and a nuanced discussion of several issues,
>> including the RNG.
> 
> Thank you for the explanations.
> 
> Assuming the user knows what s?he is doing (so I'm not bothered by
> issues 2 and 3), :cache is still a somewhat useful feature. I don't mind
> advertising it. Though, I agree we could include big fat warnings about
> it in the manual.

See the attached patch.  I’ve tried to put all my experience and best
practices in there; comments are welcome of course.

> 
> Also, :cache might be a bit misleading as it implies more than what this
> feature offer, i.e., a dumb "don't update results if contents didn't
> change". I cannot think of a better name, tho.

There are backwards compatibility implications to renaming the header,
of course.  (And I can’t think of a better name either).

Thanks,

-- 
Aaron Ecay

[-- Attachment #2: 0001-babel-update-the-manual-wrt-cache-header-arg.patch --]
[-- Type: text/x-diff, Size: 2947 bytes --]

From bb0f43948384448225323abcfe7a662d110d1389 Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Wed, 4 Nov 2015 11:57:49 +0000
Subject: [PATCH] babel: update the manual wrt :cache header arg

* doc/org.texi (cache): Update manual section.
---
 doc/org.texi | 36 +++++++++++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 5 deletions(-)

diff --git a/doc/org.texi b/doc/org.texi
index ba402bf..48ae017 100644
--- a/doc/org.texi
+++ b/doc/org.texi
@@ -16166,11 +16166,37 @@ used.
 
 The @code{:cache} header argument controls the use of in-buffer caching of
 the results of evaluating code blocks.  It can be used to avoid re-evaluating
-unchanged code blocks.  Note that the @code{:cache} header argument will not
-attempt to cache results when the @code{:session} header argument is used,
-because the results of the code block execution may be stored in the session
-outside of the Org mode buffer.  The @code{:cache} header argument can have
-one of two values: @code{yes} or @code{no}.
+unchanged code blocks.  When the cache is active, a source block is not
+re-evaluated if a result for it is present in the buffer and neither the
+header arguments (including the value of @code{:var} references) nor the text
+of the block itself has changed since the result was computed.  The feature
+helps avoid re-running long calculations.  However, there are edge cases and
+you should not rely on the cache to behave reliably in all circumstances.
+
+The caching feature works best when a babel block is a pure function of its
+arguments (see @ref{var}).  That is, the function always returns the same
+results when given the same arguments, and does not touch external resources
+(like the filesystem or the language’s RNG) in any way.
+
+The documentation of the knitr reproducible research package for the R
+language has some good discussion of issues that may arise when using the
+cache in such a context.  See @uref{http://yihui.name/knitr/demo/cache/},
+especially the sections “Even more stuff for cache?” and “Reproducibility
+with RNG”.  (Obviously, you will have to abstract away from the knitr
+implementation details which the documentation also discusses.)
+
+Note that the @code{:cache} header argument will attempt to cache results
+when the @code{:session} header argument is used, even though the results of
+the code block execution stored in the session may lead to unexpected
+results.
+
+Noweb references (see @ref{Noweb reference syntax}) are currently not
+expanded when calculating whether the text of the code block has changed.
+Perhaps in principle they ought to be, but this could introduce unexpected
+complexity.  See @uref{http://thread.gmane.org/gmane.emacs.orgmode/79046}.
+
+The @code{:cache} header argument can have one of two values: @code{yes} or
+@code{no}.
 
 @itemize @bullet
 @item @code{no}
-- 
2.6.2

Re: problems with export and :cache

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> See the attached patch.  I’ve tried to put all my experience and best
> practices in there; comments are welcome of course.

Thank you. It looks good. Some comments follow.

> +unchanged code blocks.  When the cache is active, a source block is not
> +re-evaluated if a result for it is present in the buffer and neither the
> +header arguments (including the value of @code{:var} references) nor the text
> +of the block itself has changed since the result was computed.  The feature
> +helps avoid re-running long calculations.  However, there are edge cases and
> +you should not rely on the cache to behave reliably in all circumstances.
> +
> +The caching feature works best when a babel block is a pure function of its
> +arguments (see @ref{var}).  That is, the function always returns the
> same

  (see @ref{var}) => (@pxref{var})

> +results when given the same arguments, and does not touch external resources
> +(like the filesystem or the language’s RNG) in any way.
> +
> +The documentation of the knitr reproducible research package for the R
> +language has some good discussion of issues that may arise when using the
> +cache in such a context.  See @uref{http://yihui.name/knitr/demo/cache/},
> +especially the sections “Even more stuff for cache?” and “Reproducibility
> +with RNG”.

Shouldn't this be : ``Even more stuff for cache?''. Ditto for the other
title.

> (Obviously, you will have to abstract away from the knitr
> +implementation details which the documentation also discusses.)

I think this last paragraph could go in a footnote at the end of the
paragraph before it.

> +Note that the @code{:cache} header argument will attempt to cache results
> +when the @code{:session} header argument is used, even though the results of
> +the code block execution stored in the session may lead to unexpected
> +results.
> +
> +Noweb references (see @ref{Noweb reference syntax}) are currently not

(see @ref{Noweb reference syntax}) => (@pxref{Noweb reference syntax})

> +expanded when calculating whether the text of the code block has changed.
> +Perhaps in principle they ought to be, but this could introduce unexpected
> +complexity.  See @uref{http://thread.gmane.org/gmane.emacs.orgmode/79046}.
> +
> +The @code{:cache} header argument can have one of two values: @code{yes} or
> +@code{no}.

Regards,

-- 
Nicolas Goaziou

Re: problems with export and :cache

[Aaron Ecay]

Hi Nicolas,

Thanks for the comments.  I pushed a patch with all your suggestions
applied.

-- 
Aaron Ecay

Re: problems with export and :cache

[Nicolas Goaziou]

Aaron Ecay <aaronecay@gmail.com> writes:

> Thanks for the comments.  I pushed a patch with all your suggestions
> applied.

Thank you.

Regards,