exporting documents w/ babel results w/o evaluating babel blocks

exporting documents w/ babel results w/o evaluating babel blocks

[Ken Mankoff]

Hi List,

I'm repeating a question here from a recent bug report because I have a feeling official-looking bug reports might get read less than normal questions. Sorry for the re-post.

As of an Org git commit a few weeks ago, Org exporting (and therefore Org) has become basically unusable for me. I used to be able to export code block results without evaluating the block during the export. I can no longer do this.

The following headers used to support the above behavior:

#+BEGIN_SRC python :exports results :results output 
#+BEGIN_SRC python :exports results :results file
  (and possibly with ":session foo" also)

Is anyone running the current git head? And if so, can you export a document that consists of the following, or something similar to the following (i.e. do you use ":cache t" or ":cache nil")?

#+BEGIN_SRC python :results output :exports results :session foo
print("hello, world")
#+END_SRC
#+RESULTS:
: hello, world

If you can export this without running the code, what settings are you using? In particular, what is the value of =org-export-babel-evaluate=?

Thanks,

  -k.

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Eric S Fraga]

On Friday, 20 May 2016 at 15:57, Ken Mankoff wrote:
> Hi List,
>
> I'm repeating a question here from a recent bug report because I have
> a feeling official-looking bug reports might get read less than normal
> questions. Sorry for the re-post.
>
> As of an Org git commit a few weeks ago, Org exporting (and therefore
> Org) has become basically unusable for me. I used to be able to export
> code block results without evaluating the block during the export. I
> can no longer do this.

I can export your example without evaluating the babel block and the
results appear in the exported output.  My org is relatively recent but
may be of a similar vintage to yours (see my signature).

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-775-g3308a5

Re: exporting documents w/ babel results w/o evaluating babel blocks

[John Hendy]

On Fri, May 20, 2016 at 10:57 AM, Ken Mankoff <mankoff@gmail.com> wrote:
> Hi List,
>
> I'm repeating a question here from a recent bug report because I have a feeling official-looking bug reports might get read less than normal questions. Sorry for the re-post.
>
> As of an Org git commit a few weeks ago, Org exporting (and therefore Org) has become basically unusable for me. I used to be able to export code block results without evaluating the block during the export. I can no longer do this.
>
> The following headers used to support the above behavior:
>
> #+BEGIN_SRC python :exports results :results output
> #+BEGIN_SRC python :exports results :results file
>   (and possibly with ":session foo" also)

Well, you're exporting them at least *once*, right? If not, where are
the results coming from?

Perhaps my use case is similar. I often write up an org doc with a
pretty heavy setup heading containing my main data reading in,
manipulation, statistics, etc. Maybe up to 100k total rows in a data
frame in R. Once I run that block the first time I'm working on the
document I just put =:eval no= in the src block options.

I do the same as I'm tweaking plots. Every code block I create has
:eval yes initially and once I'm satisfied with the results I just
change to :eval no and the generated results (for me, typically a
#+results line containing a link to a pdf plot generated by my code
block) are still included.

Does this help at all? Sorry if I'm not understanding

>
> Is anyone running the current git head? And if so, can you export a document that consists of the following, or something similar to the following (i.e. do you use ":cache t" or ":cache nil")?
>
> #+BEGIN_SRC python :results output :exports results :session foo
> print("hello, world")
> #+END_SRC
> #+RESULTS:
> : hello, world
>
> If you can export this without running the code, what settings are you using? In particular, what is the value of =org-export-babel-evaluate=?

Mine is set to t. Interestingly, when I export the above after
deleting the results bit, no new results are generated. When I C-c
C-c, they are replaced. When I add :eval no, it does't appear to run
the code.


Hope that helps,
John

>
> Thanks,
>
>   -k.
>

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Ken Mankoff]

Eric: You're running something newer (by date) than the commit which changed the behavior, which was:

ec615b1 - Fix `org-export-babel-evaluate' handling (2016-04-28)

But with all the branches in git, I don't know if you have that commit or not. Anyway, I'm not sure why it works for you. It doesn't appear to work for John and I the way it used to. Do you have a global "eval: no" set somewhere?

On 2016-05-20 at 12:14, John Hendy <jw.hendy@gmail.com> wrote:
> On Fri, May 20, 2016 at 10:57 AM, Ken Mankoff <mankoff@gmail.com> wrote:
>> As of an Org git commit a few weeks ago, Org exporting (and therefore
>> Org) has become basically unusable for me. I used to be able to
>> export code block results without evaluating the block during the
>> export. I can no longer do this.
>>
>
> Well, you're exporting them at least *once*, right? If not, where are
> the results coming from?

Yes, but only once. Or sometimes I generate the figure elsewhere (via org-edit-special) and then manually put [[file:fig.pdf]] below the #+RESULTS:.

> I do the same as I'm tweaking plots. Every code block I create has
> :eval yes initially and once I'm satisfied with the results I just
> change to :eval no and the generated results (for me, typically a
> #+results line containing a link to a pdf plot generated by my code
> block) are still included.
>
> Does this help at all? Sorry if I'm not understanding

You understand, and this sort-of helps. I have never used :eval before. Things just didn't evaluate by default, and I like it like that.

":eval no" fixes the problem at export, but then I can't evaluate the code myself manually when not exporting.

> Mine is set to t. Interestingly, when I export the above after
> deleting the results bit, no new results are generated. When I C-c
> C-c, they are replaced. When I add :eval no, it does't appear to run
> the code.

Yes same here. This new behavior really sucks.

I'm still trying to find a way where:
  + code results do export
  + code does not export
  + code does not evaluate at export
  + code can still be evaluated by me

This was the behavior prior to

ec615b1 - Fix `org-export-babel-evaluate' handling (2016-04-28)

But not since.

  -k.

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Eric S Fraga]

On Friday, 20 May 2016 at 16:45, Ken Mankoff wrote:
> Eric: You're running something newer (by date) than the commit which
> changed the behavior, which was:
>
> ec615b1 - Fix `org-export-babel-evaluate' handling (2016-04-28)
>
> But with all the branches in git, I don't know if you have that commit
> or not. Anyway, I'm not sure why it works for you. It doesn't appear
> to work for John and I the way it used to. Do you have a global "eval:
> no" set somewhere?

I do have that commit incorporated in the version I am running from what
I see in the git log.

When I export, org asks whether to evaluate the code block and I respond
with no.  The existing results block is therefore exported but not
changed.  If I say yes, a new results block is generated and exported
(and I can tell because it is different to what was there before).

I have org-export-babel-evaluate set to t but also
org-confirm-babel-evaluate to t.

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-775-g3308a5

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Ken Mankoff]

Hi Eric,

On 2016-05-20 at 12:59, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> When I export, org asks whether to evaluate the code block and I
> respond with no. The existing results block is therefore exported but
> not changed. If I say yes, a new results block is generated and
> exported (and I can tell because it is different to what was there
> before).
>
> I have org-export-babel-evaluate set to t but also
> org-confirm-babel-evaluate to t.

Thanks for clarifying your workflow. That seems to work for a small example. But what would you do if you had a manuscript with 100 code blocks in it? And in general you're exporting it a lot without changing anything in code, just editing the non-code text. But occasionally you want to modify a code block (and its results) too?

In the above scenario, I used to edit the code, and =C-c C-c= it manually to get the updates into the results. When I exported the document (C-e l l), it would export in ~1 second (even a >100 page document) and not prompt me about executing code blocks.

  -k

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Eric S Fraga]

On Friday, 20 May 2016 at 17:06, Ken Mankoff wrote:

[...]

> Thanks for clarifying your workflow. That seems to work for a small
> example. But what would you do if you had a manuscript with 100 code
> blocks in it? And in general you're exporting it a lot without
> changing anything in code, just editing the non-code text. But
> occasionally you want to modify a code block (and its results) too?

So, if I understand correctly, you have org-export-babel-evaluate set to
nil?  If I set that, I no longer get asked whether I want anything
evaluated but the (existing) results do get exported.  I just tried it
to confirm.

> In the above scenario, I used to edit the code, and =C-c C-c= it
> manually to get the updates into the results. When I exported the
> document (C-e l l), it would export in ~1 second (even a >100 page
> document) and not prompt me about executing code blocks.

Yes, I do this as well in such a case.

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-775-g3308a5

Re: exporting documents w/ babel results w/o evaluating babel blocks

[John Hendy]

On Fri, May 20, 2016 at 12:11 PM, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> On Friday, 20 May 2016 at 17:06, Ken Mankoff wrote:
>
> [...]
>
>> Thanks for clarifying your workflow. That seems to work for a small
>> example. But what would you do if you had a manuscript with 100 code
>> blocks in it? And in general you're exporting it a lot without
>> changing anything in code, just editing the non-code text. But
>> occasionally you want to modify a code block (and its results) too?
>
> So, if I understand correctly, you have org-export-babel-evaluate set to
> nil?  If I set that, I no longer get asked whether I want anything
> evaluated but the (existing) results do get exported.  I just tried it
> to confirm.

Using this block,

#+begin_src R :exports results
print("hello")
#+end_src

I get the following (with *no* pre-existing #+results block manually
created with C-c C-c):

| *o-e-b-e* | *o-c-b-e* | *buffer*  | *doc*            |
|-----------+-----------+-----------+------------------|
| nil       | t         | nothing   | =print("hello")= |
| t         | t         | nothing   | =hello=          |
| t         | nil       | nothing   | =hello=          |
| nil       | nil       | nothing   | =print("hello")= |
|-----------+-----------+-----------+------------------|

Variables org-export-babel-evaluate and org-confirm-babel-evaluate
abbreviated above.

I find two things bizarre:
- these options (well, just o-e-b-e it seems) would change what's exported?
- why without C-c C-c which initiates the #+results block, exporting
which causes babel to execute code doesn't insert the results block.
Is this intended or for a reason?

I've not really looked into these variables and just use :eval yes/no
to toggle things. I find it easy enough to create all my blocks with
:eval no, and do a quick change to =yes= as I'm iteratively exporting.
Finally, I can do a replace-string to catch any missed :eval yes's
before a final export.

>
>> In the above scenario, I used to edit the code, and =C-c C-c= it
>> manually to get the updates into the results. When I exported the
>> document (C-e l l), it would export in ~1 second (even a >100 page
>> document) and not prompt me about executing code blocks.
>
> Yes, I do this as well in such a case.
>
> --
> : Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-775-g3308a5

Re: exporting documents w/ babel results w/o evaluating babel blocks

[John Hendy]

On Fri, May 20, 2016 at 11:45 AM, Ken Mankoff <mankoff@gmail.com> wrote:
>
> Eric: You're running something newer (by date) than the commit which changed the behavior, which was:
>
> ec615b1 - Fix `org-export-babel-evaluate' handling (2016-04-28)
>
> But with all the branches in git, I don't know if you have that commit or not. Anyway, I'm not sure why it works for you. It doesn't appear to work for John and I the way it used to. Do you have a global "eval: no" set somewhere?
>
> On 2016-05-20 at 12:14, John Hendy <jw.hendy@gmail.com> wrote:
>> On Fri, May 20, 2016 at 10:57 AM, Ken Mankoff <mankoff@gmail.com> wrote:
>>> As of an Org git commit a few weeks ago, Org exporting (and therefore
>>> Org) has become basically unusable for me. I used to be able to
>>> export code block results without evaluating the block during the
>>> export. I can no longer do this.
>>>
>>
>> Well, you're exporting them at least *once*, right? If not, where are
>> the results coming from?
>
> Yes, but only once. Or sometimes I generate the figure elsewhere (via org-edit-special) and then manually put [[file:fig.pdf]] below the #+RESULTS:.
>
>> I do the same as I'm tweaking plots. Every code block I create has
>> :eval yes initially and once I'm satisfied with the results I just
>> change to :eval no and the generated results (for me, typically a
>> #+results line containing a link to a pdf plot generated by my code
>> block) are still included.
>>
>> Does this help at all? Sorry if I'm not understanding
>
> You understand, and this sort-of helps. I have never used :eval before. Things just didn't evaluate by default, and I like it like that.
>
> ":eval no" fixes the problem at export, but then I can't evaluate the code myself manually when not exporting.

Not in the buffer so as to replace results, but you *can* go into
interactive mode (if python has one) with =C-c '=. I use this
constantly, which allows me to ignore whatever :eval is set to, and
just putz around in the new code block buffer to my heart's content.
I'm approx. 100% R usage, so this still gives me access to the R
pop-up device to view plots or an R buffer to see results. You're
right though, there's no way to update the results block without
toggling :eval, manually running, and then resetting :eval back to no.

John

>
>> Mine is set to t. Interestingly, when I export the above after
>> deleting the results bit, no new results are generated. When I C-c
>> C-c, they are replaced. When I add :eval no, it does't appear to run
>> the code.
>
> Yes same here. This new behavior really sucks.
>
> I'm still trying to find a way where:
>   + code results do export
>   + code does not export
>   + code does not evaluate at export
>   + code can still be evaluated by me
>
> This was the behavior prior to
>
> ec615b1 - Fix `org-export-babel-evaluate' handling (2016-04-28)
>
> But not since.
>
>   -k.

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Ken Mankoff]

On 2016-05-20 at 13:11, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> On Friday, 20 May 2016 at 17:06, Ken Mankoff wrote:
> So, if I understand correctly, you have org-export-babel-evaluate set
> to nil?

yes.

> If I set that, I no longer get asked whether I want anything evaluated
> but the (existing) results do get exported. I just tried it to
> confirm.

Yes results get exported, but so does code. Both, always, even if ":exports none" or ":exports results".

There may have been a misunderstanding, but when I mentioned this in the bug report, I was told this is a "feature".

http://thread.gmane.org/gmane.emacs.orgmode/107230/focus=107231

  -k.

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Ken Mankoff]

Nicolas Goaziou added to To: field, because he committed the change.

Question: Why does "org-export-babel-evaluate" change what is exported? Shouldn't it just change what is evaluated at export time? That is what the doc string implies. A recent commit changed this behavior.

  -k.

On 2016-05-20 at 13:23, John Hendy <jw.hendy@gmail.com> wrote:
> Variables org-export-babel-evaluate and org-confirm-babel-evaluate
> abbreviated above.
>
> I find two things bizarre:
> - these options (well, just o-e-b-e it seems) would change what's exported?

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Nick Dokos]

Ken Mankoff <mankoff@gmail.com> writes:

> On 2016-05-20 at 13:11, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>> On Friday, 20 May 2016 at 17:06, Ken Mankoff wrote:
>> So, if I understand correctly, you have org-export-babel-evaluate set
>> to nil?
>
> yes.
>

Are you getting an evaluation on export? I don't get that with o-e-b-e
set to nil.

>> If I set that, I no longer get asked whether I want anything evaluated
>> but the (existing) results do get exported. I just tried it to
>> confirm.
>
> Yes results get exported, but so does code. Both, always, even if ":exports none" or ":exports results".
>

With ":exports results" and o-e-b-e set to nil, I get no evaluation on
export, but I get both code and results in the output.

> There may have been a misunderstanding, but when I mentioned this in
> the bug report, I was told this is a "feature".
>
> http://thread.gmane.org/gmane.emacs.orgmode/107230/focus=107231
>

Not sure what exactly Chuck meant is a feature, but IIUC, the fact
that I get both code and results even if I specify ":exports results"
looks like a bug to me.

I'm on more-or-less latest master:
Org-mode version 8.3.4 (release_8.3.4-824-ga02fe8 @
/home/nick/elisp/org-mode/lisp/)

--
Nick

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Charles C. Berry]

On Fri, 20 May 2016, Nick Dokos wrote:

> Ken Mankoff <mankoff@gmail.com> writes:
>
[deleted discussion of  `org-export-babel-evaluate' settings]

>
> With ":exports results" and o-e-b-e set to nil, I get no evaluation on
> export, but I get both code and results in the output.
>
>> There may have been a misunderstanding, but when I mentioned this in
>> the bug report, I was told this is a "feature".
>>
>> http://thread.gmane.org/gmane.emacs.orgmode/107230/focus=107231
>>
>
> Not sure what exactly Chuck meant is a feature, but IIUC, the fact
> that I get both code and results even if I specify ":exports results"
> looks like a bug to me.
>

I meant this, as of commit ec615b1..., `org-export-babel-evaluate' set to
`nil' keeps the exporter from running this line

       (org-export-execute-babel-code)

during exports. So, 'no Babel code is run' in the sense that the above
line does not execute.

src-blocks and inline-src-blocks are neither run nor removed, and no
#+results:<etc> or {{{results()}}} are added, removed, or
modified. Babel handles all that. The exporter merely formats those
things once Babel is done.

So the bug, if any, is in the docstring in failing to mention that
everything that babel does is switched off.

Since the behavior that the OP wanted can be had by setting babel
header args, I don't see this as a bug even though the behavior
changed in a way that surprised him.

Chuck

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Ken Mankoff]

On 2016-05-20 at 14:46, Nick Dokos <ndokos@gmail.com> wrote:

> Are you getting an evaluation on export? I don't get that with o-e-b-e
> set to nil.

No I was not, but things were still exporting, which was confusing. I agree, seems like a bug.

> ... the fact that I get both code and results even if I specify
> ":exports results" looks like a bug to me.

Yes.

  -k.

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Nick Dokos]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> On Fri, 20 May 2016, Nick Dokos wrote:
>
>> Ken Mankoff <mankoff@gmail.com> writes:
>>
> [deleted discussion of  `org-export-babel-evaluate' settings]
>
>>
>> With ":exports results" and o-e-b-e set to nil, I get no evaluation on
>> export, but I get both code and results in the output.
>>
>>> There may have been a misunderstanding, but when I mentioned this in
>>> the bug report, I was told this is a "feature".
>>>
>>> http://thread.gmane.org/gmane.emacs.orgmode/107230/focus=107231
>>>
>>
>> Not sure what exactly Chuck meant is a feature, but IIUC, the fact
>> that I get both code and results even if I specify ":exports results"
>> looks like a bug to me.
>>
>
> I meant this, as of commit ec615b1..., `org-export-babel-evaluate' set to
> `nil' keeps the exporter from running this line
>
>       (org-export-execute-babel-code)
>
> during exports. So, 'no Babel code is run' in the sense that the above
> line does not execute.
>
> src-blocks and inline-src-blocks are neither run nor removed, and no
> #+results:<etc> or {{{results()}}} are added, removed, or
> modified. Babel handles all that. The exporter merely formats those
> things once Babel is done.
>
> So the bug, if any, is in the docstring in failing to mention that
> everything that babel does is switched off.
>
> Since the behavior that the OP wanted can be had by setting babel
> header args, I don't see this as a bug even though the behavior
> changed in a way that surprised him.
>

I'm not sure I followed the whole discussion so maybe I'm missing
something. Setting o-e-b-e to nil does what it says it does: it does not
evaluate the babel block on export - that's as it should be AFAICT.

But that still leaves the problem that ":exports results" is not obeyed:
both code and results are exported - that's a bug, no?

--
Nick

Re: exporting documents w/ babel results w/o evaluating babel blocks

[John Hendy]

On Fri, May 20, 2016 at 4:20 PM, Charles C. Berry <ccberry@ucsd.edu> wrote:
> On Fri, 20 May 2016, Nick Dokos wrote:
>
>> Ken Mankoff <mankoff@gmail.com> writes:
>>
> [deleted discussion of  `org-export-babel-evaluate' settings]
>
>>
>> With ":exports results" and o-e-b-e set to nil, I get no evaluation on
>> export, but I get both code and results in the output.
>>
>>> There may have been a misunderstanding, but when I mentioned this in
>>> the bug report, I was told this is a "feature".
>>>
>>> http://thread.gmane.org/gmane.emacs.orgmode/107230/focus=107231
>>>
>>
>> Not sure what exactly Chuck meant is a feature, but IIUC, the fact
>> that I get both code and results even if I specify ":exports results"
>> looks like a bug to me.
>>
>
> I meant this, as of commit ec615b1..., `org-export-babel-evaluate' set to
> `nil' keeps the exporter from running this line
>
>       (org-export-execute-babel-code)
>
> during exports. So, 'no Babel code is run' in the sense that the above
> line does not execute.
>
> src-blocks and inline-src-blocks are neither run nor removed, and no
> #+results:<etc> or {{{results()}}} are added, removed, or
> modified. Babel handles all that. The exporter merely formats those
> things once Babel is done.
>
> So the bug, if any, is in the docstring in failing to mention that
> everything that babel does is switched off.

I don't entirely follow. Is there an intermediate step between a .org
src block, babel, and the exporter? If so, the behavior I see could
make sense. If not, then it doesn't.

As in, why is babel "execution" (as in o-e-b-e to non-nil) required to
*not* export this code into the resultant pdf?

#+begin_src R :exports results :results output
  dat <- 1:10
  print(dat)
#+end_src

With o-e-b-e set to nil, I get the code in the exported document,
hence my asking. If I turn o-e-b-e on, I don't (but get the results).
So perhaps the real missing bit from the documentation is something
like "all babel functionality is turned off, and babel is responsible
for executing code and generating/updating any results, as well as
controlling src block export controls (results vs. code vs. both). In
other words, without babel, src blocks in .org files are treated like
example blocks during export."

Is that accurate?

I think one of the main points of confusion in this thread still has
to do with why o-e-b-e switches what gets exported. If no code/results
are run/added/removed/modified as you mention, it doesn't speak to why
code export is always done for o-e-b-e = nil, but not otherwise. This
suggests babel is doing more than "executing code;" it's somehow
influencing what makes it into the export document. At least for me,
that was an unexpected interpretation of "everything babel does is
switched off."

In fact, I just had an idea... I have o-e-b-e set to always, but just
turned org-confirm-babel-evaluate to on (I typically have it off). If
export the above and type "no" to make sure the code block is not
executed I do *not* get the code in the document. I think this
confirms that babel's responsibilities include *what* plain text in
.org the exporter formats into the output format.

My suggestion would be to create a different variable to handle
formatting vs. execution unless things were designed this way? It
seems from all the comments that the majority of users expect the
time-heavy *code* not to be run at each export but still prefer the
blocks to behave as they intended. Theoretically, with all of my
results generated (e.g. into .pdf, .png, etc. plots; ascii tables,
whatever) shouldn't I be able to have o-e-b-e set to whatever and get
the same document? My document no longer depends on any of the code...
so I don't need it run, but the exported document will behave like
:exports both instead of :exports results just from turning babel off.


John

>
> Since the behavior that the OP wanted can be had by setting babel
> header args, I don't see this as a bug even though the behavior
> changed in a way that surprised him.
>
> Chuck
>

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Charles C. Berry]

John and Nick,

Comments in line below.

TL;DR: Org babel headers give excellent control over what gets run,
when it gets run, and how. Users should use them. Don't reset
`org-export-babel-evaluate'.

Chuck

On Sun, 22 May 2016, John Hendy wrote:

> On Fri, May 20, 2016 at 4:20 PM, Charles C. Berry <ccberry@ucsd.edu> wrote:
>> On Fri, 20 May 2016, Nick Dokos wrote:
>>
>>> Ken Mankoff <mankoff@gmail.com> writes:
>>>
>> [deleted discussion of  `org-export-babel-evaluate' settings]
>>
>>>
>>> With ":exports results" and o-e-b-e set to nil, I get no evaluation on
>>> export, but I get both code and results in the output.
>>>
>>>> There may have been a misunderstanding, but when I mentioned this in
>>>> the bug report, I was told this is a "feature".
>>>>
>>>> http://thread.gmane.org/gmane.emacs.orgmode/107230/focus=107231
>>>>
>>>
>>> Not sure what exactly Chuck meant is a feature, but IIUC, the fact
>>> that I get both code and results even if I specify ":exports results"
>>> looks like a bug to me.
>>>
>>
>> I meant this, as of commit ec615b1..., `org-export-babel-evaluate' set to
>> `nil' keeps the exporter from running this line
>>
>>       (org-export-execute-babel-code)
>>
>> during exports. So, 'no Babel code is run' in the sense that the above
>> line does not execute.
>>
>> src-blocks and inline-src-blocks are neither run nor removed, and no
>> #+results:<etc> or {{{results()}}} are added, removed, or
>> modified. Babel handles all that. The exporter merely formats those
>> things once Babel is done.
>>
>> So the bug, if any, is in the docstring in failing to mention that
>> everything that babel does is switched off.
>
> I don't entirely follow. Is there an intermediate step between a .org
> src block, babel, and the exporter? If so, the behavior I see could
> make sense. If not, then it doesn't.
>
> As in, why is babel "execution" (as in o-e-b-e to non-nil) required to
> *not* export this code into the resultant pdf?
>
> #+begin_src R :exports results :results output
>  dat <- 1:10
>  print(dat)
> #+end_src
>
> With o-e-b-e set to nil, I get the code in the exported document,
> hence my asking. If I turn o-e-b-e on, I don't (but get the results).
> So perhaps the real missing bit from the documentation is something
> like "all babel functionality is turned off, and babel is responsible
> for executing code and generating/updating any results, as well as
> controlling src block export controls (results vs. code vs. both). In
> other words, without babel, src blocks in .org files are treated like
> example blocks during export."
>
> Is that accurate?
>

Yes. That is the point. Babel does it all.

(Caveat: 'like example blocks' is not precisely true, since the
exporter calls a src-block transcoder for the formatting.)

> I think one of the main points of confusion in this thread still has
> to do with why o-e-b-e switches what gets exported. If no code/results
> are run/added/removed/modified as you mention, it doesn't speak to why
> code export is always done for o-e-b-e = nil, but not otherwise. This
> suggests babel is doing more than "executing code;" it's somehow
> influencing what makes it into the export document. At least for me,
> that was an unexpected interpretation of "everything babel does is
> switched off."

Babel is a fairly complicated beast.

If you want to grok what it is doing during export, you could instrument 
`org-export-as' (do `(info "(elisp) Edebug")' if you need to know about 
this), export something with babel code in it, and step thru to just 
before the line `(org-export-execute-babel-code)'. At that point have a 
look at the temp buffer the exporter is working in. (IIRC, it will have 
<2> suffixed to the name of the buffer from which you started the export.) 
Then do one more step to run the babel code and look again - be sure it is 
*still* the buffer named with the *<2> format that you are inspecting now. 
You will see what it is that Babel has done.

In particular, a src block with no results initially, but with
`:exports results' as a header will be gone, but results will now be
seen.

A src block that is has `#+begin-src emacs-lisp :exports code' will
survive, but the header arg will be gone. So, transcoders in the
exporter will not see the header arg.

>
> In fact, I just had an idea... I have o-e-b-e set to always, but just
> turned org-confirm-babel-evaluate to on (I typically have it off). If
> export the above and type "no" to make sure the code block is not
> executed I do *not* get the code in the document. I think this
> confirms that babel's responsibilities include *what* plain text in
> .org the exporter formats into the output format.
>
> My suggestion would be to create a different variable to handle
> formatting vs. execution unless things were designed this way? It
> seems from all the comments that the majority of users expect the
> time-heavy *code* not to be run at each export but still prefer the
> blocks to behave as they intended.

This is what `:eval never-export' was created for.

> Theoretically, with all of my
> results generated (e.g. into .pdf, .png, etc. plots; ascii tables,
> whatever) shouldn't I be able to have o-e-b-e set to whatever and get
> the same document? My document no longer depends on any of the code...
> so I don't need it run, but the exported document will behave like
> :exports both instead of :exports results just from turning babel off.
>

It sounds simple, but babel has a lot woven into it and getting the 
behavior you suggest by setting variables like o-e-b-e in emacs without 
breaking things is tricky. Consider what happens when you have something 
like:

#+BEGIN_SRC emacs-lisp :var x=abc() :noweb yes :exports both
<<format-some-code(y=x)>>
#+END_SRC

With o-e-b-e nil, what should happen? If nothing runs, x does not get set 
because abc() does not run, nor does the noweb code substitution take 
place because that requires the `format-some-code' src-block to run. If 
only some things `ought to' be run, how does the program decide which 
ones?

Currently, the user can fine tune what is run and what is not during
export using Babel headers. Let babel run and the headers will sort
out the rest.

Frankly, I do not see why users would want to mess with o-e-b-e except
in unusual cases - like debugging a complicated document, where having
Babel changing things around could just make things harder.

If there is a use case for a capability that is not well supported by
existing headers it would be good to have an example.


[snip]

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Nick Dokos]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> TL;DR: Org babel headers give excellent control over what gets run,
> when it gets run, and how. Users should use them. Don't reset
> `org-export-babel-evaluate'.
>
> ...
>
>> I think one of the main points of confusion in this thread still has
>> to do with why o-e-b-e switches what gets exported. If no code/results
>> are run/added/removed/modified as you mention, it doesn't speak to why
>> code export is always done for o-e-b-e = nil, but not otherwise. This
>> suggests babel is doing more than "executing code;" it's somehow
>> influencing what makes it into the export document. At least for me,
>> that was an unexpected interpretation of "everything babel does is
>> switched off."
>
> Babel is a fairly complicated beast.
>

We can all agree on that :-)

> If you want to grok what it is doing during export, you could instrument 
> `org-export-as' (do `(info "(elisp) Edebug")' if you need to know about 
> this), export something with babel code in it, and step thru to just 
> before the line `(org-export-execute-babel-code)'. At that point have a 
> look at the temp buffer the exporter is working in. (IIRC, it will have 
> <2> suffixed to the name of the buffer from which you started the export.) 
> Then do one more step to run the babel code and look again - be sure it is 
> *still* the buffer named with the *<2> format that you are inspecting now. 
> You will see what it is that Babel has done.
>
> In particular, a src block with no results initially, but with
> `:exports results' as a header will be gone, but results will now be
> seen.
>

I see.

> A src block that is has `#+begin-src emacs-lisp :exports code' will
> survive, but the header arg will be gone. So, transcoders in the
> exporter will not see the header arg.
>

I see.

>>
>> In fact, I just had an idea... I have o-e-b-e set to always, but just
>> turned org-confirm-babel-evaluate to on (I typically have it off). If
>> export the above and type "no" to make sure the code block is not
>> executed I do *not* get the code in the document. I think this
>> confirms that babel's responsibilities include *what* plain text in
>> .org the exporter formats into the output format.
>>
>> My suggestion would be to create a different variable to handle
>> formatting vs. execution unless things were designed this way? It
>> seems from all the comments that the majority of users expect the
>> time-heavy *code* not to be run at each export but still prefer the
>> blocks to behave as they intended.
>
> This is what `:eval never-export' was created for.
>
>> Theoretically, with all of my
>> results generated (e.g. into .pdf, .png, etc. plots; ascii tables,
>> whatever) shouldn't I be able to have o-e-b-e set to whatever and get
>> the same document? My document no longer depends on any of the code...
>> so I don't need it run, but the exported document will behave like
>> :exports both instead of :exports results just from turning babel off.
>>
>
> It sounds simple, but babel has a lot woven into it and getting the 
> behavior you suggest by setting variables like o-e-b-e in emacs without 
> breaking things is tricky. Consider what happens when you have something 
> like:
>
> #+BEGIN_SRC emacs-lisp :var x=abc() :noweb yes :exports both
> <<format-some-code(y=x)>>
> #+END_SRC
>
> With o-e-b-e nil, what should happen? If nothing runs, x does not get set 
> because abc() does not run, nor does the noweb code substitution take 
> place because that requires the `format-some-code' src-block to run. If 
> only some things `ought to' be run, how does the program decide which 
> ones?
>
> Currently, the user can fine tune what is run and what is not during
> export using Babel headers. Let babel run and the headers will sort
> out the rest.

Thanks for the explanation: it certainly helped me understand somewhat
better what is happening. This is a fairly complicated set of
interactions however, and it *is* confusing: in the long run, babel
should be simplified; in particular, settings should be as orthogonal as
possible (global as well as header setting, although I realize that's
easier said than done - and backward compatibility might have to go
by the wayside in order for this to be accomplished).

>
> Frankly, I do not see why users would want to mess with o-e-b-e except
> in unusual cases - like debugging a complicated document, where having
> Babel changing things around could just make things harder.
>

OK - but then users need guidance that says "don't do that!". Can we add a
big warning to the doc for o-e-b-e?

> If there is a use case for a capability that is not well supported by
> existing headers it would be good to have an example.
>

There's the use case that John describes: I've evaluated everything,
checked everything, I'm ready for export, I don't want babel to touch
the results - and I have a million blocks, so I'd rather have a global
setting than go in with individual headers.

If you've covered this in a previous reply, please ignore me: I've only
paid intermittent attention to the thread, so apologies for missing a
big chunk of what has gone on before in the thread.

In any case, although I'm not happy about the state of things, I
understand better why they are as they are.

Thanks!
--
Nick

Re: exporting documents w/ babel results w/o evaluating babel blocks

[John Hendy]

On Mon, May 23, 2016 at 1:27 PM, Nick Dokos <ndokos@gmail.com> wrote:
> "Charles C. Berry" <ccberry@ucsd.edu> writes:

[snip]

>
>> If there is a use case for a capability that is not well supported by
>> existing headers it would be good to have an example.
>>
>
> There's the use case that John describes: I've evaluated everything,
> checked everything, I'm ready for export, I don't want babel to touch
> the results - and I have a million blocks, so I'd rather have a global
> setting than go in with individual headers.
>
> If you've covered this in a previous reply, please ignore me: I've only
> paid intermittent attention to the thread, so apologies for missing a
> big chunk of what has gone on before in the thread.
>

I actually have never used o-e-b-e prior to this thread. I just dove
in as I was curious about it. My solution is using :eval yes/no as
desired. I often fiddle with blocks one by one, and when I'm done and
want to export the whole doc, my "file-wide" solution is simply:

M-x replace-string RET :eval yes RET :eval no

Works great for me. It can be a little tedious to find a mistake and
quick change no -> yes just to C-c C-c on it and then turn yes -> no,
but it works well enough. I can't see toggling a system-wide variable
related to babel execution as being any less cumbersome, actually.

After all the hoopla about o-e-b-e being a bad idea, and learning that
it doesn't just handle eval but formatting, too, I wonder what the
purpose of the variable *is* for?


John

> In any case, although I'm not happy about the state of things, I
> understand better why they are as they are.
>
> Thanks!
> --
> Nick
>
>

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Charles C. Berry]

On Mon, 23 May 2016, Nick Dokos wrote:

> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> TL;DR: Org babel headers give excellent control over what gets run,
>> when it gets run, and how. Users should use them. Don't reset
>> `org-export-babel-evaluate'.
>>
>> ...
>>
[snip]

>>
>> Frankly, I do not see why users would want to mess with o-e-b-e except
>> in unusual cases - like debugging a complicated document, where having
>> Babel changing things around could just make things harder.
>>
>
> OK - but then users need guidance that says "don't do that!". Can we add a
> big warning to the doc for o-e-b-e?


Seems like a good idea.

>
>> If there is a use case for a capability that is not well supported by
>> existing headers it would be good to have an example.
>>
>
> There's the use case that John describes: I've evaluated everything,
> checked everything, I'm ready for export, I don't want babel to touch
> the results - and I have a million blocks, so I'd rather have a global
> setting than go in with individual headers.
>
> If you've covered this in a previous reply, please ignore me: I've only
> paid intermittent attention to the thread, so apologies for missing a
> big chunk of what has gone on before in the thread.
>

Kinda covered it before, but just to elaborate. This line

#+PROPERTY: header-args :eval never-export

will allow interactive execution via the `org-babel-execute-*' functions. 
So you can redo one src block, all src blocks in a subtree, or all src 
blocks in the buffer as you would with `:eval yes'. But on export, it acts 
like `:eval no'.

And of course, the global setting can be overridden by the headers on
individual src blocks, so finer control is possible.

Chuck

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Grant Rettke]

On Sun, May 22, 2016 at 4:52 PM, Charles C. Berry <ccberry@ucsd.edu> wrote:
> TL;DR: Org babel headers give excellent control over what gets run,
> when it gets run, and how. Users should use them.

Definitely!

> Don't reset `org-export-babel-evaluate'.

Why not?

It can be nice to disable org-export-babel-evaluate by setting it to `nil'.

That way you separate your workflow into two distinct steps: one for
execution and one for weaving.

It would be fun to "see" everyone's Org-Mode workflows; I bet there
are a lot of nice approaches.

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Andreas Kiermeier]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

I second that.
I like exporting everything to LaTeX without having to re-run all the code,
which in many cases can add considerable time.
Cheers,
Andreas

On 24 May 2016 at 11:04, Grant Rettke <gcr@wisdomandwonder.com> wrote:

> On Sun, May 22, 2016 at 4:52 PM, Charles C. Berry <ccberry@ucsd.edu>
> wrote:
> > TL;DR: Org babel headers give excellent control over what gets run,
> > when it gets run, and how. Users should use them.
>
> Definitely!
>
> > Don't reset `org-export-babel-evaluate'.
>
> Why not?
>
> It can be nice to disable org-export-babel-evaluate by setting it to `nil'.
>
> That way you separate your workflow into two distinct steps: one for
> execution and one for weaving.
>
> It would be fun to "see" everyone's Org-Mode workflows; I bet there
> are a lot of nice approaches.
>
>

[-- Attachment #2: Type: text/html, Size: 1618 bytes --]

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Ista Zahn]

On Tue, May 24, 2016 at 6:17 AM, Andreas Kiermeier
<andreas.kiermeier@gmail.com> wrote:
> I second that.
> I like exporting everything to LaTeX without having to re-run all the code,
> which in many cases can add considerable time.

Which you can still do, I guess by

(setq org-babel-default-header-args
      (cons '(:eval . "never-export")
   (assq-delete-all :noweb org-babel-default-header-args)))

instead of the old way

(setq org-export-babel-evaluate nil)

I'm not saying this is good or right. IMO the old behavior of
org-export-babel-evaluate made sense and was useful; the new behavior
is surprising and I have a hard time seeing how it is useful. One can
argue (as Chuck has) that the new behavior is fine sense we have
another way of achieving the desired results; that may be, but I have
yet to see an explanation of why the new behavior is desirable.

It this change is not going to be reversed than the doc string for
org-export-babel-evaluate needs to be updated, and something should go
in the NEWS file warning people (like me) who have been relying on the
old behavior.

Best,
Ista

> Cheers,
> Andreas
>
> On 24 May 2016 at 11:04, Grant Rettke <gcr@wisdomandwonder.com> wrote:
>>
>> On Sun, May 22, 2016 at 4:52 PM, Charles C. Berry <ccberry@ucsd.edu>
>> wrote:
>> > TL;DR: Org babel headers give excellent control over what gets run,
>> > when it gets run, and how. Users should use them.
>>
>> Definitely!
>>
>> > Don't reset `org-export-babel-evaluate'.
>>
>> Why not?
>>
>> It can be nice to disable org-export-babel-evaluate by setting it to
>> `nil'.
>>
>> That way you separate your workflow into two distinct steps: one for
>> execution and one for weaving.
>>
>> It would be fun to "see" everyone's Org-Mode workflows; I bet there
>> are a lot of nice approaches.
>>
>

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Anthony Cowley]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

On Tuesday, May 24, 2016, Ista Zahn <istazahn@gmail.com> wrote:

> On Tue, May 24, 2016 at 6:17 AM, Andreas Kiermeier
> <andreas.kiermeier@gmail.com <javascript:;>> wrote:
>
> It this change is not going to be reversed than the doc string for
> org-export-babel-evaluate needs to be updated, and something should go
> in the NEWS file warning people (like me) who have been relying on the
> old behavior.
>
> Best,
> Ista


I, too, used the workflow of toggling oebe. Having a default header
argument to control evaluation is fine as an alternative, but I don’t see
the point in the change either. In any case, it seems like this is a common
issue and deserving of fairly prominent documentation.

Anthony

[-- Attachment #2: Type: text/html, Size: 1078 bytes --]

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Charles C. Berry]

On Tue, 24 May 2016, Ista Zahn wrote:

> On Tue, May 24, 2016 at 6:17 AM, Andreas Kiermeier
> <andreas.kiermeier@gmail.com> wrote:
>> I second that.
>> I like exporting everything to LaTeX without having to re-run all the code,
>> which in many cases can add considerable time.
>
> Which you can still do, I guess by
>
> (setq org-babel-default-header-args
>      (cons '(:eval . "never-export")
>   (assq-delete-all :noweb org-babel-default-header-args)))
>
> instead of the old way
>
> (setq org-export-babel-evaluate nil)
>
> I'm not saying this is good or right. IMO the old behavior of
> org-export-babel-evaluate made sense and was useful; the new behavior
> is surprising and I have a hard time seeing how it is useful. One can
> argue (as Chuck has) that the new behavior is fine sense we have
> another way of achieving the desired results; that may be, but I have
> yet to see an explanation of why the new behavior is desirable.

The change was in this commit

   commit ec615b192d703a0201ceefd46897e4636ff00a38
   Author: Nicolas Goaziou <mail@nicolasgoaziou.fr>
   Date:   Thu Apr 28 17:25:31 2016 +0200

and was in response to a bug noted in

http://permalink.gmane.org/gmane.emacs.orgmode/106767

IIRC, you need to read the thread - not just the first post - to get what 
gives.

>
> It this change is not going to be reversed than the doc string for
> org-export-babel-evaluate needs to be updated, and something should go
> in the NEWS file warning people (like me) who have been relying on the
> old behavior.

Since the change was a bugfix, it seems unlikely that it will be reversed 
unless another bugfix is proposed.

I could change the docstring, but I wanted to let the discussion ripen a 
bit first to see if anyone stepped forward with a patch that would do just 
that. Or even suggest that the variable be deleted entirely.

For those who are just now tuning in, something like

#+PROPERTY: :eval never-export

should give the behavior most users of o-e-b-e nil would desire.

Chuck

Re: exporting documents w/ babel results w/o evaluating babel blocks

[Charles C. Berry]

Correction:

#+PROPERTY: header-args :eval never-export

is the right incantation below.

Chuck

On Tue, 24 May 2016, Charles C. Berry wrote:

> On Tue, 24 May 2016, Ista Zahn wrote:
>
>> On Tue, May 24, 2016 at 6:17 AM, Andreas Kiermeier
>> <andreas.kiermeier@gmail.com> wrote:
>>> I second that.
>>> I like exporting everything to LaTeX without having to re-run all the 
>>> code,
>>> which in many cases can add considerable time.
>> 
>> Which you can still do, I guess by
>> 
>> (setq org-babel-default-header-args
>>      (cons '(:eval . "never-export")
>>   (assq-delete-all :noweb org-babel-default-header-args)))
>> 
>> instead of the old way
>> 
>> (setq org-export-babel-evaluate nil)
>> 
>> I'm not saying this is good or right. IMO the old behavior of
>> org-export-babel-evaluate made sense and was useful; the new behavior
>> is surprising and I have a hard time seeing how it is useful. One can
>> argue (as Chuck has) that the new behavior is fine sense we have
>> another way of achieving the desired results; that may be, but I have
>> yet to see an explanation of why the new behavior is desirable.
>
> The change was in this commit
>
>  commit ec615b192d703a0201ceefd46897e4636ff00a38
>  Author: Nicolas Goaziou <mail@nicolasgoaziou.fr>
>  Date:   Thu Apr 28 17:25:31 2016 +0200
>
> and was in response to a bug noted in
>
> http://permalink.gmane.org/gmane.emacs.orgmode/106767
>
> IIRC, you need to read the thread - not just the first post - to get what 
> gives.
>
>> 
>> It this change is not going to be reversed than the doc string for
>> org-export-babel-evaluate needs to be updated, and something should go
>> in the NEWS file warning people (like me) who have been relying on the
>> old behavior.
>
> Since the change was a bugfix, it seems unlikely that it will be reversed 
> unless another bugfix is proposed.
>
> I could change the docstring, but I wanted to let the discussion ripen a bit 
> first to see if anyone stepped forward with a patch that would do just that. 
> Or even suggest that the variable be deleted entirely.
>
> For those who are just now tuning in, something like
>
> #+PROPERTY: :eval never-export
>
> should give the behavior most users of o-e-b-e nil would desire.
>
> Chuck
>
>

Charles C. Berry                 Dept of Family Medicine & Public Health
cberry at ucsd edu               UC San Diego / La Jolla, CA 92093-0901
http://famprevmed.ucsd.edu/faculty/cberry/