[babel] exports, caching, remote execution

[babel] exports, caching, remote execution

[Austin Frank]

[-- Attachment #1.1: Type: text/plain, Size: 1081 bytes --]

Hey all--

Two (hopefully quick) questions:

1) Does the exporter respect the :cache argument?  When I evaluate a
   buffer, I can tell that cached blocks are not re-run, as expected.
   When I export to \LaTeX or PDF, it seems that all blocks in the file
   are re-run.  Is there a way to force the exporter to respect caching?

2) Is there a way to generate graphical output on a remote host and
   still have it included in export?  A block like

   #+BEGIN_SRC R :exports results :dir /user@example.com: :file test.png
   plot(my-object)
   #+END_SRC
   
   Gives me
   
   #+results:
   [[file:/scpc:user@example.com:/users/home/user/test.png]]

   When I export to latex or pdf, the graphic isn't included because the
   file for export is local but the graphic is remote.  Is there a way
   to include remotely generated graphics in a locally exported file?
   Something like downloading via scp before exporting?


Thanks for any help,
/au
   
-- 
Austin Frank
http://aufrank.net
GPG Public Key (D7398C2F): http://aufrank.net/personal.asc

[-- Attachment #1.2: Type: application/pgp-signature, Size: 194 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] exports, caching, remote execution

[Eric Schulte]

Hi Austin,

Austin Frank <austin.frank@gmail.com> writes:

> Hey all--
>
> Two (hopefully quick) questions:
>
> 1) Does the exporter respect the :cache argument?  When I evaluate a
>    buffer, I can tell that cached blocks are not re-run, as expected.
>    When I export to \LaTeX or PDF, it seems that all blocks in the file
>    are re-run.  Is there a way to force the exporter to respect caching?
>

I believe the exporter does respect caching, the following minimal
example worked (i.e. was not re-run) for me on export to html.  Could
you provide an example that demonstrates the problem?

--8<---------------cut here---------------start------------->8---
** cache on export
do we export cached blocks

#+begin_src sh :cache yes :exports results
  date
#+end_src

#+results[06ed73c6d8d022cf9c323d92af885952865add17]:
: Thu Jun 17 07:35:19 PDT 2010
--8<---------------cut here---------------end--------------->8---

>
> 2) Is there a way to generate graphical output on a remote host and
>    still have it included in export?

Not that I'm currently aware of.  Dan has handled most of the remote
execution work so he may know more than me in this regard.

>    A block like
>
>    #+source: rplot
>    #+BEGIN_SRC R :exports results :dir /user@example.com: :file test.png
>    plot(my-object)
>    #+END_SRC
>    
>    Gives me
>    
>    #+results:
>    [[file:/scpc:user@example.com:/users/home/user/test.png]]
>

Is "scpc" in the line above a transport protocol?  Maybe this should be
an org-mode wide features, i.e. the ability to resolve remote file
references with C-c C-o and on export.  Does that sound reasonable, and
would it take care of the need in this particular case?

>
>    When I export to latex or pdf, the graphic isn't included because
>    the file for export is local but the graphic is remote.  Is there a
>    way to include remotely generated graphics in a locally exported
>    file?  Something like downloading via scp before exporting?
>

A block like the following may be an acceptable workaround in the near
term (notice I named your block above).

#+begin_src sh :file test.png :var remote=rplot :exports results
  scp $remote ./test.png
#+end_src

>
>
> Thanks for any help,
> /au

Thanks -- Eric

Re: [babel] exports, caching, remote execution

[Austin Frank]

[-- Attachment #1.1: Type: text/plain, Size: 1673 bytes --]

Eric--

Thanks for the reply.

On Thu, Jun 17 2010, Eric Schulte wrote:

> I believe the exporter does respect caching, the following minimal
> example worked (i.e. was not re-run) for me on export to html.  Could
> you provide an example that demonstrates the problem?

Yes, I'll keep an eye on this and try to narrow it down to an example if
I can.

> Not that I'm currently aware of.  Dan has handled most of the remote
> execution work so he may know more than me in this regard.

And thanks to him for that!  I tipped him off to using TRAMP with ESS,
and he took it and ran further than I ever hoped!

> Is "scpc" in the line above a transport protocol?

Yes, it is scp over an existing ssh connection.  The following org-style
link goes to the relevant info page.

[[info:tramp:External%20methods][info:tramp:External methods]]

> Maybe this should be an org-mode wide features, i.e. the ability to
> resolve remote file references with C-c C-o and on export.  Does that
> sound reasonable, and would it take care of the need in this
> particular case?

C-c C-o actually works!  I was extremely encouraged to see this, and it
got my hopes up for export =)

If remote file references could be resolved during export, that would be
ideal.

> A block like the following may be an acceptable workaround in the near
> term (notice I named your block above).
>
> #+begin_src sh :file test.png :var remote=rplot :exports results
>   scp $remote ./test.png
> #+end_src

Nice suggestions, I'll try it later today.

Thanks,
/au

-- 
Austin Frank
http://aufrank.net
GPG Public Key (D7398C2F): http://aufrank.net/personal.asc

[-- Attachment #1.2: Type: application/pgp-signature, Size: 194 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] exports, caching, remote execution

[David Maus]

[-- Attachment #1.1: Type: text/plain, Size: 4235 bytes --]

Eric Schulte wrote:

>Is "scpc" in the line above a transport protocol?  Maybe this should be
>an org-mode wide features, i.e. the ability to resolve remote file
>references with C-c C-o and on export.  Does that sound reasonable, and
>would it take care of the need in this particular case?

I've created two IDEA issue in the issue file for this and another
operation on links especially during export: Link resolving and link
dereferencing.

Just out of my head without many thoughs about how to implement:

1.

Following the URI specs (RFC 3986) resolving and dereferencing means:

"the process of determining an access mechanism and the appropriate
parameters necessary to dereference a URI (sc. resolving, D.M.); this
resolution may require several iterations.  To use that access
mechanism to perform an action on the URI's resource is to
"dereference" the URI." (1.2.2)

The basic idea is, that an Org mode link is a URI identifying a
resource; upon export it might be necessary or desired to provide
either an access mechanism suitable for the output format to access
the entity referenced by the URI or another URI (see below).

E.g.: An info reference to the Org mode manual [[info:org]] could be
resolved to a web link [[http://orgmode.org/manual/]] when exporting
to html.

Resolving is already done for files when exporting to html:
[[file:foo.baz]] is resolved to a relative web link, pointing to
foo.baz.

Dereferencing a link like

[[file:/scpc:user@example.com:/users/home/user/test.png]]

could or would hence be part of resolving a file link.  It fetches the
resource (test.png), puts it into current directory and returns a link
suitable for the export context.

Another example would be a hypothetical Org mode link type "rfc" that
opens and stores links to RFCs using rfcview-mode.

When exporting an Org mode document to html, the function
`org-rfc-resolve-link' is called and returns a web link pointing to a
web resource of the RFC.  When exporting to latex, the function
`org-rfc-resolve-link' returns a link of type "bib" with the BibTeX
key of the RFC (e.g. rfc:4287 -> bib:RFC4287), assuming that a
reference to a RFC in a LaTeX document is a bibliographic reference.

2.

To complete this way of link handling we require a third function:
Link markup.  An Org mode link handler may provide a markup-link
function that takes the exporting context and the link and returns
suitable markup of LINK in CONTEXT.

E.g. the handler for BibTeX links provides the function
`org-bib-markup-link' that returns proper markup of a bibliographic
reference.  In context of LaTeX export his would be a \cite command
with the BibTeX key of the bib link (bib:RFC4287 -> \cite{4287}).

3.

For the sake of Lispness and to avoid repeated parsing, we define a
Lisp representation of an Org link to be:

((TYPE AI RESOURCE) . PROPS)

Where

  - TYPE :: Is a symbol of the type ("scheme") of the link

  - RESOURCE :: Is the path or name of the referenced resource

  - AI :: Is t, if RESOURCE contains an authority, otherwise it is nil.

          Cf. RFC3986, 3.2; in the written representation of a URI the
          presence of an authority is indicated by the two slashes
          after the colon.

  - PROPS :: Is a property list of additional link properties
             (e.g. description)

Examples:

  - [[http://orgmode.org/manual/][Org mode manual]] is represented as:

    ((http t "orgmode.org/manual/") . (:description "Org mode manual"))

  - [[info:org][Org mode manual]] is represented as

    ((info nil "org") . (:description "Org mode manual"))

  - [[file:foo/baz.org]] is represented as

    ((file nil "foo/baz.org"))

Having this, resolving a link is modifying TYPE, RESOURCE and AI.

E.g. ((file nil "foo/baz.org")) -> ((http nil "foo/baz.html"))

     ((file nil "/scp:user@example.com:/users/home/user/test.png"))

     -> ((file nil "test.png")) [dereferenced]

     -> ((http nil "test.png")) [finally resolved]

4.

The question of how to actually resolve a link is up to the respective
link handler and I am not sure how to provide easy customization of
such a system.

Comments are welcome,

 -- David


--
OpenPGP... 0x99ADB83B5A4478E6
Jabber.... dmjena@jabber.org
Email..... dmaus@ictsoc.de

[-- Attachment #1.2: Type: application/pgp-signature, Size: 230 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [babel] exports, caching, remote execution

[Tom Short]

Eric Schulte <schulte.eric <at> gmail.com> writes:

> 
> Hi Austin,
> 
> Austin Frank <austin.frank <at> gmail.com> writes:
> 
> > Hey all--
> >
> > Two (hopefully quick) questions:
> >
> > 1) Does the exporter respect the :cache argument?  When I evaluate a
> >    buffer, I can tell that cached blocks are not re-run, as expected.
> >    When I export to \LaTeX or PDF, it seems that all blocks in the file
> >    are re-run.  Is there a way to force the exporter to respect caching?
> >
> 
> I believe the exporter does respect caching, the following minimal
> example worked (i.e. was not re-run) for me on export to html.  Could
> you provide an example that demonstrates the problem?

Eric, here's an example where the exporter does not respect caching. - Tom

#+BABEL: :session *R* :results output :exports both :cache yes

* A test of caching

#+begin_src R 
  cat("random result:", runif(1), "\n")
  Sys.sleep(2)
  alarm()
#+end_src 

#+results[b2549fac8a1ec2923ae289d47ce55fb2853dd1de]:
: random result: 0.2799064

#+begin_src R 
  cat("random result:", runif(1), "\n")
  Sys.sleep(2)
  alarm()
#+end_src 

#+results[b2549fac8a1ec2923ae289d47ce55fb2853dd1de]:
: random result: 0.1625634

** cache on export
do we export cached blocks

#+begin_src emacs-lisp :cache yes :exports results
  (random)
#+end_src

#+results[46632b4fe2e3a23e847953c95adcba58c270b381]:
: 490528137

Re: Re: [babel] exports, caching, remote execution

[Eric Schulte]

Hi Tom,

Thanks for pointing this out.  It appears that during export the code
block is evaluated in an altered version of the original buffer in which
(for some reason) Babel is sometimes not able to find properties which
are tucked away in either a #+Babel: line at the top of the file, or as
properties of an enclosing headline level.

This certainly merits a closer investigation, and has been recorded as a
bug.  Most likely it will require picking apart the subtleties of the
actions taken by the exporter.

Thanks -- Eric

Tom Short <tshort@epri.com> writes:

> Eric Schulte <schulte.eric <at> gmail.com> writes:
>
>> 
>> Hi Austin,
>> 
>> Austin Frank <austin.frank <at> gmail.com> writes:
>> 
>> > Hey all--
>> >
>> > Two (hopefully quick) questions:
>> >
>> > 1) Does the exporter respect the :cache argument?  When I evaluate a
>> >    buffer, I can tell that cached blocks are not re-run, as expected.
>> >    When I export to \LaTeX or PDF, it seems that all blocks in the file
>> >    are re-run.  Is there a way to force the exporter to respect caching?
>> >
>> 
>> I believe the exporter does respect caching, the following minimal
>> example worked (i.e. was not re-run) for me on export to html.  Could
>> you provide an example that demonstrates the problem?
>
> Eric, here's an example where the exporter does not respect caching. - Tom
>
> #+BABEL: :session *R* :results output :exports both :cache yes
>
> * A test of caching
>
> #+begin_src R 
>   cat("random result:", runif(1), "\n")
>   Sys.sleep(2)
>   alarm()
> #+end_src 
>
> #+results[b2549fac8a1ec2923ae289d47ce55fb2853dd1de]:
> : random result: 0.2799064
>
> #+begin_src R 
>   cat("random result:", runif(1), "\n")
>   Sys.sleep(2)
>   alarm()
> #+end_src 
>
> #+results[b2549fac8a1ec2923ae289d47ce55fb2853dd1de]:
> : random result: 0.1625634
>
> ** cache on export
> do we export cached blocks
>
> #+begin_src emacs-lisp :cache yes :exports results
>   (random)
> #+end_src
>
> #+results[46632b4fe2e3a23e847953c95adcba58c270b381]:
> : 490528137
>
>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: [babel] exports, caching, remote execution

[Eric Schulte]

Hi Tom,

This turned out to be a fairly complicated issue.

During export of a narrowed region the narrowed region will be coppied
to a temporary buffer, when exporting from the buffer all buffer-wide,
and headline-wide parameters which are not included in the region are
ignored.  This results in a different set of header arguments being used
for interactive evaluation than were used for export evaluation.  These
differences in header arguments then invalidate the cache saved during
the interactive evaluation.

I've just put together and pushed a fix which involves jumping back to
the original org file to collect header arguments in context during
export.  This appears to resolve the issue (at least for me).  Please
let me know if it works for you.

Best -- Eric

Tom Short <tshort@epri.com> writes:

> Eric Schulte <schulte.eric <at> gmail.com> writes:
>
>> 
>> Hi Austin,
>> 
>> Austin Frank <austin.frank <at> gmail.com> writes:
>> 
>> > Hey all--
>> >
>> > Two (hopefully quick) questions:
>> >
>> > 1) Does the exporter respect the :cache argument?  When I evaluate a
>> >    buffer, I can tell that cached blocks are not re-run, as expected.
>> >    When I export to \LaTeX or PDF, it seems that all blocks in the file
>> >    are re-run.  Is there a way to force the exporter to respect caching?
>> >
>> 
>> I believe the exporter does respect caching, the following minimal
>> example worked (i.e. was not re-run) for me on export to html.  Could
>> you provide an example that demonstrates the problem?
>
> Eric, here's an example where the exporter does not respect caching. - Tom
>
> #+BABEL: :session *R* :results output :exports both :cache yes
>
> * A test of caching
>
> #+begin_src R 
>   cat("random result:", runif(1), "\n")
>   Sys.sleep(2)
>   alarm()
> #+end_src 
>
> #+results[b2549fac8a1ec2923ae289d47ce55fb2853dd1de]:
> : random result: 0.2799064
>
> #+begin_src R 
>   cat("random result:", runif(1), "\n")
>   Sys.sleep(2)
>   alarm()
> #+end_src 
>
> #+results[b2549fac8a1ec2923ae289d47ce55fb2853dd1de]:
> : random result: 0.1625634
>
> ** cache on export
> do we export cached blocks
>
> #+begin_src emacs-lisp :cache yes :exports results
>   (random)
> #+end_src
>
> #+results[46632b4fe2e3a23e847953c95adcba58c270b381]:
> : 490528137
>
>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode