[RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Aaron Ecay]

* lisp/ob-core.el (org-babel-generate-file-param): New function.
(org-babel-get-src-block-info): Use it.
---
 lisp/ob-core.el | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

This patch allows the autogeneration of result file names from source
block names.  Thus, instead of:

#+name: one
#+begin_src R :results file graphics :file one.png
  ...
#+end_src

One can write (with the same result, that one.png is generated):

#+name: one
#+begin_src R :results file graphics :file png
  ...
#+end_src

The benefit comes from the reduced duplication of information in the
file.  It also becomes possible to use the usual property inheritance
mechanisms (via #+PROPERTY lines or :PROPERTY: drawers) to specify
result file types for multiple source blocks at once.

This patch also introduces the :output-dir property, which can be used
to redirect all the output from blocks in a (file/subtree) to a subdirectory.

The patch treats any :file args containing a period as before, so
backwards compatibility with old files should in most cases be maintained.

If this patch looks good, I can provide tests and documentation.

Aaron

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1348f04..2defabf 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -282,7 +282,9 @@ Returns a list
 	(setq info (org-babel-parse-inline-src-block-match))))
     ;; resolve variable references and add summary parameters
     (when (and info (not light))
-      (setf (nth 2 info) (org-babel-process-params (nth 2 info))))
+      (setf (nth 2 info) (org-babel-generate-file-param
+			  name
+			  (org-babel-process-params (nth 2 info)))))
     (when info (append info (list name indent head)))))

 (defvar org-babel-exp-reference-buffer nil
@@ -2890,6 +2892,35 @@ For the format of SAFE-LIST, see `org-babel-safe-header-args'."
 		      (member (cdr pair) (cdr entry)))
 		     (t nil)))))))

+(defun org-babel-generate-file-param (src-name params)
+  "Calculate the filename for source block results.
+
+The directory is calculated from the :output-dir property of the
+source block; if not specified, use the current directory.
+
+If the source block has a #+NAME and the :file parameter does not
+contain any period characters, then the :file parameter is
+treated as an extension, and the output file name is the
+concatenation of the directory (as calculated above), the block
+name, a period, and the parameter value as a file extension.
+Otherwise, the :file parameter is treated as a full file name,
+and the output file name is the directory (as calculated above)
+plus the parameter value."
+  (let* ((file-cons (assq :file params))
+	 (file (cdr-safe file-cons))
+	 (dir (or (cdr-safe (assq :output-dir params)) "")))
+    (when (and file (not (string= dir "")))
+      (make-directory dir t)
+      (unless (string-match "/\\'" dir)
+	(setq dir (concat dir "/"))))
+    (cond
+     ((not file) nil)
+     ((or (string-match "\\." file) (not src-name))
+      ;; Either the file name already has an extension, or there is no
+      ;; name for this source block -> we cannot do anything
+      (setcdr file-cons (concat dir file)))
+     (t (setcdr file-cons (concat dir src-name "." file))))
+    params))

 ;;; Used by backends: R, Maxima, Octave.
 (defun org-babel-graphical-output-file (params)
--
1.9.2

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Bastien]

Hi Aaron,

thanks for the patch.

Aaron Ecay <aaronecay@gmail.com> writes:

> #+name: one
> #+begin_src R :results file graphics :file png
>   ...
> #+end_src

What happens when there is :file png with no #+name line?

> The benefit comes from the reduced duplication of information in the
> file.  It also becomes possible to use the usual property inheritance
> mechanisms (via #+PROPERTY lines or :PROPERTY: drawers) to specify
> result file types for multiple source blocks at once.
>
> This patch also introduces the :output-dir property, which can be used
> to redirect all the output from blocks in a (file/subtree) to a
> subdirectory.

Does :output-dir accept absolute or relative paths?  I'm asking
because you speak of "subdirectory", but both should be accepted
IMHO.

> The patch treats any :file args containing a period as before, so
> backwards compatibility with old files should in most cases be
> maintained.

Maybe there are cases where the :file value does not take an extension
but the user still want to write the output to this file?  How would
your patch handle this?

> If this patch looks good, I can provide tests and documentation.

Looks useful to me, but ":file png" looks wrong, with too much
implicit.  We should find something less confusing.

Thanks,

-- 
 Bastien

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Aaron Ecay]

Hi Bastien,

Thanks for your feedback.

2014ko apirilak 22an, Bastien-ek idatzi zuen:

[...]

>> #+name: one
>> #+begin_src R :results file graphics :file png
>> ...
>> #+end_src
>
> What happens when there is :file png with no #+name line?

This case will be treated as before the patch: output will go to the
“png” file.  (The :output-dir property, if present, will apply.)

> Does :output-dir accept absolute or relative paths?  I'm asking
> because you speak of "subdirectory", but both should be accepted
> IMHO.

I agree.  I think the current patch does this as long as :output-dir is
an absolute pathname, but I have not tested that case.  I will

>
> Maybe there are cases where the :file value does not take an extension
> but the user still want to write the output to this file?  How would
> your patch handle this?

At present, it doesn’t.

> Looks useful to me, but ":file png" looks wrong, with too much
> implicit.  We should find something less confusing.

One option would be to use :file-ext instead, to generate a :file
parameter.  I didn’t go this route because autogenerating :file from
other parameters seemed like too much magic.  But your points in the
other direction are good.

How does this sound as an algorithm:
1. if :file is present, behave exactly as we do now
2. if :file is absent but :file-ext and a #+name is present, generate a
   :file parameter from :output-dir, the #+name, and :file-ext.

Open questions:
1. should :file-ext without a #+name be a no-op, or an error?
2. should :output-dir apply to the :file case as well?

--
Aaron Ecay

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Eric Schulte]

>> Looks useful to me, but ":file png" looks wrong, with too much
>> implicit.  We should find something less confusing.
>
> One option would be to use :file-ext instead, to generate a :file
> parameter.  I didn’t go this route because autogenerating :file from
> other parameters seemed like too much magic.  But your points in the
> other direction are good.
>
> How does this sound as an algorithm:
> 1. if :file is present, behave exactly as we do now
> 2. if :file is absent but :file-ext and a #+name is present, generate a
>    :file parameter from :output-dir, the #+name, and :file-ext.
>

This sounds like a great approach to me.

>
> Open questions:
> 1. should :file-ext without a #+name be a no-op, or an error?

One option might be to borrow naming behavior from the comment
functionality in ob-tangle which looks like the following (from line 426
in ob-tangle.el).

(let (...
      (source-name
       (intern (or (nth 4 info)       ; explicit #+name:
                   (format "%s:%d"    ; constructed from header and position
                           (or (ignore-errors (nth 4 (org-heading-components)))
                               "No heading")
                           block-counter))))
      ...))

> 
> 2. should :output-dir apply to the :file case as well?
>

If you mean "should :output-dir be used as the base when :file is a
relative pathname" then I'd say "yes", and I think if this isn't the
current behavior then the current behavior should be changed.

Thanks for this nice patch,
Eric

>
> --
> Aaron Ecay
>

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Bastien]

Hi Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> How does this sound as an algorithm:
> 1. if :file is present, behave exactly as we do now
> 2. if :file is absent but :file-ext and a #+name is present, generate a
>    :file parameter from :output-dir, the #+name, and :file-ext.

I suggest this one:

1. if :file is present, behave exactly as we do now

2. if :file-ext is present:

   - if #+name is present, generate a :file parameter
     from :output-dir, #+name and :file-ext

   - otherwise, generate the :file parameter from
     :output-dir, the headline or the title or the
     current file name and :file-ext

Just falling back on something sensible when :file is absent
and :file-ext is specified.
   
> Open questions:
> 1. should :file-ext without a #+name be a no-op, or an error?

See above.

> 2. should :output-dir apply to the :file case as well?

To me yes.

In overall I think would be good, but I'd like Eric and other
babelist around here to have a look before we commit this.
So perhaps another round of patch testing will be good.

Thanks!

-- 
 Bastien

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Eric Schulte]

Bastien <bzg@gnu.org> writes:

> Hi Aaron,
>
> Aaron Ecay <aaronecay@gmail.com> writes:
>
>> How does this sound as an algorithm:
>> 1. if :file is present, behave exactly as we do now
>> 2. if :file is absent but :file-ext and a #+name is present, generate a
>>    :file parameter from :output-dir, the #+name, and :file-ext.
>
> I suggest this one:
>
> 1. if :file is present, behave exactly as we do now
>
> 2. if :file-ext is present:
>
>    - if #+name is present, generate a :file parameter
>      from :output-dir, #+name and :file-ext
>
>    - otherwise, generate the :file parameter from
>      :output-dir, the headline or the title or the
>      current file name and :file-ext
>
> Just falling back on something sensible when :file is absent
> and :file-ext is specified.
>    

FWIW I'm in full agreement here, see my other email for one sensible
alternative.  (Sorry to split up my reply, the hazards of an
asynch/batch email pull/read/write/push setup.)

>
>> Open questions:
>> 1. should :file-ext without a #+name be a no-op, or an error?
>
> See above.
>
>> 2. should :output-dir apply to the :file case as well?
>
> To me yes.
>
> In overall I think would be good, but I'd like Eric and other
> babelist around here to have a look before we commit this.
> So perhaps another round of patch testing will be good.
>

I agree here.

Thanks again,

>
> Thanks!

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Bastien]

Hi,

Eric Schulte <schulte.eric@gmail.com> writes:

> One option might be to borrow naming behavior from the comment
> functionality in ob-tangle which looks like the following (from line 426
> in ob-tangle.el).

Indeed, it's safer than just using the buffer file name.

Aaron, please go ahead with the patch when you want,
thanks again for it!

-- 
 Bastien

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Achim Gratz]

Aaron Ecay writes:
> How does this sound as an algorithm:
> 1. if :file is present, behave exactly as we do now
> 2. if :file is absent but :file-ext and a #+name is present, generate a
>    :file parameter from :output-dir, the #+name, and :file-ext.

3. If both :file and :file-ext are present, :file takes precedence and
:file-ext is ignored.

> 1. should :file-ext without a #+name be a no-op, or an error?

Since you are hoping to inherit these via properties, this should be a
nop if not applicable.

> 2. should :output-dir apply to the :file case as well?

I'd say no since it seems this would be backwards-incompatible.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]

Hi Eric, Bastien, Achim,

Thanks so much for the feedback.  I’ve adopted the :file-ext approach
suggested by Bastien, leaving the previous default behavior in place for
blocks with a :file argument.

2014ko apirilak 22an, Eric Schulte-ek idatzi zuen:

[...]


> One option might be to borrow naming behavior from the comment
> functionality in ob-tangle which looks like the following (from line 426
> in ob-tangle.el).
>
> (let (...
>       (source-name
>        (intern (or (nth 4 info)       ; explicit #+name:
>                    (format "%s:%d"    ; constructed from header and position
>                            (or (ignore-errors (nth 4 (org-heading-components)))
>                                "No heading")
>                            block-counter))))
>       ...))

I’m not sure I like this approach.  It relies on counting source
blocks, so an addition/deletion of a block could change the index.
I’m worried that this can lead to the accumulation of many output
files: heading:1.ext, heading:2.ext, ... all with no clear indication
of what block they were spawned by.  It would also be possible for
the result links in the buffer to become inconsistent with the actual
block:auto-generated name mapping.

I think I would prefer the code in this patch to do nothing in this case
(not create a :file value), but for language-specific code that needs a
:file to raise an error to prompt the user to add a name.

>
>>
>> 2. should :output-dir apply to the :file case as well?
>>
>
> If you mean "should :output-dir be used as the base when :file is a
> relative pathname" then I'd say "yes", and I think if this isn't the
> current behavior then the current behavior should be changed.

Achim raises a backwards compatibility concern.  I am not sure how
serious it is: the default settings (no :output-dir) are backwards
compatible, and if users set that arg we ought to just give them what
they ask for.

Nonetheless, the new version of the patch conservatively obeys Achim’s
suggestion.  I can change this to your suggestion, if that is the
consensus.

To address a comment from Bastien: :output-dir accepts absolute as well
as relative directory names.  Referring to a “subdirectory” was a
mistake on my part; the docs in the new patch should be clearer.

The updated patch (now with docs and tests) is attached to this email.

Thanks again,

--
Aaron Ecay

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-core.el-allow-the-auto-generation-of-output-file-.patch --]
[-- Type: text/x-diff, Size: 7429 bytes --]

From 4b428820432752117c60b79da0a79fd4e50e4ba1 Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Tue, 22 Apr 2014 15:13:48 -0400
Subject: [PATCH] ob-core.el: allow the auto-generation of output file names
 for src blocks.

* lisp/ob-core.el (org-babel-generate-file-param): New function.
(org-babel-get-src-block-info): Use it.
* testing/lisp/test-ob.el (test-org-babel/file-ext-and-output-dir):
New test.
* doc/org.texi (Specific header arguments): Add doc for :file-ext and
:output-dir header args.
---
 doc/org.texi               | 27 +++++++++++++++++++++++++++
 lisp/ob-core.el            | 34 ++++++++++++++++++++++++++++++++++
 testing/examples/babel.org | 34 ++++++++++++++++++++++++++++++++++
 testing/lisp/test-ob.el    | 14 ++++++++++++++
 4 files changed, 109 insertions(+)

diff --git a/doc/org.texi b/doc/org.texi
index 2546be1..79cc044 100644
--- a/doc/org.texi
+++ b/doc/org.texi
@@ -14406,6 +14406,8 @@ argument in lowercase letters.  The following header arguments are defined:
                                 be collected and handled
 * file::                        Specify a path for file output
 * file-desc::                   Specify a description for file results
+* file-ext::                    Specify an extension for file output
+* output-dir::                  Specify a directory to write file output to
 * dir::                         Specify the default (possibly remote)
                                 directory for code block execution
 * exports::                     Export code and/or results
@@ -14840,6 +14842,31 @@ description for file code block results which are inserted as Org mode links
 with no value the link path will be placed in both the ``link'' and the
 ``description'' portion of the Org mode link.
 
+@node file-ext
+@subsubsection @code{:file-ext}
+@cindex @code{:file-ext}, src header argument
+
+The value of the @code{:file-ext} header argument is used to provide an
+extension to write the file output to.  It is combined with the
+@code{#+NAME:} of the source block and the value of the @ref{output-dir}
+header argument to generate a complete file name.
+
+This header arg will be overridden by @code{:file}, and thus has no effect
+when the latter is specified.
+
+@node output-dir
+@subsubsection @code{:output-dir}
+@cindex @code{:output-dir}, src header argument
+
+The value of the @code{:output-dir} header argument is used to provide a
+directory to write the file output to.  It may specify an absolute directory
+(beginning with @code{/}) or a relative directory (without @code{/}).  It is
+combined with the @code{#+NAME:} of the source block and the value of the
+@ref{output-dir} header argument to generate a complete file name.
+
+This header arg will be overridden by @code{:file}, and thus has no effect
+when the latter is specified.
+
 @node dir
 @subsubsection @code{:dir} and remote execution
 @cindex @code{:dir}, src header argument
diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1348f04..4a3683d 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -283,6 +283,8 @@ Returns a list
     ;; resolve variable references and add summary parameters
     (when (and info (not light))
       (setf (nth 2 info) (org-babel-process-params (nth 2 info))))
+    (when info
+      (setf (nth 2 info) (org-babel-generate-file-param name (nth 2 info))))
     (when info (append info (list name indent head)))))
 
 (defvar org-babel-exp-reference-buffer nil
@@ -2890,6 +2892,38 @@ For the format of SAFE-LIST, see `org-babel-safe-header-args'."
 		      (member (cdr pair) (cdr entry)))
 		     (t nil)))))))
 
+(defun org-babel-generate-file-param (src-name params)
+  "Calculate the filename for source block results.
+
+The directory is calculated from the :output-dir property of the
+source block; if not specified, use the current directory.
+
+If the source block has a #+NAME and the :file parameter does not
+contain any period characters, then the :file parameter is
+treated as an extension, and the output file name is the
+concatenation of the directory (as calculated above), the block
+name, a period, and the parameter value as a file extension.
+Otherwise, the :file parameter is treated as a full file name,
+and the output file name is the directory (as calculated above)
+plus the parameter value."
+  (let* ((file-cons (assq :file params))
+	 (file-ext-cons (assq :file-ext params))
+	 (file-ext (cdr-safe file-ext-cons))
+	 (dir (or (cdr-safe (assq :output-dir params)) "")))
+    ;; Only operate if:
+    ;; 1. :file is not given
+    ;; 2. :file-ext is given
+    ;; 3. the source block has a #+name
+    (if (and (not file-cons) file-ext src-name)
+	(progn
+	  ;; Create the :output-dir if it does not exist
+	  (when (and file-ext (not (string= dir "")))
+	    (make-directory dir t)
+	    (unless (string-match "/\\'" dir)
+	      (setq dir (concat dir "/"))))
+	  (cons (cons :file (concat dir src-name "." file-ext)) params))
+      ;; Otherwise return params unmodified
+      params)))
 
 ;;; Used by backends: R, Maxima, Octave.
 (defun org-babel-graphical-output-file (params)
diff --git a/testing/examples/babel.org b/testing/examples/babel.org
index 449824f..68ba65a 100644
--- a/testing/examples/babel.org
+++ b/testing/examples/babel.org
@@ -458,3 +458,37 @@ function definition
                 "0"))))
   (format "elisp a:%d, b:%d, c:%d, d:%d, e:%d" a b c d e)
 #+END_SRC
+
+* =:file-ext= and =:output-dir= header args
+  :PROPERTIES:
+  :ID:       93573e1d-6486-442e-b6d0-3fedbdc37c9b
+  :END:
+#+name: file-ext-basic
+#+BEGIN_SRC emacs-lisp :file-ext txt
+nil
+#+END_SRC
+
+#+name: file-ext-dir-relative
+#+BEGIN_SRC emacs-lisp :file-ext txt :output-dir foo
+nil
+#+END_SRC
+
+#+name: file-ext-dir-relative-slash
+#+BEGIN_SRC emacs-lisp :file-ext txt :output-dir foo/
+nil
+#+END_SRC
+
+#+name: file-ext-dir-absolute
+#+BEGIN_SRC emacs-lisp :file-ext txt :output-dir /tmp
+nil
+#+END_SRC
+
+#+name: file-ext-file-wins
+#+BEGIN_SRC emacs-lisp :file-ext txt :file foo.bar
+nil
+#+END_SRC
+
+#+name: file-ext-file-wins2
+#+BEGIN_SRC emacs-lisp :output-dir xxx :file foo.bar
+nil
+#+END_SRC
diff --git a/testing/lisp/test-ob.el b/testing/lisp/test-ob.el
index c0ca493..fb791d6 100644
--- a/testing/lisp/test-ob.el
+++ b/testing/lisp/test-ob.el
@@ -1237,6 +1237,20 @@ echo \"$data\"
 		       (org-babel-execute-src-block)))
       (should (= noweb-expansions-in-cache-var 2)))))
 
+(ert-deftest test-org-babel/file-ext-and-output-dir ()
+  (org-test-at-id "93573e1d-6486-442e-b6d0-3fedbdc37c9b"
+    (macrolet ((at-next (&rest body)
+			`(progn
+			   (org-babel-next-src-block)
+			   (save-match-data ,@body))))
+      (at-next (should (equal "file-ext-basic.txt" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
+      (at-next (should (equal "foo/file-ext-dir-relative.txt" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
+      (at-next (should (equal "foo/file-ext-dir-relative-slash.txt" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
+      (at-next (should (equal "/tmp/file-ext-dir-absolute.txt" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
+      (at-next (should (equal "foo.bar" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
+      (at-next (should (equal "foo.bar" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
+      )))
+
 (provide 'test-ob)
 
 ;;; test-ob ends here
-- 
1.9.2

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Achim Gratz]

Aaron Ecay writes:
> Thanks so much for the feedback.  I’ve adopted the :file-ext approach
> suggested by Bastien, leaving the previous default behavior in place for
> blocks with a :file argument.

THanks for taking care, I'll not have time to look at this for the
remainder of the week, though…


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Bastien]

Hi Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> Thanks so much for the feedback.  I’ve adopted the :file-ext approach
> suggested by Bastien, leaving the previous default behavior in place for
> blocks with a :file argument.

Looks good -- please apply in master.

Thanks!

PS: I assume you have commit and push access, otherwise send me your
public key and I'll give it to you.

-- 
 Bastien

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Eric Schulte]

Hi Aaron,

Thanks for this patch, and especially for including documentation.  It
looks good to me, and it (largely [1]) passes all tests.

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Eric, Bastien, Achim,
>
> Thanks so much for the feedback.  I’ve adopted the :file-ext approach
> suggested by Bastien, leaving the previous default behavior in place for
> blocks with a :file argument.
>
> 2014ko apirilak 22an, Eric Schulte-ek idatzi zuen:
>
> [...]
>
>
>> One option might be to borrow naming behavior from the comment
>> functionality in ob-tangle which looks like the following (from line 426
>> in ob-tangle.el).
>>
>> (let (...
>>       (source-name
>>        (intern (or (nth 4 info)       ; explicit #+name:
>>                    (format "%s:%d"    ; constructed from header and position
>>                            (or (ignore-errors (nth 4 (org-heading-components)))
>>                                "No heading")
>>                            block-counter))))
>>       ...))
>
> I’m not sure I like this approach.  It relies on counting source
> blocks, so an addition/deletion of a block could change the index.
> I’m worried that this can lead to the accumulation of many output
> files: heading:1.ext, heading:2.ext, ... all with no clear indication
> of what block they were spawned by.  It would also be possible for
> the result links in the buffer to become inconsistent with the actual
> block:auto-generated name mapping.
>
> I think I would prefer the code in this patch to do nothing in this case
> (not create a :file value), but for language-specific code that needs a
> :file to raise an error to prompt the user to add a name.
>

Fair enough, especially given that this default will be applied to *all*
code blocks, this seems like a reasonable approach.

>
>>
>>>
>>> 2. should :output-dir apply to the :file case as well?
>>>
>>
>> If you mean "should :output-dir be used as the base when :file is a
>> relative pathname" then I'd say "yes", and I think if this isn't the
>> current behavior then the current behavior should be changed.
>
> Achim raises a backwards compatibility concern.  I am not sure how
> serious it is: the default settings (no :output-dir) are backwards
> compatible, and if users set that arg we ought to just give them what
> they ask for.
>
> Nonetheless, the new version of the patch conservatively obeys Achim’s
> suggestion.  I can change this to your suggestion, if that is the
> consensus.
>

Please do make this change, I'd then be happy to apply the resulting
patch.

Thanks again!
Eric

>
> To address a comment from Bastien: :output-dir accepts absolute as well
> as relative directory names.  Referring to a “subdirectory” was a
> mistake on my part; the docs in the new patch should be clearer.
>
> The updated patch (now with docs and tests) is attached to this email.
>
> Thanks again,
>
> --
> Aaron Ecay
>
> From 4b428820432752117c60b79da0a79fd4e50e4ba1 Mon Sep 17 00:00:00 2001
> From: Aaron Ecay <aaronecay@gmail.com>
> Date: Tue, 22 Apr 2014 15:13:48 -0400
> Subject: [PATCH] ob-core.el: allow the auto-generation of output file names
>  for src blocks.
>
> * lisp/ob-core.el (org-babel-generate-file-param): New function.
> (org-babel-get-src-block-info): Use it.
> * testing/lisp/test-ob.el (test-org-babel/file-ext-and-output-dir):
> New test.
> * doc/org.texi (Specific header arguments): Add doc for :file-ext and
> :output-dir header args.
> ---
>  doc/org.texi               | 27 +++++++++++++++++++++++++++
>  lisp/ob-core.el            | 34 ++++++++++++++++++++++++++++++++++
>  testing/examples/babel.org | 34 ++++++++++++++++++++++++++++++++++
>  testing/lisp/test-ob.el    | 14 ++++++++++++++
>  4 files changed, 109 insertions(+)
>
> diff --git a/doc/org.texi b/doc/org.texi
> index 2546be1..79cc044 100644
> --- a/doc/org.texi
> +++ b/doc/org.texi
> @@ -14406,6 +14406,8 @@ argument in lowercase letters.  The following header arguments are defined:
>                                  be collected and handled
>  * file::                        Specify a path for file output
>  * file-desc::                   Specify a description for file results
> +* file-ext::                    Specify an extension for file output
> +* output-dir::                  Specify a directory to write file output to
>  * dir::                         Specify the default (possibly remote)
>                                  directory for code block execution
>  * exports::                     Export code and/or results
> @@ -14840,6 +14842,31 @@ description for file code block results which are inserted as Org mode links
>  with no value the link path will be placed in both the ``link'' and the
>  ``description'' portion of the Org mode link.
>  
> +@node file-ext
> +@subsubsection @code{:file-ext}
> +@cindex @code{:file-ext}, src header argument
> +
> +The value of the @code{:file-ext} header argument is used to provide an
> +extension to write the file output to.  It is combined with the
> +@code{#+NAME:} of the source block and the value of the @ref{output-dir}
> +header argument to generate a complete file name.
> +
> +This header arg will be overridden by @code{:file}, and thus has no effect
> +when the latter is specified.
> +
> +@node output-dir
> +@subsubsection @code{:output-dir}
> +@cindex @code{:output-dir}, src header argument
> +
> +The value of the @code{:output-dir} header argument is used to provide a
> +directory to write the file output to.  It may specify an absolute directory
> +(beginning with @code{/}) or a relative directory (without @code{/}).  It is
> +combined with the @code{#+NAME:} of the source block and the value of the
> +@ref{output-dir} header argument to generate a complete file name.
> +
> +This header arg will be overridden by @code{:file}, and thus has no effect
> +when the latter is specified.
> +
>  @node dir
>  @subsubsection @code{:dir} and remote execution
>  @cindex @code{:dir}, src header argument
> diff --git a/lisp/ob-core.el b/lisp/ob-core.el
> index 1348f04..4a3683d 100644
> --- a/lisp/ob-core.el
> +++ b/lisp/ob-core.el
> @@ -283,6 +283,8 @@ Returns a list
>      ;; resolve variable references and add summary parameters
>      (when (and info (not light))
>        (setf (nth 2 info) (org-babel-process-params (nth 2 info))))
> +    (when info
> +      (setf (nth 2 info) (org-babel-generate-file-param name (nth 2 info))))
>      (when info (append info (list name indent head)))))
>  
>  (defvar org-babel-exp-reference-buffer nil
> @@ -2890,6 +2892,38 @@ For the format of SAFE-LIST, see `org-babel-safe-header-args'."
>  		      (member (cdr pair) (cdr entry)))
>  		     (t nil)))))))
>  
> +(defun org-babel-generate-file-param (src-name params)
> +  "Calculate the filename for source block results.
> +
> +The directory is calculated from the :output-dir property of the
> +source block; if not specified, use the current directory.
> +
> +If the source block has a #+NAME and the :file parameter does not
> +contain any period characters, then the :file parameter is
> +treated as an extension, and the output file name is the
> +concatenation of the directory (as calculated above), the block
> +name, a period, and the parameter value as a file extension.
> +Otherwise, the :file parameter is treated as a full file name,
> +and the output file name is the directory (as calculated above)
> +plus the parameter value."
> +  (let* ((file-cons (assq :file params))
> +	 (file-ext-cons (assq :file-ext params))
> +	 (file-ext (cdr-safe file-ext-cons))
> +	 (dir (or (cdr-safe (assq :output-dir params)) "")))
> +    ;; Only operate if:
> +    ;; 1. :file is not given
> +    ;; 2. :file-ext is given
> +    ;; 3. the source block has a #+name
> +    (if (and (not file-cons) file-ext src-name)
> +	(progn
> +	  ;; Create the :output-dir if it does not exist
> +	  (when (and file-ext (not (string= dir "")))
> +	    (make-directory dir t)
> +	    (unless (string-match "/\\'" dir)
> +	      (setq dir (concat dir "/"))))
> +	  (cons (cons :file (concat dir src-name "." file-ext)) params))
> +      ;; Otherwise return params unmodified
> +      params)))
>  
>  ;;; Used by backends: R, Maxima, Octave.
>  (defun org-babel-graphical-output-file (params)
> diff --git a/testing/examples/babel.org b/testing/examples/babel.org
> index 449824f..68ba65a 100644
> --- a/testing/examples/babel.org
> +++ b/testing/examples/babel.org
> @@ -458,3 +458,37 @@ function definition
>                  "0"))))
>    (format "elisp a:%d, b:%d, c:%d, d:%d, e:%d" a b c d e)
>  #+END_SRC
> +
> +* =:file-ext= and =:output-dir= header args
> +  :PROPERTIES:
> +  :ID:       93573e1d-6486-442e-b6d0-3fedbdc37c9b
> +  :END:
> +#+name: file-ext-basic
> +#+BEGIN_SRC emacs-lisp :file-ext txt
> +nil
> +#+END_SRC
> +
> +#+name: file-ext-dir-relative
> +#+BEGIN_SRC emacs-lisp :file-ext txt :output-dir foo
> +nil
> +#+END_SRC
> +
> +#+name: file-ext-dir-relative-slash
> +#+BEGIN_SRC emacs-lisp :file-ext txt :output-dir foo/
> +nil
> +#+END_SRC
> +
> +#+name: file-ext-dir-absolute
> +#+BEGIN_SRC emacs-lisp :file-ext txt :output-dir /tmp
> +nil
> +#+END_SRC
> +
> +#+name: file-ext-file-wins
> +#+BEGIN_SRC emacs-lisp :file-ext txt :file foo.bar
> +nil
> +#+END_SRC
> +
> +#+name: file-ext-file-wins2
> +#+BEGIN_SRC emacs-lisp :output-dir xxx :file foo.bar
> +nil
> +#+END_SRC
> diff --git a/testing/lisp/test-ob.el b/testing/lisp/test-ob.el
> index c0ca493..fb791d6 100644
> --- a/testing/lisp/test-ob.el
> +++ b/testing/lisp/test-ob.el
> @@ -1237,6 +1237,20 @@ echo \"$data\"
>  		       (org-babel-execute-src-block)))
>        (should (= noweb-expansions-in-cache-var 2)))))
>  
> +(ert-deftest test-org-babel/file-ext-and-output-dir ()
> +  (org-test-at-id "93573e1d-6486-442e-b6d0-3fedbdc37c9b"
> +    (macrolet ((at-next (&rest body)
> +			`(progn
> +			   (org-babel-next-src-block)
> +			   (save-match-data ,@body))))
> +      (at-next (should (equal "file-ext-basic.txt" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
> +      (at-next (should (equal "foo/file-ext-dir-relative.txt" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
> +      (at-next (should (equal "foo/file-ext-dir-relative-slash.txt" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
> +      (at-next (should (equal "/tmp/file-ext-dir-absolute.txt" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
> +      (at-next (should (equal "foo.bar" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
> +      (at-next (should (equal "foo.bar" (cdr (assq :file (nth 2 (org-babel-get-src-block-info t)))))))
> +      )))
> +
>  (provide 'test-ob)
>  
>  ;;; test-ob ends here


Footnotes: 
[1]  It took some extra work and manual refreshing of org-id's to get
     test-org-babel/file-ext-and-output-dir to pass.  For some reason
     this tests still fails when I run "make test-dirty", but passes
     when run interactively.  This could easily be particular to my
     system.

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Aaron Ecay]

Hi Eric, Bastien, Achim, et al.,

Thanks again for the feedback.  I just pushed the revised patch to master.

2014ko maiatzak 4an, Eric Schulte-ek idatzi zuen:

[...]

>>> One option might be to borrow naming behavior from the comment
>>> functionality in ob-tangle which looks like the following (from line 426
>>> in ob-tangle.el).
>>>
>>> (let (...
>>> (source-name
>>> (intern (or (nth 4 info)       ; explicit #+name:
>>> (format "%s:%d"    ; constructed from header and position
>>> (or (ignore-errors (nth 4 (org-heading-components)))
>>> "No heading")
>>> block-counter))))
>>> ...))
>>
>> I’m not sure I like this approach.  It relies on counting source
>> blocks, so an addition/deletion of a block could change the index.
>> I’m worried that this can lead to the accumulation of many output
>> files: heading:1.ext, heading:2.ext, ... all with no clear indication
>> of what block they were spawned by.  It would also be possible for
>> the result links in the buffer to become inconsistent with the actual
>> block:auto-generated name mapping.
>>
>> I think I would prefer the code in this patch to do nothing in this case
>> (not create a :file value), but for language-specific code that needs a
>> :file to raise an error to prompt the user to add a name.
>>
>
> Fair enough, especially given that this default will be applied to *all*
> code blocks, this seems like a reasonable approach.

I went ahead with my suggested approach here.

[...]

>> Achim raises a backwards compatibility concern.  I am not sure how
>> serious it is: the default settings (no :output-dir) are backwards
>> compatible, and if users set that arg we ought to just give them what
>> they ask for.
>>
>> Nonetheless, the new version of the patch conservatively obeys Achim’s
>> suggestion.  I can change this to your suggestion, if that is the
>> consensus.
>>
>
> Please do make this change, I'd then be happy to apply the resulting
> patch.

Done, as you and Bastien suggested.

Thanks,

--
Aaron Ecay

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Achim Gratz]

Aaron Ecay writes:
> Thanks again for the feedback.  I just pushed the revised patch to master.
>
>>> I think I would prefer the code in this patch to do nothing in this case
>>> (not create a :file value), but for language-specific code that needs a
>>> :file to raise an error to prompt the user to add a name.
>>>
>>
>> Fair enough, especially given that this default will be applied to *all*
>> code blocks, this seems like a reasonable approach.
>
> I went ahead with my suggested approach here.

That breaks the tests for Octave and Maxima; since you're intentionally
not keeping backwards compatibility here this should be fixed in the
tests, I'd think.

Also, I'd think you should be using "user-error" instead of error to
generate the messages.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Bastien]

Hi,

Achim Gratz <Stromeko@nexgo.de> writes:

> That breaks the tests for Octave and Maxima; since you're intentionally
> not keeping backwards compatibility here this should be fixed in the
> tests, I'd think.

Agreed.  Aaron, can you take care of this?

> Also, I'd think you should be using "user-error" instead of error to
> generate the messages.

Agreed too.

-- 
 Bastien

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Aaron Ecay]

Hi Achim,

2014ko maiatzak 14an, Achim Gratz-ek idatzi zuen:
> That breaks the tests for Octave and Maxima; since you're intentionally
> not keeping backwards compatibility here this should be fixed in the
> tests, I'd think.

Fixed.  (It actually required changes to the code, not the tests, since
my commit made org-babel-graphical-output-file stricter).

>
> Also, I'd think you should be using "user-error" instead of error to
> generate the messages.

Good catch (especially since I recently pushed a patch changing some
errors to user-errors *blush*).  Fixed.

Thanks,

--
Aaron Ecay

Re: [RFC] [PATCH] ob-core.el: allow the auto-generation of output file names for src blocks.

[Achim Gratz]

Aaron Ecay writes:
> Fixed.  (It actually required changes to the code, not the tests, since
> my commit made org-babel-graphical-output-file stricter).

OK.

> Good catch (especially since I recently pushed a patch changing some
> errors to user-errors *blush*).  Fixed.

Code review works. :-)


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra