[BUG][babel] Tangling into directories does not add directories to org file

[BUG][babel] Tangling into directories does not add directories to org file

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

When in the tangling taerget a directory is specified, this is not
reflected in the comments in the tangled file when comments are set to
include links:

,----
|    * `link' The code block is wrapped in comments which contain
|      pointers back to the original Org file from which the code was
|      tangled.
`----

In the org file =prodMixStands.org=:

,----
| :PROPERTIES:
| :header-args+: :tangle ./R/update.cache.R
| :END:
| #+begin_src R 
| test
| #+end_src
`----

results in the file =update.cache.R= in a link to the file
=prodMixStands.org= and not =./../prodMixstands.org=:

,----
|  ## [[file:prodMixStands.org::*Begin][Begin:1]]
`----

This is particularly bad when using ess and developer mode to develop a
package.

Cheers,

Rainer

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: Type: application/pgp-signature, Size: 494 bytes --]

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Rainer M Krug]

I'd like to ping this as I think it
is an important bug. It e.g. make debugging of R packages, whose source
is in org, more difficult, (the .R files are tangled into ./R/) as ess
does not find the source of the R code in R as it uses the links in the
comments to find the org file.

Temporary solution: I created a link to the org file in the ./R directory.

Cheers,

Rainer

Rainer M Krug <Rainer@krugs.de> writes:

> When in the tangling taerget a directory is specified, this is not
> reflected in the comments in the tangled file when comments are set to
> include links:
>
> ,----
> |    * `link' The code block is wrapped in comments which contain
> |      pointers back to the original Org file from which the code was
> |      tangled.
> `----
>
> In the org file =prodMixStands.org=:
>
> ,----
> | :PROPERTIES:
> | :header-args+: :tangle ./R/update.cache.R
> | :END:
> | #+begin_src R 
> | test
> | #+end_src
> `----
>
> results in the file =update.cache.R= in a link to the file
> =prodMixStands.org= and not =./../prodMixstands.org=:
>
> ,----
> |  ## [[file:prodMixStands.org::*Begin][Begin:1]]
> `----
>
> This is particularly bad when using ess and developer mode to develop a
> package.
>
> Cheers,
>
> Rainer

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Grant Rettke]

Sounds like a valuable use of R.

Do you have a writeup on the approach or is it a standard feature of
org and ess?

On Thu, Oct 9, 2014 at 7:37 AM, Rainer M Krug <Rainer@krugs.de> wrote:
>
> I'd like to ping this as I think it
> is an important bug. It e.g. make debugging of R packages, whose source
> is in org, more difficult, (the .R files are tangled into ./R/) as ess
> does not find the source of the R code in R as it uses the links in the
> comments to find the org file.
>
> Temporary solution: I created a link to the org file in the ./R directory.
>
> Cheers,
>
> Rainer
>
> Rainer M Krug <Rainer@krugs.de> writes:
>
>> When in the tangling taerget a directory is specified, this is not
>> reflected in the comments in the tangled file when comments are set to
>> include links:
>>
>> ,----
>> |    * `link' The code block is wrapped in comments which contain
>> |      pointers back to the original Org file from which the code was
>> |      tangled.
>> `----
>>
>> In the org file =prodMixStands.org=:
>>
>> ,----
>> | :PROPERTIES:
>> | :header-args+: :tangle ./R/update.cache.R
>> | :END:
>> | #+begin_src R
>> | test
>> | #+end_src
>> `----
>>
>> results in the file =update.cache.R= in a link to the file
>> =prodMixStands.org= and not =./../prodMixstands.org=:
>>
>> ,----
>> |  ## [[file:prodMixStands.org::*Begin][Begin:1]]
>> `----
>>
>> This is particularly bad when using ess and developer mode to develop a
>> package.
>>
>> Cheers,
>>
>> Rainer
>
> --
> Rainer M. Krug
> email: Rainer<at>krugs<dot>de
> PGP: 0x0F52F982
>



-- 
Grant Rettke
gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
“Wisdom begins in wonder.” --Socrates
((λ (x) (x x)) (λ (x) (x x)))
“Life has become immeasurably better since I have been forced to stop
taking it seriously.” --Thompson

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 7007 bytes --]

Grant Rettke <gcr@wisdomandwonder.com> writes:

> Sounds like a valuable use of R.

It definitely is.


>
> Do you have a writeup on the approach or is it a standard feature of
> org and ess?

No - but please bug me - I can post in the intermediate future an
example demonstrating the points.

But the main points are:

* Setup
** I added the following to my emacs.org
*** Add local variable in tangled files if they match [.]R
The variable =org-babel-tangled-file t= enables ess to trace back the
source from the .R file to the .org file, using the comments which
have to include the org links.

See the thread "Automatically adding local variables to tangled file" for details

#+begin_src emacs-lisp
(defvar org-babel-tangled-file nil
  "If non-nill, current file was tangled with org-babel-tangle")

(put 'org-babel-tangled-file 'safe-local-variable 'booleanp)

(defun org-babel-mark-file-as-tangled ()
  (when (buffer-file-name)
    (when  (string-match "[.]R" (buffer-file-name))
      (add-file-local-variable 'org-babel-tangled-file t)
      (add-file-local-variable 'buffer-read-only t)
      (add-file-local-variable 'eval: '(auto-revert-mode))
      (basic-save-buffer))))

(add-hook 'org-babel-post-tangle-hook 'org-babel-mark-file-as-tangled)
#+end_src

** Tangling
- The files have to be tangled with the header argument =:comments link=
(or both) for this to work.
- =:padline no= is necessary (at least it was) as the empty lines in
  the tangled file caused ess to identify the wrong line in the source
  code

** This should now work.

* Use code as "package"
Effectively, I do the following:
** code
tangle all .R files to ./R *without variables transfer*. These should
contain definitions of functions and not executable code. 

** data
data files (i.e. the variables transferred from org) are transfered as
tangled files in in the ./data directory.

*** Species names and espre codes of species used by Vallet
:PROPERTIES:
:header-args+: :tangle ./data/SPECIESVALLET.R
:header-args+: :rownames yes
:header-args+: :colnames yes
:END:
#+NAME: SPECIESVALLET
|         | fullName        | shortName | espre | IFNName         | color |
|---------+-----------------+-----------+---------+-----------------+-------|
| fagus   | Fagus sylvatica | fagus     |       4 | fagus_sylvatica | red   |
| quercus | Quercus robur   | quercus   |       3 | quercus_robur   | green |
#+begin_src R :var SPECIESVALLET=SPECIESVALLET

#+end_src

which evaluates to ./data/SPECIESVALLET.R

,----
| ## Species names and espre codes of species used by Vallet
| ## :PROPERTIES:
| ## :header-args+: :tangle ./data/SPECIESVALLET.R
| ## :header-args+: :rownames yes
| ## :header-args+: :colnames yes
| ## :END:
| ## #+NAME: SPECIESVALLET
| ## |         | fullName        | shortName | espre | IFNName         | color |
| ## |---------+-----------------+-----------+---------+-----------------+-------|
| ## | fagus   | Fagus sylvatica | fagus     |       4 | fagus_sylvatica | red   |
| ## | quercus | Quercus robur   | quercus   |       3 | quercus_robur   | green |
| 
| ## [[file:prodMixStands.org::*Species%20names%20and%20espre%20codes%20of%20species%20used%20by%20Vallet][Species\ names\ and\ espre\ codes\ of\ species\ used\ by\ Vallet:1]]
| SPECIESVALLET <- local({
|      con <- textConnection(
|        "\"fullName\"	\"shortName\"	\"espre\"	\"IFNName\"	\"color\"
| \"fagus\"	\"Fagus sylvatica\"	\"fagus\"	\"4\"	\"fagus_sylvatica\"	\"red\"
| \"quercus\"	\"Quercus robur\"	\"quercus\"	\"3\"	\"quercus_robur\"	\"green\""
|      )
|      res <- utils::read.table(
|        con,
|        header    = TRUE,
|        row.names = 1,
|        sep       = "\t",
|        as.is     = TRUE
|      )
|      close(con)
|      res
|    })
| ## Species\ names\ and\ espre\ codes\ of\ species\ used\ by\ Vallet:1 ends here
| 
| ## Local Variables:
| ## org-babel-tangled-file: t
| ## buffer-read-only: t
| ## eval:: (auto-revert-mode)
| ## End:
`----

** evaluation
data and code is *never* evaluated, i.e. 
:PROPERTIES:
:header-args+: :eval never
:END:

The code and data is loaded as follow:
#+begin_src R 
library(devtools)
load_all("./")
#+end_src

and there you go.
 
To debug the code, I enable developer mode in ESS
and add the package =the_name_of_the_directory_./= to be debugged.

* Using the code
I have a second subtree (except of the one containing the code and the
data) which is evaluated (as some lengthy analysis are involved, not on export but before) and contains the actual analysis. 

* Advantages
- All code and data is enclosed in a "package" environment and,
- when the project is finished, it can be easily packed in a package
  and handed on to other users.
- documentation to the functions can be added using roxygen which can
  easily be translated into .rd files


* Final words
I hope this clarifies my approach.

For me, this is a very "natural" approach, as I like to use packages
and capsulate everything in environments / packages, and it seems very
"clean" to me to define the core of the analysis as functions in
=./R/*.R= files, the variables in =./data/*= files and the actual
analysis as mainly a few function calls.

If you have any further questions, please go on and ask.

Cheers,

Rainer





>
> Online Thu, Oct 9, 2014 at 7:37 AM, Rainer M Krug <Rainer@krugs.de> wrote:
>>
>> I'd like to ping this as I think it
>> is an important bug. It e.g. make debugging of R packages, whose source
>> is in org, more difficult, (the .R files are tangled into ./R/) as ess
>> does not find the source of the R code in R as it uses the links in the
>> comments to find the org file.
>>
>> Temporary solution: I created a link to the org file in the ./R directory.
>>
>> Cheers,
>>
>> Rainer
>>
>> Rainer M Krug <Rainer@krugs.de> writes:
>>
>>> When in the tangling taerget a directory is specified, this is not
>>> reflected in the comments in the tangled file when comments are set to
>>> include links:
>>>
>>> ,----
>>> |    * `link' The code block is wrapped in comments which contain
>>> |      pointers back to the original Org file from which the code was
>>> |      tangled.
>>> `----
>>>
>>> In the org file =prodMixStands.org=:
>>>
>>> ,----
>>> | :PROPERTIES:
>>> | :header-args+: :tangle ./R/update.cache.R
>>> | :END:
>>> | #+begin_src R
>>> | test
>>> | #+end_src
>>> `----
>>>
>>> results in the file =update.cache.R= in a link to the file
>>> =prodMixStands.org= and not =./../prodMixstands.org=:
>>>
>>> ,----
>>> |  ## [[file:prodMixStands.org::*Begin][Begin:1]]
>>> `----
>>>
>>> This is particularly bad when using ess and developer mode to develop a
>>> package.
>>>
>>> Cheers,
>>>
>>> Rainer
>>
>> --
>> Rainer M. Krug
>> email: Rainer<at>krugs<dot>de
>> PGP: 0x0F52F982
>>

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: Type: application/pgp-signature, Size: 494 bytes --]

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Grant Rettke]

On Fri, Oct 10, 2014 at 3:24 AM, Rainer M Krug <Rainer@krugs.de> wrote:
> Grant Rettke <gcr@wisdomandwonder.com> writes:
>
>> Sounds like a valuable use of R.
>
> It definitely is.
>
>
>>
>> Do you have a writeup on the approach or is it a standard feature of
>> org and ess?
>
> No - but please bug me - I can post in the intermediate future an
> example demonstrating the points.

I will read and ask, thanks.

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Aaron Ecay]

Hi Rainer,

2014ko urriak 9an, Rainer M Krug-ek idatzi zuen:
> 
> I'd like to ping this as I think it
> is an important bug. It e.g. make debugging of R packages, whose source
> is in org, more difficult, (the .R files are tangled into ./R/) as ess
> does not find the source of the R code in R as it uses the links in the
> comments to find the org file.

FWIW, I agree with you, but personally it’s hard to see myself having
time to address this in the near future.  Maybe you could try making a
patch yourself.  It sounds like the sort of thing that should be
self-contained and easy to fix (famous last words, I know...)

-- 
Aaron Ecay

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Rainer,
>
> 2014ko urriak 9an, Rainer M Krug-ek idatzi zuen:
>> 
>> I'd like to ping this as I think it
>> is an important bug. It e.g. make debugging of R packages, whose source
>> is in org, more difficult, (the .R files are tangled into ./R/) as ess
>> does not find the source of the R code in R as it uses the links in the
>> comments to find the org file.
>
> FWIW, I agree with you, but personally it’s hard to see myself having
> time to address this in the near future.  Maybe you could try making a
> patch yourself.  It sounds like the sort of thing that should be
> self-contained and easy to fix (famous last words, I know...)

It looks like this - but I am always shocked by the number of brackets
in lisp.

The "problem" is in the function =defun org-babel-spec-to-string (spec)=
in the file =lisp/ob-tangle.R= in the section

#+begin_src emacs-lisp 
  (file (if org-babel-tangle-use-relative-file-links
		   (file-relative-name (nth 1 spec))
		 (nth 1 spec)))
	 (link (let ((link (nth 2 spec)))
		 (if org-babel-tangle-use-relative-file-links
		     (when (string-match "^\\(file:\\|docview:\\)\\(.*\\)" link)
		       (let* ((type (match-string 1 link))
			      (path (match-string 2 link))
			      (origpath path)
			      (case-fold-search nil))
			 (setq path (file-relative-name path))
			 (concat type path)))
		   link)))
#+end_src

the function file-relative-name (file-relative-name FILENAME &optional
DIRECTORY). If I am not mistaken, the optional argument DIRECTORY
needs to be set to the directory of the tangled file. I assume it is
not set at the moment. 

But I have no idea how to get this path and how to pass it to the function.

Rainer

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: Type: application/pgp-signature, Size: 494 bytes --]

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Aaron Ecay]

Hi Rainer,

2014ko urriak 10an, Rainer M Krug-ek idatzi zuen:
> It looks like this - but I am always shocked by the number of brackets
> in lisp.

It’s a bit of an adjustment.  If you get into any serious lisp coding,
check out paredit – it forces you to always have balanced brackets in
the file, thus avoiding a whole class of silly but hard-to-spot mistakes
on lines ending with and avalanche of )))))

<http://www.emacswiki.org/emacs/ParEdit>

> 
> The "problem" is in the function =defun org-babel-spec-to-string (spec)=
> in the file =lisp/ob-tangle.R= in the section
> 
> #+begin_src emacs-lisp 
>   (file (if org-babel-tangle-use-relative-file-links
> 		   (file-relative-name (nth 1 spec))
> 		 (nth 1 spec)))
> 	 (link (let ((link (nth 2 spec)))
> 		 (if org-babel-tangle-use-relative-file-links
> 		     (when (string-match "^\\(file:\\|docview:\\)\\(.*\\)" link)
> 		       (let* ((type (match-string 1 link))
> 			      (path (match-string 2 link))
> 			      (origpath path)
> 			      (case-fold-search nil))
> 			 (setq path (file-relative-name path))
> 			 (concat type path)))
> 		   link)))
> #+end_src
> 
> the function file-relative-name (file-relative-name FILENAME &optional
> DIRECTORY). If I am not mistaken, the optional argument DIRECTORY
> needs to be set to the directory of the tangled file. I assume it is
> not set at the moment. 
> 
> But I have no idea how to get this path and how to pass it to the
> function.

Something like (file-name-directory (buffer-file-name)) should do the
trick, I think.  It would be good if you could test tangling from a
remote (i.e. via TRAMP) org file to a local output file and vice versa
to make sure that this DTRT in those cases as well.

Don’t hesitate to ask if you have more questions!

-- 
Aaron Ecay

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Thorsten Jolitz]

Aaron Ecay <aaronecay@gmail.com> writes:

Hi Rainer,

> 2014ko urriak 10an, Rainer M Krug-ek idatzi zuen:
>> It looks like this - but I am always shocked by the number of brackets
>> in lisp.
>
> It’s a bit of an adjustment.  If you get into any serious lisp coding,
> check out paredit – it forces you to always have balanced brackets in
> the file, thus avoiding a whole class of silly but hard-to-spot mistakes
> on lines ending with and avalanche of )))))
>
> <http://www.emacswiki.org/emacs/ParEdit>

if you are less into bondage&submission, smartparens.el is another option,
it *helps* to have balanced brackets w/o breaking your will ;-)

-- 
cheers,
Thorsten

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Grant Rettke]

On Fri, Oct 10, 2014 at 11:50 AM, Thorsten Jolitz <tjolitz@gmail.com> wrote:
> Aaron Ecay <aaronecay@gmail.com> writes:
>
> Hi Rainer,
>
>> 2014ko urriak 10an, Rainer M Krug-ek idatzi zuen:
>>> It looks like this - but I am always shocked by the number of brackets
>>> in lisp.
>>
>> It’s a bit of an adjustment.  If you get into any serious lisp coding,
>> check out paredit – it forces you to always have balanced brackets in
>> the file, thus avoiding a whole class of silly but hard-to-spot mistakes
>> on lines ending with and avalanche of )))))
>>
>> <http://www.emacswiki.org/emacs/ParEdit>
>
> if you are less into bondage&submission, smartparens.el is another option,
> it *helps* to have balanced brackets w/o breaking your will ;-)

It is more about ease-of-coding them B&D.

=turn-on-smartparens-strict-mode= add years to ones life to allow for
more hacking. Seriously.

Re: [BUG][babel] Tangling into directories does not add directories to org file

[Aaron Ecay]

Hi Rainer,

2014ko urriak 10an, Aaron Ecay-ek idatzi zuen:

[...]

>> the function file-relative-name (file-relative-name FILENAME &optional
>> DIRECTORY). If I am not mistaken, the optional argument DIRECTORY
>> needs to be set to the directory of the tangled file. I assume it is
>> not set at the moment. 
>> 
>> But I have no idea how to get this path and how to pass it to the
>> function.
>
> Something like (file-name-directory (buffer-file-name)) should do the
> trick, I think.

I just (re)-read the “...and pass it to the function” part.  The lisp
convention is that function calls look like:

(FUNCTION ARG1 ARG2 ... ARGN)

So you’d want something like:

(file (if org-babel-tangle-use-relative-file-links
          (file-relative-name (nth 1 spec) (file-name-directory (buffer-file-name))) ;;; <- addition here
        (nth 1 spec)))
(link (let ((link (nth 2 spec)))
        (if org-babel-tangle-use-relative-file-links
            (when (string-match "^\\(file:\\|docview:\\)\\(.*\\)" link)
              (let* ((type (match-string 1 link))
                     (path (match-string 2 link))
                     (origpath path)
                     (case-fold-search nil))
                (setq path (file-relative-name path (file-name-directory (buffer-file-name)))) ;;; <- addition here
                (concat type path)))
          link)))

Apologies if this is too basic an explanation, or otherwise unclear...

-- 
Aaron Ecay