R code block produces only partial output

R code block produces only partial output

[Andreas Kiermeier]

Hi all,

I've been using Org mode (latest version - see my set up below) to
perform some data analysis and have come across a problem evaluating R
code blocks. I usually use a session when working and when evaluating
a code block with describe (R package "rms") and a quantitative
variable the results in .org buffer and the *R* buffer (output)
differ. However, it's not the describe function, but something about
the output of that function. The following is a minimal example (both
with and without the describe function) to illustrate the problem.

I think it has something to do with a number of spaces followed by a
'.' in the output buffer.

Any thoughts or guidance would be greatly appreciated.

Thanks in advance.

Andreas

------- test.org ---------------------
#+STARTUP: showall indent hidestars
#+PROPERTY: header-args:R :session *R*
#+PROPERTY: header-args :results output graphics :exports both

#+BEGIN_SRC R
  require(rms)
  x <- rnorm(100)
  describe(x)
#+END_SRC

#+RESULTS:
: 75      .90      .95
:  0.69261  1.38170  1.71468
:
: lowest : -2.270 -1.878 -1.705 -1.525 -1.423
: highest:  1.739  2.032  2.061  2.150  2.480

#+BEGIN_SRC R
  x <- rnorm(100)
  y <- quantile(x, probs=seq(0,1,0.1))
  names(y) <- as.character(c("0",".1",".2",".3",".4",".5",".6",".7",".8",".9","1"))
  y
#+END_SRC

#+RESULTS:
: 6          .7          .8          .9           1
:  0.31144890  0.63292642  0.87168950  1.17596547  1.96212110
------- end test.org ---------------------


The actual output for the first code block in the *R* buffer was:

> describe(x)
x
       n  missing   unique     Mean      .05      .10      .25      .50
     100        0      100   0.1118 -1.39404 -1.02732 -0.45899  0.03756
     .75      .90      .95
 0.69261  1.38170  1.71468

lowest : -2.270 -1.878 -1.705 -1.525 -1.423
highest:  1.739  2.032  2.061  2.150  2.480


Similarly, the second code block resulted in:

> y
          0          .1          .2          .3          .4          .5
-2.23303584 -1.31137871 -0.69424743 -0.48689155 -0.16973865  0.09289862
         .6          .7          .8          .9           1
 0.31144890  0.63292642  0.87168950  1.17596547  1.96212110


And finally my set up is:

Emacs  : GNU Emacs 24.3.1 (i386-mingw-nt6.2.9200)
 of 2013-03-18 on MARVIN
Package: Org-mode version 8.2.7c (8.2.7c-elpa @
c:/Users/andre_000/Documents/.emacs.d/elpa/org-20140730/)

current state:
==============
(setq
 org-ctrl-c-ctrl-c-hook '(org-babel-hash-at-point
org-babel-execute-safely-maybe)
 org-latex-format-headline-function 'org-latex-format-headline-default-function
 org-src-fontify-natively t
 org-html-format-inlinetask-function 'ignore
 org-completion-use-iswitchb t
 org-tab-first-hook '(org-hide-block-toggle-maybe
org-src-native-tab-command-maybe
     org-babel-hide-result-toggle-maybe org-babel-header-arg-expand)
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers
 org-cycle-hide-inline-tasks org-cycle-show-empty-lines
 org-optimize-window-after-visibility-change)
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-confirm-babel-evaluate nil
 org-speed-command-hook '(org-speed-command-default-hook
org-babel-speed-command-hook)
 org-ascii-format-inlinetask-function 'org-ascii-format-inlinetask-default
 org-babel-pre-tangle-hook '(save-buffer)
 org-occur-hook '(org-first-headline-recenter)
 org-src-lang-modes '(("r" . ess-mode) ("ocaml" . tuareg) ("elisp" . emacs-lisp)
     ("ditaa" . artist) ("asymptote" . asy) ("dot" . fundamental)
     ("sqlite" . sql) ("calc" . fundamental) ("C" . c) ("cpp" . c++)
     ("C++" . c++) ("screen" . shell-script))
 org-shiftdown-final-hook '(windmove-down)
 org-html-format-headline-function 'ignore
 org-log-done 'time
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-catch-invisible-edits 'smart
 org-default-notes-file "~/.org//notes.org"
 org-latex-format-drawer-function '(lambda (name contents) contents)
 org-src-tab-acts-natively t
 org-export-backends '(ascii html icalendar latex md)
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-completion-use-ido t
 org-use-speed-commands t
 org-mode-hook '(#[nil "\300\301\302\303\304$\207"
  [org-add-hook change-major-mode-hook org-show-block-all append local] 5]
#[nil "\300\301\302\303\304$\207"
  [org-add-hook change-major-mode-hook org-babel-show-result-all
append local] 5]
org-babel-result-hide-spec org-babel-hide-all-hashes org-mode-reftex-setup
(lambda nil (flyspell-mode 1) (auto-fill-mode 1)
 (local-set-key "\220" (quote outline-previous-visible-heading))
 (local-set-key "\225" (quote outline-up-heading))
 (local-set-key "\227" (quote org-table-copy-region))
 (local-set-key "\231" (quote org-table-paste-rectangle))
 (local-set-key "\214" (quote org-table-sort-lines))
 (local-set-key "\311" (quote org-toggle-iimage-in-org)))
)
 org-shiftup-final-hook '(windmove-up)
 org-ascii-format-drawer-function '(lambda (name contents width) contents)
 org-directory "~/.org/"
 org-html-format-drawer-function '(lambda (name contents) contents)
 org-shiftleft-final-hook '(windmove-left)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-agenda-files '("~/.org/agenda.org")
 org-src-mode-hook '(org-src-babel-configure-edit-buffer
org-src-mode-configure-edit-buffer)
 org-after-todo-state-change-hook '(org-clock-out-if-current)
 org-todo-keyword-faces '(("WAITING" . ak-org-todo-warning-face))
 org-shiftright-final-hook '(windmove-right)
 org-babel-load-languages '((emacs-lisp) (R . t))
 org-babel-R-command "C:/Progra~1/R/R-3.1.1/bin/x64/R --slave --no-save"
 org-latex-format-inlinetask-function 'ignore
 org-confirm-shell-link-function 'yes-or-no-p
 )

Re: R code block produces only partial output

[Eric Schulte]

Hi Andreas,

Why are you setting the output type to "graphics" when you are trying to
return text?  I think that may be the source of your problem.

Best,
Eric

Andreas Kiermeier <andreas.kiermeier@gmail.com> writes:

> Hi all,
>
> I've been using Org mode (latest version - see my set up below) to
> perform some data analysis and have come across a problem evaluating R
> code blocks. I usually use a session when working and when evaluating
> a code block with describe (R package "rms") and a quantitative
> variable the results in .org buffer and the *R* buffer (output)
> differ. However, it's not the describe function, but something about
> the output of that function. The following is a minimal example (both
> with and without the describe function) to illustrate the problem.
>
> I think it has something to do with a number of spaces followed by a
> '.' in the output buffer.
>
> Any thoughts or guidance would be greatly appreciated.
>
> Thanks in advance.
>
> Andreas
>
> ------- test.org ---------------------
>
> #+STARTUP: showall indent hidestars
> #+PROPERTY: header-args:R :session *R*
>
> #+PROPERTY: header-args :results output graphics :exports both
>
> #+BEGIN_SRC R
>   require(rms)
>   x <- rnorm(100)
>   describe(x)
> #+END_SRC
>
>
> #+RESULTS:
> : 75      .90      .95
> :  0.69261  1.38170  1.71468
> :
> : lowest : -2.270 -1.878 -1.705 -1.525 -1.423
> : highest:  1.739  2.032  2.061  2.150  2.480
>
> #+BEGIN_SRC R
>   x <- rnorm(100)
>   y <- quantile(x, probs=seq(0,1,0.1))
>   names(y) <- as.character(c("0",".1",".2",".3",".4",".5",".6",".7",".8",".9","1"))
>   y
> #+END_SRC
>
> #+RESULTS:
> : 6          .7          .8          .9           1
> :  0.31144890  0.63292642  0.87168950  1.17596547  1.96212110
> ------- end test.org ---------------------
>
>
> The actual output for the first code block in the *R* buffer was:
>
>> describe(x)
> x
>        n  missing   unique     Mean      .05      .10      .25      .50
>      100        0      100   0.1118 -1.39404 -1.02732 -0.45899  0.03756
>      .75      .90      .95
>  0.69261  1.38170  1.71468
>
> lowest : -2.270 -1.878 -1.705 -1.525 -1.423
> highest:  1.739  2.032  2.061  2.150  2.480
>
>
> Similarly, the second code block resulted in:
>
>> y
>           0          .1          .2          .3          .4          .5
> -2.23303584 -1.31137871 -0.69424743 -0.48689155 -0.16973865  0.09289862
>          .6          .7          .8          .9           1
>  0.31144890  0.63292642  0.87168950  1.17596547  1.96212110
>
>
> And finally my set up is:
>
> Emacs  : GNU Emacs 24.3.1 (i386-mingw-nt6.2.9200)
>  of 2013-03-18 on MARVIN
> Package: Org-mode version 8.2.7c (8.2.7c-elpa @
> c:/Users/andre_000/Documents/.emacs.d/elpa/org-20140730/)
>
> current state:
> ==============
> (setq
>  org-ctrl-c-ctrl-c-hook '(org-babel-hash-at-point
> org-babel-execute-safely-maybe)
>  org-latex-format-headline-function 'org-latex-format-headline-default-function
>  org-src-fontify-natively t
>  org-html-format-inlinetask-function 'ignore
>  org-completion-use-iswitchb t
>  org-tab-first-hook '(org-hide-block-toggle-maybe
> org-src-native-tab-command-maybe
>      org-babel-hide-result-toggle-maybe org-babel-header-arg-expand)
>  org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers
>  org-cycle-hide-inline-tasks org-cycle-show-empty-lines
>  org-optimize-window-after-visibility-change)
>  org-agenda-before-write-hook '(org-agenda-add-entry-text)
>  org-confirm-babel-evaluate nil
>  org-speed-command-hook '(org-speed-command-default-hook
> org-babel-speed-command-hook)
>  org-ascii-format-inlinetask-function 'org-ascii-format-inlinetask-default
>  org-babel-pre-tangle-hook '(save-buffer)
>  org-occur-hook '(org-first-headline-recenter)
>  org-src-lang-modes '(("r" . ess-mode) ("ocaml" . tuareg) ("elisp" . emacs-lisp)
>      ("ditaa" . artist) ("asymptote" . asy) ("dot" . fundamental)
>      ("sqlite" . sql) ("calc" . fundamental) ("C" . c) ("cpp" . c++)
>      ("C++" . c++) ("screen" . shell-script))
>  org-shiftdown-final-hook '(windmove-down)
>  org-html-format-headline-function 'ignore
>  org-log-done 'time
>  org-metaup-hook '(org-babel-load-in-session-maybe)
>  org-confirm-elisp-link-function 'yes-or-no-p
>  org-catch-invisible-edits 'smart
>  org-default-notes-file "~/.org//notes.org"
>  org-latex-format-drawer-function '(lambda (name contents) contents)
>  org-src-tab-acts-natively t
>  org-export-backends '(ascii html icalendar latex md)
>  org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
>  org-completion-use-ido t
>  org-use-speed-commands t
>  org-mode-hook '(#[nil "\300\301\302\303\304$\207"
>   [org-add-hook change-major-mode-hook org-show-block-all append local] 5]
> #[nil "\300\301\302\303\304$\207"
>   [org-add-hook change-major-mode-hook org-babel-show-result-all
> append local] 5]
> org-babel-result-hide-spec org-babel-hide-all-hashes org-mode-reftex-setup
> (lambda nil (flyspell-mode 1) (auto-fill-mode 1)
>  (local-set-key "\220" (quote outline-previous-visible-heading))
>  (local-set-key "\225" (quote outline-up-heading))
>  (local-set-key "\227" (quote org-table-copy-region))
>  (local-set-key "\231" (quote org-table-paste-rectangle))
>  (local-set-key "\214" (quote org-table-sort-lines))
>  (local-set-key "\311" (quote org-toggle-iimage-in-org)))
> )
>  org-shiftup-final-hook '(windmove-up)
>  org-ascii-format-drawer-function '(lambda (name contents width) contents)
>  org-directory "~/.org/"
>  org-html-format-drawer-function '(lambda (name contents) contents)
>  org-shiftleft-final-hook '(windmove-left)
>  org-metadown-hook '(org-babel-pop-to-session-maybe)
>  org-agenda-files '("~/.org/agenda.org")
>  org-src-mode-hook '(org-src-babel-configure-edit-buffer
> org-src-mode-configure-edit-buffer)
>  org-after-todo-state-change-hook '(org-clock-out-if-current)
>  org-todo-keyword-faces '(("WAITING" . ak-org-todo-warning-face))
>  org-shiftright-final-hook '(windmove-right)
>  org-babel-load-languages '((emacs-lisp) (R . t))
>  org-babel-R-command "C:/Progra~1/R/R-3.1.1/bin/x64/R --slave --no-save"
>  org-latex-format-inlinetask-function 'ignore
>  org-confirm-shell-link-function 'yes-or-no-p
>  )
>

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: R code block produces only partial output

[Andreas Kiermeier]

On 4 August 2014 21:23, Eric Schulte <schulte.eric@gmail.com> wrote:
> Why are you setting the output type to "graphics" when you are trying to
> return text?  I think that may be the source of your problem.

Hi Eric,
thanks for the quick response.
I've had 'graphics' from my main data analysis file ... I think I got
that from WORG.
I've removed it, reloaded the file, but same outcome.
Any other thoughts?
Cheers,
Andreas

Re: R code block produces only partial output

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

Andreas Kiermeier <andreas.kiermeier@gmail.com> writes:

> On 4 August 2014 21:23, Eric Schulte <schulte.eric@gmail.com> wrote:
>> Why are you setting the output type to "graphics" when you are trying to
>> return text?  I think that may be the source of your problem.
>
> Hi Eric,
> thanks for the quick response.
> I've had 'graphics' from my main data analysis file ... I think I got
> that from WORG.
> I've removed it, reloaded the file, but same outcome.
> Any other thoughts?
> Cheers,
> Andreas

Hi Andreas,

I can't reproduce your problem.  I get the following from your minimal
example when run in an Org-mode file, and from the command line.  They
are identical.  Are you using the latest version of Org-mode?

Best,
Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: r-example.org --]
[-- Type: text/x-org, Size: 452 bytes --]

#+BEGIN_SRC R :results output
  x <- rnorm(100)
  y <- quantile(x, probs=seq(0,1,0.1))
  names(y) <- as.character(c("0",".1",".2",".3",".4",".5",".6",".7",".8",".9","1"))
  y
#+END_SRC

#+RESULTS:
:          0         .1         .2         .3         .4         .5         .6 
: -2.5388527 -1.7039145 -1.0586655 -0.4892375 -0.2083433  0.0669765  0.2674644 
:         .7         .8         .9          1 
:  0.6065544  1.0985117  1.5376634  3.2423323 


[-- Attachment #3: r-example-command-line.txt --]
[-- Type: text/plain, Size: 1151 bytes --]

$ R

R version 3.1.0 (2014-04-10) -- "Spring Dance"
Copyright (C) 2014 The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

[Previously saved workspace restored]

> x <- rnorm(100)
> y <- quantile(x, probs=seq(0,1,0.1))
> names(y) <- as.character(c("0",".1",".2",".3",".4",".5",".6",".7",".8",".9","1"))
> y
          0          .1          .2          .3          .4          .5 
-2.53624773 -1.30846042 -0.70659822 -0.43565010 -0.24318346 -0.01034625 
         .6          .7          .8          .9           1 
 0.24125644  0.49945059  0.92032314  1.36423669  2.83357915 
>

[-- Attachment #4: Type: text/plain, Size: 90 bytes --]


-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: R code block produces only partial output

[Andreas Kiermeier]

Hi Eric,
I'm running Org 8.2.7c (via ELPA) and ESS 14.05.
I've tried to run with a minimal Org setup using only the following in
my init file.
The shortcut I use to start Emacs (on Windows 8.1) is:
C:\emacs\bin\runemacs.exe -Q -l "C:\emacs\bin\minimal-org.el"

However, the results are the same as before. Any thoughts on what else
I could try?

---------
;;; Minimal setup to load latest `org-mode'

;; activate debugging
(setq debug-on-error t
      debug-on-signal nil
      debug-on-quit nil)

;; add latest org-mode to load path
(add-to-list 'load-path (expand-file-name
"C:/Users/andre_000/Documents/.emacs.d/elpa/org-20140804/"))
(add-to-list 'load-path (expand-file-name
"C:/Users/andre_000/Documents/.emacs.d/elpa/ess-20140716.2033/lisp/"))

(require 'ess-site)

(setq org-babel-R-command "C:/Progra~1/R/R-3.1.1/bin/x64/R --slave --no-save")

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . nil)
   (R . t)))
--------

Thanks,
Andreas
--
Dr Andreas Kiermeier | Director
Statistical Process Improvement Consulting and Training Pty Ltd
Mbl: +61 (4)23 028 565 | Email: andreas.kiermeier@gmail.com
Australia


On 4 August 2014 22:40, Eric Schulte <schulte.eric@gmail.com> wrote:
> Andreas Kiermeier <andreas.kiermeier@gmail.com> writes:
>
>> On 4 August 2014 21:23, Eric Schulte <schulte.eric@gmail.com> wrote:
>>> Why are you setting the output type to "graphics" when you are trying to
>>> return text?  I think that may be the source of your problem.
>>
>> Hi Eric,
>> thanks for the quick response.
>> I've had 'graphics' from my main data analysis file ... I think I got
>> that from WORG.
>> I've removed it, reloaded the file, but same outcome.
>> Any other thoughts?
>> Cheers,
>> Andreas
>
> Hi Andreas,
>
> I can't reproduce your problem.  I get the following from your minimal
> example when run in an Org-mode file, and from the command line.  They
> are identical.  Are you using the latest version of Org-mode?
>
> Best,
> Eric
>
>
> $ R
>
> R version 3.1.0 (2014-04-10) -- "Spring Dance"
> Copyright (C) 2014 The R Foundation for Statistical Computing
> Platform: x86_64-unknown-linux-gnu (64-bit)
>
> R is free software and comes with ABSOLUTELY NO WARRANTY.
> You are welcome to redistribute it under certain conditions.
> Type 'license()' or 'licence()' for distribution details.
>
>   Natural language support but running in an English locale
>
> R is a collaborative project with many contributors.
> Type 'contributors()' for more information and
> 'citation()' on how to cite R or R packages in publications.
>
> Type 'demo()' for some demos, 'help()' for on-line help, or
> 'help.start()' for an HTML browser interface to help.
> Type 'q()' to quit R.
>
> [Previously saved workspace restored]
>
>> x <- rnorm(100)
>> y <- quantile(x, probs=seq(0,1,0.1))
>> names(y) <- as.character(c("0",".1",".2",".3",".4",".5",".6",".7",".8",".9","1"))
>> y
>           0          .1          .2          .3          .4          .5
> -2.53624773 -1.30846042 -0.70659822 -0.43565010 -0.24318346 -0.01034625
>          .6          .7          .8          .9           1
>  0.24125644  0.49945059  0.92032314  1.36423669  2.83357915
>>
>
>
> --
> Eric Schulte
> https://cs.unm.edu/~eschulte
> PGP: 0x614CA05D (see https://u.fsf.org/yw)
>

Re: R code block produces only partial output

[John Hendy]

On Mon, Aug 4, 2014 at 7:46 PM, Andreas Kiermeier
<andreas.kiermeier@gmail.com> wrote:
> Hi Eric,
> I'm running Org 8.2.7c (via ELPA) and ESS 14.05.
> I've tried to run with a minimal Org setup using only the following in
> my init file.
> The shortcut I use to start Emacs (on Windows 8.1) is:
> C:\emacs\bin\runemacs.exe -Q -l "C:\emacs\bin\minimal-org.el"
>
> However, the results are the same as before. Any thoughts on what else
> I could try?
>

[snip]

I can reproduce with a minimal config and on Linux (to toss in a
confirmation). I *also* accidentally just stumbled on a way I can
toggle the behavior between correct and incorrect! It arose because my
file was not able to use the #+PROPERTY settings you have -- my
results were not obeying what I had set. I looked around and believe
the syntax is correct per this page:
- http://orgmode.org/manual/Header-arguments-in-Org-mode-properties.html

However, I guarantee that the results weren't updating to ":results
output drawer" as I had them set. I also noticed that the
"header-args:R :session *R*" also didn't seem to be working, as if I
changed the name, Emacs wouldn't ask me if I wanted to create a new R
session upon block execution. Here's what I've found:

Correct results:

#+begin_src org

#+STARTUP: showall indent hidestars
#+PROPERTY: header-args:R :session r

#+BEGIN_SRC R :results output drawer
  require(rms)
  set.seed(123)
  x <- rnorm(100)
  describe(x)
#+END_SRC

#+RESULTS:
:RESULTS:
x
       n  missing   unique     Mean      .05      .10      .25      .50
     100        0      100  0.09041 -1.26508 -1.06822 -0.49385  0.06176
     .75      .90      .95
 0.69182  1.26450  1.56653

lowest : -2.309 -1.967 -1.687 -1.549 -1.265
highest:  1.715  1.787  2.050  2.169  2.187
:END:


#+end_src


Incorrect results:

#+begin_src org

#+STARTUP: showall indent hidestars


#+BEGIN_SRC R :session r :results output drawer
  require(rms)
  set.seed(123)
  x <- rnorm(100)
  describe(x)
#+END_SRC

#+RESULTS:
:RESULTS:
75      .90      .95
 0.69182  1.26450  1.56653

lowest : -2.309 -1.967 -1.687 -1.549 -1.265
highest:  1.715  1.787  2.050  2.169  2.187
:END:

#+end_src


The reason, I think, is this (also correct):

#+begin_src org

#+STARTUP: showall indent hidestars


#+BEGIN_SRC R :results output drawer
  require(rms)
  set.seed(123)
  x <- rnorm(100)
  describe(x)
#+END_SRC

#+RESULTS:
:RESULTS:
x
       n  missing   unique     Mean      .05      .10      .25      .50
     100        0      100  0.09041 -1.26508 -1.06822 -0.49385  0.06176
     .75      .90      .95
 0.69182  1.26450  1.56653

lowest : -2.309 -1.967 -1.687 -1.549 -1.265
highest:  1.715  1.787  2.050  2.169  2.187
:END:

#+end_src

So, session settings in the #+PROPERTY config *or* no session set at
all yields correct results. Again, on a fresh start the #+PROPERTY
method does not ask to start a new R session. Last confirmation is
that if one leaves the session #+PROPERTY line (with none in the code
block) and comments out the "require(rms)" line after already having
run the block, you get the error "Could not find the function
'describe'". So, there's no session activity finding the
already-loaded library.

The remaining questions for me is why would results differ depending
on whether or not it's going through a session?

Eric: this also explains why your minimal .org file couldn't
reproduce: there's no setting of the session (again, not sure why that
matters... but it seems to).


John

Re: R code block produces only partial output

[Andreas Kiermeier]

On 5 August 2014 13:30, John Hendy <jw.hendy@gmail.com> wrote:
> I can reproduce with a minimal config and on Linux (to toss in a
> confirmation). I *also* accidentally just stumbled on a way I can
> toggle the behavior between correct and incorrect! It arose because my
> file was not able to use the #+PROPERTY settings you have -- my
> results were not obeying what I had set. I looked around and believe
> the syntax is correct per this page:
> - http://orgmode.org/manual/Header-arguments-in-Org-mode-properties.html

Thanks for confirming this John!
I couldn't see how the #+PROPERTY settings I had differed from those
in the link ... I recall getting them from there (but maybe I mistyped
something?). So I copied what you provided for the rest of my testing.

Getting rid of :session all together did produce the correct results
for me, too. However, including a :session in either #+PROPERTY or at
the beginning of the code block did not - irrespective of whether I
used *R* or simply r.

> Correct results:
>
> #+begin_src org
>
> #+STARTUP: showall indent hidestars
> #+PROPERTY: header-args:R :session r
>
> #+BEGIN_SRC R :results output drawer
>   require(rms)
>   set.seed(123)
>   x <- rnorm(100)
>   describe(x)
> #+END_SRC

[snip]

So, this example did *not* work for me.

> The reason, I think, is this (also correct):
>
> #+begin_src org
>
> #+STARTUP: showall indent hidestars
>
>
> #+BEGIN_SRC R :results output drawer
>   require(rms)
>   set.seed(123)
>   x <- rnorm(100)
>   describe(x)
> #+END_SRC

Yes, without a session it seems to work fine.

Thanks,

Andreas

Re: R code block produces only partial output

[Charles Berry]

Eric Schulte <schulte.eric <at> gmail.com> writes:

> 
> Andreas Kiermeier <andreas.kiermeier <at> gmail.com> writes:
> 
> > On 4 August 2014 21:23, Eric Schulte <schulte.eric <at> gmail.com> wrote:
> >> Why are you setting the output type to "graphics" when you are trying to

> 
> Hi Andreas,
> 
> I can't reproduce your problem.  I get the following from your minimal
> example when run in an Org-mode file, and from the command line.  They
> are identical.  Are you using the latest version of Org-mode?
> 
> Best,
> Eric
> 
[snip]
> 
> [Previously saved workspace restored]
> 
> > x <- rnorm(100)
> > y <- quantile(x, probs=seq(0,1,0.1))
> > names(y) <- 
as.character(c("0",".1",".2",".3",".4",".5",".6",".7",".8",".9","1"))
> > y
>           0          .1          .2          .3          .4          .5 
> -2.53624773 -1.30846042 -0.70659822 -0.43565010 -0.24318346 -0.01034625 
>          .6          .7          .8          .9           1 
>  0.24125644  0.49945059  0.92032314  1.36423669  2.83357915 
> >
> 
> 
> 


Eric,

As noted by Andreas and John this is a problem for session output.

org-babel-R-evaluate-session uses 

 (string-match "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)

to find the start of R output in the session.

This does not match the `          0', but matches the `         .6' 
in the output you show above, so if that had been in a session, all the 
output up to and including the '.' before the '6' would be clipped
by the following

  (substring line (match-end 1))


as Andreas output showed. 

Deleting the "\\." fixes Andreas case, but what are the circumstances 
requiring the  "\\." ?

HTH,

Chuck

Re: R code block produces only partial output

[Eric Schulte]

Charles Berry <ccberry@ucsd.edu> writes:

> Eric Schulte <schulte.eric <at> gmail.com> writes:
>
>> 
>> Andreas Kiermeier <andreas.kiermeier <at> gmail.com> writes:
>> 
>> > On 4 August 2014 21:23, Eric Schulte <schulte.eric <at> gmail.com> wrote:
>> >> Why are you setting the output type to "graphics" when you are trying to
>
>> 
>> Hi Andreas,
>> 
>> I can't reproduce your problem.  I get the following from your minimal
>> example when run in an Org-mode file, and from the command line.  They
>> are identical.  Are you using the latest version of Org-mode?
>> 
>> Best,
>> Eric
>> 
> [snip]
>> 
>> [Previously saved workspace restored]
>> 
>> > x <- rnorm(100)
>> > y <- quantile(x, probs=seq(0,1,0.1))
>> > names(y) <- 
> as.character(c("0",".1",".2",".3",".4",".5",".6",".7",".8",".9","1"))
>> > y
>>           0          .1          .2          .3          .4          .5 
>> -2.53624773 -1.30846042 -0.70659822 -0.43565010 -0.24318346 -0.01034625 
>>          .6          .7          .8          .9           1 
>>  0.24125644  0.49945059  0.92032314  1.36423669  2.83357915 
>> >
>> 
>> 
>> 
>
>
> Eric,
>
> As noted by Andreas and John this is a problem for session output.
>
> org-babel-R-evaluate-session uses 
>
>  (string-match "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
>
> to find the start of R output in the session.
>
> This does not match the `          0', but matches the `         .6' 
> in the output you show above, so if that had been in a session, all the 
> output up to and including the '.' before the '6' would be clipped
> by the following
>
>   (substring line (match-end 1))
>
>
> as Andreas output showed. 
>
> Deleting the "\\." fixes Andreas case, but what are the circumstances 
> requiring the  "\\." ?
>

I don't know.

The Babel R support needs a dedicated maintainer.  This was a role Dan
Davison originally filled.  I've partially filled in since, but as a
*very* light R user I'm not the best person.

>
> HTH,
>
> Chuck
>
>

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: R code block produces only partial output

[John Hendy]

On Tue, Aug 5, 2014 at 2:02 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
> Charles Berry <ccberry@ucsd.edu> writes:
>
>> Eric Schulte <schulte.eric <at> gmail.com> writes:

[snip]

>> Eric,
>>
>> As noted by Andreas and John this is a problem for session output.
>>
>> org-babel-R-evaluate-session uses
>>
>>  (string-match "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
>>
>> to find the start of R output in the session.
>>
>> This does not match the `          0', but matches the `         .6'
>> in the output you show above, so if that had been in a session, all the
>> output up to and including the '.' before the '6' would be clipped
>> by the following
>>
>>   (substring line (match-end 1))
>>
>>
>> as Andreas output showed.
>>
>> Deleting the "\\." fixes Andreas case, but what are the circumstances
>> requiring the  "\\." ?
>>
>
> I don't know.

I'm not sure either, but was curious if someone could translate the
regex into "plain language." Maybe I could observe some typical
outputs and chime in since I use R regularly? From noob-level regex
stuff, it's looking for a new line followed by some number of spaces,
a ">" and at least one period and numbers?

If I'm recalling correctly, I also noticed that babel spit out the
output from loading the rms package on the first run with :session,
but I don't think it did it with non-session. I'll have to try that
again to be sure. Re-running the block removed the package load
message.


John

>
> The Babel R support needs a dedicated maintainer.  This was a role Dan
> Davison originally filled.  I've partially filled in since, but as a
> *very* light R user I'm not the best person.
>
>>
>> HTH,
>>
>> Chuck
>>
>>
>
> --
> Eric Schulte
> https://cs.unm.edu/~eschulte
> PGP: 0x614CA05D (see https://u.fsf.org/yw)
>

Re: R code block produces only partial output

[Nick Dokos]

John Hendy <jw.hendy@gmail.com> writes:

> On Tue, Aug 5, 2014 at 2:02 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>> Charles Berry <ccberry@ucsd.edu> writes:
>>
>>> Eric Schulte <schulte.eric <at> gmail.com> writes:
>
> [snip]
>
>>> Eric,
>>>
>>> As noted by Andreas and John this is a problem for session output.
>>>
>>> org-babel-R-evaluate-session uses
>>>
>>>  (string-match "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
>>>
>>> to find the start of R output in the session.
>>>
>>> This does not match the `          0', but matches the `         .6'
>>> in the output you show above, so if that had been in a session, all the
>>> output up to and including the '.' before the '6' would be clipped
>>> by the following
>>>
>>>   (substring line (match-end 1))
>>>
>>>
>>> as Andreas output showed.
>>>
>>> Deleting the "\\." fixes Andreas case, but what are the circumstances
>>> requiring the  "\\." ?
>>>
>>
>> I don't know.
>
> I'm not sure either, but was curious if someone could translate the
> regex into "plain language." Maybe I could observe some typical
> outputs and chime in since I use R regularly? From noob-level regex
> stuff, it's looking for a new line followed by some number of spaces,
> a ">" and at least one period and numbers?
>

It says[fn:1]

^                          anchor the match at the beginning of the line

\\([ ]*[>+\\.][ ]?\\)+     match any number of spaces followed by one of
                           the three characters >, + or . (a literal
                           period) followed by 0 or 1 space. If there is
                           a match, remember what is matched as group 1
                           (that's what the escaped parentheses
                           \\(...\\) do). Match one or more of these
                           (that's what the + at the end does).

\\([[0-9]+\\|[ ]\\)        match either an emtpy space or a sequence
                           of one or more of the characters [ or 0-9
                           i.e. an opening square bracket or a digit.
                           remember what is matched as group 2.

The latter will match [0[1[2[3 e.g. which does not sound right.

The best way to find out what a regexp will match is to start with
a buffer containing example strings that you are trying to match
and example string that you are trying *not* to match, then invoke

    M-x regexp-builder

and paste the regexp inside the empty set of quotes, then check the highligted
matches to see if they agree with your expectations.

Footnotes:

[fn:1] Crossing fingers and toes, hoping I've got it right...

-- 
Nick

Re: R code block produces only partial output

[Nick Dokos]

Nick Dokos <ndokos@gmail.com> writes:

> John Hendy <jw.hendy@gmail.com> writes:
>
>> On Tue, Aug 5, 2014 at 2:02 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>>> Charles Berry <ccberry@ucsd.edu> writes:
>>>
>>>> Eric Schulte <schulte.eric <at> gmail.com> writes:
>>
>> [snip]
>>
>>>> Eric,
>>>>
>>>> As noted by Andreas and John this is a problem for session output.
>>>>
>>>> org-babel-R-evaluate-session uses
>>>>
>>>>  (string-match "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
>>>>
>>>> to find the start of R output in the session.
>>>>
>>>> This does not match the `          0', but matches the `         .6'
>>>> in the output you show above, so if that had been in a session, all the
>>>> output up to and including the '.' before the '6' would be clipped
>>>> by the following
>>>>
>>>>   (substring line (match-end 1))
>>>>
>>>>
>>>> as Andreas output showed.
>>>>
>>>> Deleting the "\\." fixes Andreas case, but what are the circumstances
>>>> requiring the  "\\." ?
>>>>
>>>
>>> I don't know.
>>
>> I'm not sure either, but was curious if someone could translate the
>> regex into "plain language." Maybe I could observe some typical
>> outputs and chime in since I use R regularly? From noob-level regex
>> stuff, it's looking for a new line followed by some number of spaces,
>> a ">" and at least one period and numbers?
>>
>
> It says[fn:1]
>
> ^                          anchor the match at the beginning of the line
>
> \\([ ]*[>+\\.][ ]?\\)+     match any number of spaces followed by one of
>                            the three characters >, + or . (a literal
>                            period) followed by 0 or 1 space. If there is
>                            a match, remember what is matched as group 1
>                            (that's what the escaped parentheses
>                            \\(...\\) do). Match one or more of these
>                            (that's what the + at the end does).
>
> \\([[0-9]+\\|[ ]\\)        match either an emtpy space or a sequence
>                            of one or more of the characters [ or 0-9
>                            i.e. an opening square bracket or a digit.
>                            remember what is matched as group 2.
>
> The latter will match [0[1[2[3 e.g. which does not sound right.
>
> The best way to find out what a regexp will match is to start with
> a buffer containing example strings that you are trying to match
> and example string that you are trying *not* to match, then invoke
>
>     M-x regexp-builder
>
> and paste the regexp inside the empty set of quotes, then check the highligted
> matches to see if they agree with your expectations.
>
> Footnotes:
>
> [fn:1] Crossing fingers and toes, hoping I've got it right...

Bah, got it wrong: backslash is not special within a character class, so

[>+\\.]

matches > or + or \ or . (a period - which is also not special in a
character class).

I'm not sure what the regexp is supposed to match, but I'm almost sure
that it is wrong :-)

-- 
Nick

Re: R code block produces only partial output

[Charles C. Berry]

On Tue, 5 Aug 2014, John Hendy wrote:

> On Tue, Aug 5, 2014 at 2:02 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>> Charles Berry <ccberry@ucsd.edu> writes:
>>
>>> Eric Schulte <schulte.eric <at> gmail.com> writes:
>
> [snip]
>
>>> Eric,
>>>
>>> As noted by Andreas and John this is a problem for session output.
>>>
>>> org-babel-R-evaluate-session uses
>>>
>>>  (string-match "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
>>>
>>> to find the start of R output in the session.
>>>
>>> This does not match the `          0', but matches the `         .6'
>>> in the output you show above, so if that had been in a session, all the
>>> output up to and including the '.' before the '6' would be clipped
>>> by the following
>>>
>>>   (substring line (match-end 1))
>>>
>>>
>>> as Andreas output showed.
>>>
>>> Deleting the "\\." fixes Andreas case, but what are the circumstances
>>> requiring the  "\\." ?
>>>
>>
>> I don't know.
>
> I'm not sure either, but was curious if someone could translate the
> regex into "plain language." Maybe I could observe some typical
> outputs and chime in since I use R regularly? From noob-level regex
> stuff, it's looking for a new line followed by some number of spaces,
> a ">" and at least one period and numbers?
>


One of > or + or . and then some more.

You can get the 'English' at

 	http://www.myezapp.com/apps/dev/regexp/show.ws


But a better approach IMO is to open an R transcript, then M-x 
regex-builder and paste that regex into the window in place of the "". The 
matches should highlight.

Or use M-x occur (and edit the string to rid quotes and extra 
backslashes). But that will not show what you are missing.

Chuck

Re: R code block produces only partial output

[Aaron Ecay]

Hi Eric,

2014ko abuztuak 5an, Eric Schulte-ek idatzi zuen:
> 

[...]

> I don't know.
> 
> The Babel R support needs a dedicated maintainer.  This was a role Dan
> Davison originally filled.  I've partially filled in since, but as a
> *very* light R user I'm not the best person.

I use babel’s R support (almost) every day for my work, and I know
enough elisp to be getting along.  I’ve not had tons of time for
org/emacs development lately, but I think I can make enough time to
maintain ob-R.el.

There are others who could probably do just as good (if not indeed
better) a job.  Rainer’s name springs to mind as someone who has
submitted several patches for ob-R lately, but there are probably
others who I’m not calling to mind immediately.  I’d be happy to
share with other willing volunteers, either as a joint maintainership
or (perhaps preferably) alternating the position in a 3- to 6-month
rota.

-- 
Aaron Ecay

Re: R code block produces only partial output

[Eric Schulte]

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Eric,
>
> 2014ko abuztuak 5an, Eric Schulte-ek idatzi zuen:
>> 
>
> [...]
>
>> I don't know.
>> 
>> The Babel R support needs a dedicated maintainer.  This was a role Dan
>> Davison originally filled.  I've partially filled in since, but as a
>> *very* light R user I'm not the best person.
>
> I use babel’s R support (almost) every day for my work, and I know
> enough elisp to be getting along.  I’ve not had tons of time for
> org/emacs development lately, but I think I can make enough time to
> maintain ob-R.el.
>

That would be great, and a personal relief.  Perhaps you could begin
with a patch for the regexp issue in this thread?

>
> There are others who could probably do just as good (if not indeed
> better) a job.  Rainer’s name springs to mind as someone who has
> submitted several patches for ob-R lately, but there are probably
> others who I’m not calling to mind immediately.  I’d be happy to
> share with other willing volunteers, either as a joint maintainership
> or (perhaps preferably) alternating the position in a 3- to 6-month
> rota.

Yes, and some form of sharing would of course be great.  Multiple active
maintainers is better than just one.  I think the main thing is to have
someone who, in threads like this one with many good suggestions, will
take the initiative to move from discussion to a proposed patch, and
then to apply that patch down the line.

Thanks!
Eric

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D (see https://u.fsf.org/yw)

Re: R code block produces only partial output

[Aaron Ecay]

Hi Eric,

2014ko abuztuak 6an, Eric Schulte-ek idatzi zuen:

[...]

> Perhaps you could begin with a patch for the regexp issue in this
> thread?

I have pushed a patch which allows us to avoid the regex issue
entirely by using a native R method to capture the session output to a
file.

This introduces the change that the output no longer appears in the
session buffer, but I think that’s actually an improvement: we were not
previously echoing the commands to the buffer, such that the output
would show up “out of the blue” without any indication of how it got
there.

-- 
Aaron Ecay

Re: R code block produces only partial output

[Charles C. Berry]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2388 bytes --]

On Wed, 6 Aug 2014, Aaron Ecay wrote:

> Hi Eric,
>
> 2014ko abuztuak 6an, Eric Schulte-ek idatzi zuen:
>
> [...]
>
>> Perhaps you could begin with a patch for the regexp issue in this
>> thread?
>
> I have pushed a patch which allows us to avoid the regex issue
> entirely by using a native R method to capture the session output to a
> file.
>
> This introduces the change that the output no longer appears in the
> session buffer, but I think that’s actually an improvement: we were not
> previously echoing the commands to the buffer, such that the output
> would show up “out of the blue” without any indication of how it got
> there.

Hi Aaron,

I like what you are trying to do, but ...

1) The change has at least one bug: Remote sessions are broken by this 
change.

2) The behavior of :results output is modified in ways that might not be 
desired. i.e. warnings and errors will not show up in the output.

Can you revert this change until the bugs are sorted out and consensus 
about the proper handling of cases like '2' is reached?

Can I also suggest that in the future before a change is pushed, that the 
patch is announced so we can try it out or at least eyeball it and discuss
issues/bugs?

Details:

Issue 1) ===========

If I open a *.org file on a remote machine and C-c C-c on a src block that 
has `:session :results output', after the usual session startup the src 
block fails. The session buffer shows this

====
Error in file(file, if (append) "a" else "w") :
   cannot open the connection
In addition: Warning message:
In file(file, if (append) "a" else "w") :
   cannot open file '/scpc:berry@<DELETED.URL>:/tmp/R-1155xWV': 
No such file or directory
> 
===

The file '/tmp/R-1155xWV' was created.

I think if the tramp file localname is used. it might work. I do not know 
tramp, but maybe something like

  (let output-file-localname
   (if (tramp-tramp-file-p output-file)
       (tramp-file-name-localname
        (tramp-dissect-file-name output-file))
     output-file))

is good enough.


Issue 2) ===========

ECM:

#+NAME: aa
#+BEGIN_SRC R :session R2 :results output
warning("this is a warning")
1+1
#+END_SRC

#+RESULTS: aa
: [1] 2

For some purposes having the warnings in the #+RESULTS: block is helpful.

And when revising code, having the errors in the #+RESULTS helps - 
especially if I have to put aside work in progress.

HTH,

Chuck

Re: R code block produces only partial output

[Aaron Ecay]

Hi Chuck,

Thanks for your feedback.

2014ko abuztuak 7an, "Charles C. Berry"-ek idatzi zuen:
> Hi Aaron,
> 
> I like what you are trying to do, but ...
> 
> 1) The change has at least one bug: Remote sessions are broken by this 
> change.
> 
> 2) The behavior of :results output is modified in ways that might not be 
> desired. i.e. warnings and errors will not show up in the output.
> 
> Can you revert this change until the bugs are sorted out and consensus 
> about the proper handling of cases like '2' is reached?

OK.

> 
> Can I also suggest that in the future before a change is pushed, that the 
> patch is announced so we can try it out or at least eyeball it and discuss
> issues/bugs?

Good idea.

> 
> Details:
> 
> Issue 1) ===========
> 
> If I open a *.org file on a remote machine and C-c C-c on a src block that 
> has `:session :results output', after the usual session startup the src 
> block fails. The session buffer shows this
> 
> ====
> Error in file(file, if (append) "a" else "w") :
>    cannot open the connection
> In addition: Warning message:
> In file(file, if (append) "a" else "w") :
>    cannot open file '/scpc:berry@<DELETED.URL>:/tmp/R-1155xWV': 
> No such file or directory
>> 
> ===
> 
> The file '/tmp/R-1155xWV' was created.
> 
> I think if the tramp file localname is used. it might work. I do not know 
> tramp, but maybe something like
> 
>   (let output-file-localname
>    (if (tramp-tramp-file-p output-file)
>        (tramp-file-name-localname
>         (tramp-dissect-file-name output-file))
>      output-file))
> 
> is good enough.

This looks promising – I’ll work on it.

> 
> 
> Issue 2) ===========
> 
> ECM:
> 
> #+NAME: aa
> #+BEGIN_SRC R :session R2 :results output
> warning("this is a warning")
> 1+1
> #+END_SRC
> 
> #+RESULTS: aa
> : [1] 2
> 
> For some purposes having the warnings in the #+RESULTS: block is helpful.
> 
> And when revising code, having the errors in the #+RESULTS helps - 
> especially if I have to put aside work in progress.

Hmm.  Certainly, the previous behavior should be retained for now.  In
the longer term, I’d like to see a system whereby R errors trigger elisp
errors.  This is so that the execution of a whole document (subtree,
etc.) will be halted by the first error, rather than continuing what may
be a long series of commands that will not give valid output.  What do
you think?

Thanks,

-- 
Aaron Ecay

Re: R code block produces only partial output

[Charles C. Berry]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1934 bytes --]

On Thu, 7 Aug 2014, Aaron Ecay wrote:

> Hi Chuck,
>
> Thanks for your feedback.
>
> 2014ko abuztuak 7an, "Charles C. Berry"-ek idatzi zuen:
>> Hi Aaron,
>>
>> I like what you are trying to do, but ...
>>
>> 1) The change has at least one bug: Remote sessions are broken by this
>> change.
>>
>> 2) The behavior of :results output is modified in ways that might not be
>> desired. i.e. warnings and errors will not show up in the output.
>>

[snip issue 1 discussion]

>>
>> Issue 2) ===========
>>
>> ECM:
>>
>> #+NAME: aa
>> #+BEGIN_SRC R :session R2 :results output
>> warning("this is a warning")
>> 1+1
>> #+END_SRC
>>
>> #+RESULTS: aa
>> : [1] 2
>>
>> For some purposes having the warnings in the #+RESULTS: block is helpful.
>>
>> And when revising code, having the errors in the #+RESULTS helps -
>> especially if I have to put aside work in progress.
>
> Hmm.  Certainly, the previous behavior should be retained for now.  In
> the longer term, I’d like to see a system whereby R errors trigger elisp
> errors.  This is so that the execution of a whole document (subtree,
> etc.) will be halted by the first error, rather than continuing what may
> be a long series of commands that will not give valid output.  What do
> you think?
>

I need a while to sort through this. stop(), warning(), and message() will 
print to the session but not show up in what capture.output retains.

sink() has the ability to capture those things, but there is added 
baggage.

I fear some study of ?conditions is needed. My knowledge of condition 
handling in R is scant.

As for stopping on error, I think that anything that changes current 
behavior at this late date ought to be configurable.

FWIW, when I export documents, I sometimes get innocuous errors that I am 
happy did not stop the run in its tracks - like formatting one table fails 
with an error but all else went through. And sometimes I wish it had 
stopped.

Best,

Chuck

Re: R code block produces only partial output

[Thomas S. Dye]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> As for stopping on error, I think that anything that changes current
> behavior at this late date ought to be configurable.

+1, e.g., silent, message, stop.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: R code block produces only partial output

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 3227 bytes --]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> On Thu, 7 Aug 2014, Aaron Ecay wrote:
>
>> Hi Chuck,
>>
>> Thanks for your feedback.
>>
>> 2014ko abuztuak 7an, "Charles C. Berry"-ek idatzi zuen:
>>> Hi Aaron,
>>>
>>> I like what you are trying to do, but ...
>>>
>>> 1) The change has at least one bug: Remote sessions are broken by this
>>> change.
>>>
>>> 2) The behavior of :results output is modified in ways that might not be
>>> desired. i.e. warnings and errors will not show up in the output.
>>>
>
> [snip issue 1 discussion]
>
>>>
>>> Issue 2) ===========
>>>
>>> ECM:
>>>
>>> #+NAME: aa
>>> #+BEGIN_SRC R :session R2 :results output
>>> warning("this is a warning")
>>> 1+1
>>> #+END_SRC
>>>
>>> #+RESULTS: aa
>>> : [1] 2
>>>
>>> For some purposes having the warnings in the #+RESULTS: block is helpful.
>>>
>>> And when revising code, having the errors in the #+RESULTS helps -
>>> especially if I have to put aside work in progress.
>>
>> Hmm.  Certainly, the previous behavior should be retained for now.  In
>> the longer term, I’d like to see a system whereby R errors trigger elisp
>> errors.  This is so that the execution of a whole document (subtree,
>> etc.) will be halted by the first error, rather than continuing what may
>> be a long series of commands that will not give valid output.  What do
>> you think?
>>
>
> I need a while to sort through this. stop(), warning(), and message()
> will print to the session but not show up in what capture.output
> retains.
>
> sink() has the ability to capture those things, but there is added
> baggage.
>
> I fear some study of ?conditions is needed. My knowledge of condition
> handling in R is scant.
>
> As for stopping on error, I think that anything that changes current
> behavior at this late date ought to be configurable.
>
> FWIW, when I export documents, I sometimes get innocuous errors that I
> am happy did not stop the run in its tracks - like formatting one
> table fails with an error but all else went through. And sometimes I
> wish it had stopped.

I agree here with Charles. An example where R errors do *not* trigger do
not abort export are graphs: the graph generation, when using R, is
wrapped in a tryCatch() block and shows the error in the exported
document as well as in the R session but does not abort. 

A case where it is very useful to continue export is when creating
lecture notes or slides to demonstrate errors.

If there is the wish for R errors to trigger elisp errors, a header
argument would be needed to specify if errors and warnings in R should
trigger errors in elisp or if they should be ignored and the error
message displayed and the export should be continued.

Cheers,

Rainer

>
> Best,
>
> Chuck

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

[-- Attachment #2: Type: application/pgp-signature, Size: 494 bytes --]

Re: R code block produces only partial output

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

Hi Chuck, hi all,

Attached to this message is a draft patch to complete this idea.  It
should address the issue of including warnings, errors, and messages in
output.

It uses the “evaluate” R package, originally created for the knitr
literate programming system, to do most of the heavy lifting, and
includes a new mechanism to load this package into babel sessions (and
to prompt the user to install it from CRAN, if not already installed).

It does not yet address Chuck’s first issue of remote sessions, just
because I haven’t had time to look at it in detail (but I still think it
should be easy, based on the code sample/hint that Chuck gave).

What do you think of the approach broadly?

[-- Attachment #2: 0001-ob-R-overhaul-handling-of-output-results-type-in-a-s.patch --]
[-- Type: text/x-diff, Size: 5051 bytes --]

From ea99c10d148f71e5384144728a5714ecc5ae47ca Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Sat, 16 Aug 2014 00:49:05 -0400
Subject: [PATCH] ob-R: overhaul handling of :output results type in a session
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/ob-R.el (org-babel-R-check-evaluate-package): New function.
(org-babel-R-initiate-session): Use it.
(org-babel-R-evaluate-session): Use the evaluate package to capture
session output.

This uses the “evaluate” R package[1] to capture the
output (incl. warnings, errors, and messages) from a babel R session.
This avoids the output showing up in the session buffer, and dodges
some previous issues with removing R prompts (>) when scraping the
output from the session buffer.

[1] <http://cran.r-project.org/web/packages/evaluate/index.html>
---
 lisp/ob-R.el | 68 +++++++++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 49 insertions(+), 19 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 41b943c..08ef9c2 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -245,6 +245,22 @@ This function is called by `org-babel-execute-src-block'."
 	  ((stringp  value) (format "%s <- %S" name (org-no-properties value)))
 	  (t                (format "%s <- %S" name (prin1-to-string value))))))
 
+(defvar ess-execute-in-process-buffer)
+(defun org-babel-R-check-evaluate-package (&optional recursive)
+  (save-window-excursion
+    (let ((ess-execute-in-process-buffer nil)
+	  (r-buff (current-buffer)))
+      (ess-execute "library(evaluate)" nil "org-babel-R-auto")
+      (when (with-current-buffer "*org-babel-R-auto*"
+	      (goto-char (point-min))
+	      (search-forward "Error" nil t))
+	(if (and (not recursive)
+		 (y-or-n-p "Cannot load the evaluate package required for babel session support, would you like to install it from CRAN?"))
+	    (progn
+	      (message "Downloading and installing package (may take some time)")
+	      (ess-execute "install.packages(\"evaluate\")" nil "org-babel-R-auto")
+	      (org-babel-R-check-evaluate-package t))
+	  (user-error "R package evaluate is required, but not available."))))))
 
 (defvar ess-ask-for-ess-directory) ; dynamically scoped
 (defun org-babel-R-initiate-session (session params)
@@ -261,7 +277,9 @@ This function is called by `org-babel-execute-src-block'."
 	  (when (get-buffer session)
 	    ;; Session buffer exists, but with dead process
 	    (set-buffer session))
-	  (require 'ess) (R)
+	  (require 'ess)
+	  (R)
+	  (org-babel-R-check-evaluate-package)
 	  (rename-buffer
 	   (if (bufferp session)
 	       (buffer-name session)
@@ -366,7 +384,7 @@ last statement in BODY, as elisp."
 (defvar ess-eval-visibly-p)
 
 (defun org-babel-R-evaluate-session
-  (session body result-type result-params column-names-p row-names-p)
+    (session body result-type result-params column-names-p row-names-p)
   "Evaluate BODY in SESSION.
 If RESULT-TYPE equals 'output then return standard output as a
 string.  If RESULT-TYPE equals 'value then return the value of the
@@ -396,23 +414,35 @@ last statement in BODY, as elisp."
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
     (output
-     (mapconcat
-      'org-babel-chomp
-      (butlast
-       (delq nil
-	     (mapcar
-	      (lambda (line) (when (> (length line) 0) line))
-	      (mapcar
-	       (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		      "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
-		     (substring line (match-end 1))
-		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+     (let* ((output-file (org-babel-temp-file "R-"))
+	    (sentinel-file (concat output-file "-sentinel")))
+       (org-babel-comint-eval-invisibly-and-wait-for-file
+	session sentinel-file
+	(format
+"
+tryCatch(.org.eval.result <- evaluate(%S, new_device = FALSE, stop_on_error = 0L),
+         finally = {
+             capture.output(
+                 withCallingHandlers(
+                     replay(Filter(function (x) !inherits(x, \"source\"),
+                                   .org.eval.result)),
+                     message = function (x) {
+                         cat(x$message);
+                         invokeRestart(\"muffleMessage\")
+                     }),
+                 file=%S)
+             file.create(%S)
+         })
+"
+		(org-babel-chomp body)
+		output-file
+		sentinel-file))
+       (with-temp-buffer
+	 (insert-file-contents output-file)
+	 (goto-char (point-min))
+	 (flush-lines "^$")
+	 (delete-trailing-whitespace)
+	 (buffer-string))))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
-- 
2.0.4


[-- Attachment #3: Type: text/plain, Size: 25 bytes --]


Thanks,

-- 
Aaron Ecay

Re: R code block produces only partial output

[Charles C. Berry]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2060 bytes --]

On Fri, 15 Aug 2014, Aaron Ecay wrote:

> Hi Chuck, hi all,
>
> Attached to this message is a draft patch to complete this idea.  It
> should address the issue of including warnings, errors, and messages in
> output.
>
> It uses the “evaluate” R package, originally created for the knitr
> literate programming system, to do most of the heavy lifting, and
> includes a new mechanism to load this package into babel sessions (and
> to prompt the user to install it from CRAN, if not already installed).
>
> It does not yet address Chuck’s first issue of remote sessions, just
> because I haven’t had time to look at it in detail (but I still think it
> should be easy, based on the code sample/hint that Chuck gave).
>
> What do you think of the approach broadly?
>

Aaron,

I think doing something along these lines makes sense. evaluate() obviates 
the need for tryCatch() and capture.output(), and it makes customizing 
error/warning/message stuff clean. Its use might also lead to cleaner code 
for R graphics handling.

But adding a dependency on the evaluate package is a significant step. 
This could be a nuisance for users whose code runs just fine right now. It 
would need to be installed anywhere R is executed, e.g. in remote 
sessions. The user would need to install it in her private directory if it 
is not on the system; some shops actually discourage this.

So, making it optional might be necessary.

---

I looked at the patch briefly. Some comments:

- You can ditch tryCatch and capture.output. You might browse
   knitr:::block_exec to see how it uses evaluate().

- Wrap the code in local() to keep objects you create from persisting
   where they might not be wanted. I think using
 	local({ res <- evaluate(input, envir=parent.frame(2),...); <...>})
   will get assignments from `input' properly placed.

- let stop_on_error, keep_warning, and keep_message args be customizable
   or depend on a header arg. (Then I can stop wrapping require() in
   suppressPackageStartupMessages() which I always misspell. Argh!)

HTH,

Chuck

Re: R code block produces only partial output

[Aaron Ecay]

Hi Chuck,

Thanks for your feedback.

2014ko abuztuak 16an, "Charles C. Berry"-ek idatzi zuen:
> Aaron,
> 
> I think doing something along these lines makes sense. evaluate() obviates 
> the need for tryCatch() and capture.output(), and it makes customizing 
> error/warning/message stuff clean. Its use might also lead to cleaner code 
> for R graphics handling.
> 
> But adding a dependency on the evaluate package is a significant step. 
> This could be a nuisance for users whose code runs just fine right now. It 
> would need to be installed anywhere R is executed, e.g. in remote 
> sessions. The user would need to install it in her private directory if it 
> is not on the system; some shops actually discourage this.
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

That’s ... special.  Do you have experience with such environments?

> 
> So, making it optional might be necessary.

I don’t like this idea; it means that we’d have to support the old,
hacky way of doing things indefinitely.  I’d like to hear more from
people for whom local installation of R packages creates an issue.

> 
> ---
> 
> I looked at the patch briefly. Some comments:
> 
> - You can ditch tryCatch and capture.output. You might browse
>    knitr:::block_exec to see how it uses evaluate().

I’m not sure I see what you mean.  The tryCatch exists to ensure that
the sentinel file is created, to signal emacs that the R code is done
running.  It might not be strictly necessary, as long as the rest of
babel’s injected code is error-free (evaluate takes care of catching
errors in user code).  But emacs will busy-wait indefinitely if the
creation of that file does not happen, so I have tried to play it safe.

As for capture.output, the knitr function you reference is doing a lot
of heavy lifting; I don’t understand it all.  I just need to get the
result of evaluate() into a file somehow.  There are other ways of doing
this than capture.output + replay, but it seems like they’d be just as
complicated.

> 
> - Wrap the code in local() to keep objects you create from persisting
>    where they might not be wanted. I think using
>  	local({ res <- evaluate(input, envir=parent.frame(2),...); <...>})
>    will get assignments from `input' properly placed.

I omitted to rm() .org.eval.result for debugging purposes, but in the
final patch I will do so.  I much prefer an explicit variable creation /
rm() to environment-hacking.

> 
> - let stop_on_error, keep_warning, and keep_message args be customizable
>    or depend on a header arg. (Then I can stop wrapping require() in
>    suppressPackageStartupMessages() which I always misspell. Argh!)

I don’t like this, since it would not carry over to the other 3/4
cases (value results in a session; either type of result outside of a
session).  I’d like to get this patch working first, and then focus on
the others.  It may be that we want to move to using the evaluate
package in all cases, in which case this suggestion would be workable.

-- 
Aaron Ecay

Re: R code block produces only partial output

[Achim Gratz]

Aaron Ecay writes:
>> The user would need to install it in her private directory if it 
>> is not on the system; some shops actually discourage this.
>                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> That’s ... special.  Do you have experience with such environments?

I take it you've never dealt with a corporate IT department.  Go to
dilbert.com and do a search for "Mordac" to get an idea what you'd be up
against.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: R code block produces only partial output

[Aaron Ecay]

Hi Achim,

2014ko abuztuak 17an, Achim Gratz-ek idatzi zuen:
> I take it you've never dealt with a corporate IT department.  Go to
> dilbert.com and do a search for "Mordac" to get an idea what you'd be up
> against.

I’ve read Dilbert, and am familiar with the Mordac character.

R is capable of installing packages to a user-specified directory,
without requiring root or any other special privileges.  So IT cannot
literally prevent R users from installing their own packages; the
requirement must rather come as a statement of policy.  I’d like to
understand more about how such regimes work and how org could work
with them, ideally from people who have direct experience with them.

Otherwise, it would be disappointing if the fear that an unidentifiable
somebody somewhere might not be able to install R packages derailed the
improvement of babel’s R support.

Thanks,

-- 
Aaron Ecay

Re: R code block produces only partial output

[Achim Gratz]

Aaron Ecay writes:
> R is capable of installing packages to a user-specified directory,
> without requiring root or any other special privileges.  So IT cannot
> literally prevent R users from installing their own packages;

They can, all they have to do is to take away the ability to connect to
the outside network and that's what is increasingly being done (for
other reasons, mind you).

> the requirement must rather come as a statement of policy.

That is another common thing, if only to shift the responsibility if the
other measures don't actually prevent that.

> I’d like to
> understand more about how such regimes work and how org could work
> with them, ideally from people who have direct experience with them.

Ideally, don't require the use of a non-core package.  The beef with
things like CTAN, CRAN or CPAN is that they require extra maintenance
beyond the pure installation of some software and some specific
knowledge of the software in question and that's just not going to
happen in some places.

> Otherwise, it would be disappointing if the fear that an unidentifiable
> somebody somewhere might not be able to install R packages derailed the
> improvement of babel’s R support.

The request was to keep a fallback to core.  I have no comment at this
time if it would be possible or not.  Since ultimately it's the session
support of Emacs that is clunky (and that affects most other languages
as well), maybe an effort to improve that would be more generally
helpful than working around it in different ways in each language.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: R code block produces only partial output

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 2685 bytes --]

2014ko abuztuak 19an, Achim Gratz-ek idatzi zuen:
> 
> Aaron Ecay writes:
>> R is capable of installing packages to a user-specified directory,
>> without requiring root or any other special privileges.  So IT cannot
>> literally prevent R users from installing their own packages;
> 
> They can, all they have to do is to take away the ability to connect to
> the outside network and that's what is increasingly being done (for
> other reasons, mind you).
> 
>> the requirement must rather come as a statement of policy.
> 
> That is another common thing, if only to shift the responsibility if the
> other measures don't actually prevent that.
> 
>> I’d like to
>> understand more about how such regimes work and how org could work
>> with them, ideally from people who have direct experience with them.
> 
> Ideally, don't require the use of a non-core package.  The beef with
> things like CTAN, CRAN or CPAN is that they require extra maintenance
> beyond the pure installation of some software and some specific
> knowledge of the software in question and that's just not going to
> happen in some places.
> 
>> Otherwise, it would be disappointing if the fear that an unidentifiable
>> somebody somewhere might not be able to install R packages derailed the
>> improvement of babel’s R support.
> 
> The request was to keep a fallback to core.  I have no comment at this
> time if it would be possible or not.  Since ultimately it's the session
> support of Emacs that is clunky (and that affects most other languages
> as well), maybe an effort to improve that would be more generally
> helpful than working around it in different ways in each language.

Well, I think that it’s going to be difficult to make babel a better
literate programming solution for R if we restrict ourselves not to
use the state-of-the-art R package for low-level literate programming
support.  Org is full of features which one needs to install other
software to use, and I’m comfortable with the idea that babel’s R
support should require the evaluate package.  However, it’s difficult
to argue this point of view when no one has spoken up about their own
requirements, and a spirit of conservatism in the face of vague
imagined difficulties persists.

The attached patch fixes the problem that touched off this thread.
It’s only debatably nicer than the present approach, but it has the
independently desirable property of segregating babel-driven output
from the session buffer.  It incorporates what I think is a fix for
the tramp bug Charles pointed out.

It’s as yet only lightly tested, and as always comments are welcome.

[-- Attachment #2: 0001-ob-R-overhaul-handling-of-output-results-type-in-a-s.patch --]
[-- Type: text/x-diff, Size: 3890 bytes --]

From 5f3b4ec203b8bc14c3da975368481c3767ab6f02 Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Sat, 16 Aug 2014 00:49:05 -0400
Subject: [PATCH] ob-R: overhaul handling of :output results type in a session
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* (org-babel-R-evaluate-session): Capture output using ‘ess-execute’.

This function dumps the output from R into a buffer; we then use some
tricks to clean it of spurious text.
---
 lisp/ob-R.el | 63 +++++++++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 45 insertions(+), 18 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 41b943c..2d8220b 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -364,9 +364,10 @@ last statement in BODY, as elisp."
     (output (org-babel-eval org-babel-R-command body))))
 
 (defvar ess-eval-visibly-p)
+(defvar ess-execute-in-process-buffer)
 
 (defun org-babel-R-evaluate-session
-  (session body result-type result-params column-names-p row-names-p)
+    (session body result-type result-params column-names-p row-names-p)
   "Evaluate BODY in SESSION.
 If RESULT-TYPE equals 'output then return standard output as a
 string.  If RESULT-TYPE equals 'value then return the value of the
@@ -396,23 +397,49 @@ last statement in BODY, as elisp."
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
     (output
-     (mapconcat
-      'org-babel-chomp
-      (butlast
-       (delq nil
-	     (mapcar
-	      (lambda (line) (when (> (length line) 0) line))
-	      (mapcar
-	       (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		      "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
-		     (substring line (match-end 1))
-		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+     (let* ((sentinel-file (org-babel-temp-file "R-")))
+       ;; This code borrowed from
+       ;; `org-babel-comint-eval-invisibly-and-wait-for-file'
+       (unless (org-babel-comint-buffer-livep session)
+	 (error "Buffer %s does not exist or has no process" session))
+       (if (file-exists-p sentinel-file) (delete-file sentinel-file))
+       (with-current-buffer session
+	 (let ((ess-execute-in-process-buffer nil))
+	   (ess-execute (format
+			 "
+.org.old.opts <- options(prompt = \"# Org babel prompt\\n\", continue = \"# Org babel prompt\\n\")
+cat(\"# Org babel begin output\n\")
+%s
+cat(\"# Org babel end output\n\")
+options(.org.old.opts)
+rm(.org.old.opts)
+file.create(%S)
+"
+			 (org-babel-chomp body)
+			 (if (tramp-tramp-file-p sentinel-file)
+			     (tramp-file-name-localname
+			      (tramp-dissect-file-name sentinel-file))
+			   sentinel-file))
+			nil "org-babel-R-output"))
+	 ;; From Tramp 2.1.19 the following cache flush is not necessary
+	 (if (file-remote-p default-directory)
+	     (let (v)
+	       (with-parsed-tramp-file-name default-directory nil
+		 (tramp-flush-directory-property v "")))))
+       (while (not (file-exists-p sentinel-file)) (sit-for 0.25))
+       ;; End borrowed code.
+       (with-current-buffer "*org-babel-R-output*"
+	 (goto-char (point-min))
+	 (search-forward "# Org babel begin output")
+	 (delete-region (point-min) (1+ (point)))
+	 (goto-char (point-max))
+	 (search-backward "# Org babel end output")
+	 (delete-region (point) (point-max))
+	 (goto-char (point-min))
+	 (flush-lines "^$")
+	 (flush-lines "^# Org babel prompt$")
+	 (delete-trailing-whitespace)
+	 (buffer-string))))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
-- 
2.0.4


[-- Attachment #3: Type: text/plain, Size: 15 bytes --]

-- 
Aaron Ecay

Re: R code block produces only partial output

[Andreas Kiermeier]

[-- Attachment #1: Type: text/plain, Size: 3416 bytes --]

Hi Aaron,
thanks for keeping on this.
I personally don't have a problem with requiring an R package.
In fact, users will have a rather limited experience with R if they cannot
install add-on packages (for whatever reason) - this is one of the things
that makes R such an attractive option for data analysis.
While there might be some IT departments that are very restrictive, I also
think (no data behind this) that the majority of our users are not in that
situation.
Anyway, that's my two cents worth.
​Cheers,
Andreas


On 23 August 2014 18:02, Aaron Ecay <aaronecay@gmail.com> wrote:

> 2014ko abuztuak 19an, Achim Gratz-ek idatzi zuen:
> >
> > Aaron Ecay writes:
> >> R is capable of installing packages to a user-specified directory,
> >> without requiring root or any other special privileges.  So IT cannot
> >> literally prevent R users from installing their own packages;
> >
> > They can, all they have to do is to take away the ability to connect to
> > the outside network and that's what is increasingly being done (for
> > other reasons, mind you).
> >
> >> the requirement must rather come as a statement of policy.
> >
> > That is another common thing, if only to shift the responsibility if the
> > other measures don't actually prevent that.
> >
> >> I’d like to
> >> understand more about how such regimes work and how org could work
> >> with them, ideally from people who have direct experience with them.
> >
> > Ideally, don't require the use of a non-core package.  The beef with
> > things like CTAN, CRAN or CPAN is that they require extra maintenance
> > beyond the pure installation of some software and some specific
> > knowledge of the software in question and that's just not going to
> > happen in some places.
> >
> >> Otherwise, it would be disappointing if the fear that an unidentifiable
> >> somebody somewhere might not be able to install R packages derailed the
> >> improvement of babel’s R support.
> >
> > The request was to keep a fallback to core.  I have no comment at this
> > time if it would be possible or not.  Since ultimately it's the session
> > support of Emacs that is clunky (and that affects most other languages
> > as well), maybe an effort to improve that would be more generally
> > helpful than working around it in different ways in each language.
>
> Well, I think that it’s going to be difficult to make babel a better
> literate programming solution for R if we restrict ourselves not to
> use the state-of-the-art R package for low-level literate programming
> support.  Org is full of features which one needs to install other
> software to use, and I’m comfortable with the idea that babel’s R
> support should require the evaluate package.  However, it’s difficult
> to argue this point of view when no one has spoken up about their own
> requirements, and a spirit of conservatism in the face of vague
> imagined difficulties persists.
>
> The attached patch fixes the problem that touched off this thread.
> It’s only debatably nicer than the present approach, but it has the
> independently desirable property of segregating babel-driven output
> from the session buffer.  It incorporates what I think is a fix for
> the tramp bug Charles pointed out.
>
> It’s as yet only lightly tested, and as always comments are welcome.
>
> --
> Aaron Ecay
>
>

[-- Attachment #2: Type: text/html, Size: 4822 bytes --]

Re: R code block produces only partial output

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

Here’s another version of the previous patch which fixes a couple bugs
related to
1) changing the window configuration unwarrantedly
2) syntactically invalid R code (mismatched parens or quotes) could hang
   emacs

It’s easy to get R to throw an error in the mismatched-parens case, but
more difficult to recover from the mismatched-quotes one.  Thus, when
babel detects such a case it bails and signals a user-error in emacs.

(Under the present code in master, both the mismatched-parens and
mismatched-quotes cases hang emacs pending a manual C-g.)

The patch is now firmly into the territory of gross hacks, so I’d much
prefer the evaluate-based approach if it can be made palatable.


[-- Attachment #2: 0001-ob-R-overhaul-handling-of-output-results-type-in-a-s.patch --]
[-- Type: text/x-diff, Size: 4099 bytes --]

From afa24835f2a170c59a044dcf3bcf0ee765e8b568 Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Sat, 16 Aug 2014 00:49:05 -0400
Subject: [PATCH] ob-R: overhaul handling of :output results type in a session
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* (org-babel-R-evaluate-session): Capture output using ‘ess-execute’.

This function dumps the output from R into a buffer; we then use some
tricks to clean it of spurious text.
---
 lisp/ob-R.el | 69 ++++++++++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 51 insertions(+), 18 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 41b943c..fbd4af4 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -364,9 +364,10 @@ last statement in BODY, as elisp."
     (output (org-babel-eval org-babel-R-command body))))
 
 (defvar ess-eval-visibly-p)
+(defvar ess-execute-in-process-buffer)
 
 (defun org-babel-R-evaluate-session
-  (session body result-type result-params column-names-p row-names-p)
+    (session body result-type result-params column-names-p row-names-p)
   "Evaluate BODY in SESSION.
 If RESULT-TYPE equals 'output then return standard output as a
 string.  If RESULT-TYPE equals 'value then return the value of the
@@ -396,23 +397,55 @@ last statement in BODY, as elisp."
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
     (output
-     (mapconcat
-      'org-babel-chomp
-      (butlast
-       (delq nil
-	     (mapcar
-	      (lambda (line) (when (> (length line) 0) line))
-	      (mapcar
-	       (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		      "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
-		     (substring line (match-end 1))
-		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+     (let* ((sentinel-file (org-babel-temp-file "R-")))
+       ;; This code borrowed from
+       ;; `org-babel-comint-eval-invisibly-and-wait-for-file'
+       (unless (org-babel-comint-buffer-livep session)
+	 (error "Buffer %s does not exist or has no process" session))
+       (if (file-exists-p sentinel-file) (delete-file sentinel-file))
+       (with-current-buffer session
+	 (let ((ess-execute-in-process-buffer nil)
+	       (temp-buffer-show-function #'ignore))
+	   (ess-execute (format
+			 "
+.org.old.opts <- options(prompt = \"# Org babel prompt\\n\", continue = \"# Org babel prompt\\n\")
+cat(\"# Org babel begin output\n\")
+{
+%s
+#ORG-BABEL-OOPS\"#'
+}
+cat(\"# Org babel end output\n\")
+options(.org.old.opts)
+rm(.org.old.opts)
+file.create(%S)
+"
+			 (org-babel-chomp body)
+			 (if (tramp-tramp-file-p sentinel-file)
+			     (tramp-file-name-localname
+			      (tramp-dissect-file-name sentinel-file))
+			   sentinel-file))
+			nil "org-babel-R-output"))
+	 ;; From Tramp 2.1.19 the following cache flush is not necessary
+	 (if (file-remote-p default-directory)
+	     (let (v)
+	       (with-parsed-tramp-file-name default-directory nil
+		 (tramp-flush-directory-property v "")))))
+       (while (not (file-exists-p sentinel-file)) (sit-for 0.25))
+       ;; End borrowed code.
+       (with-current-buffer "*org-babel-R-output*"
+	 (goto-char (point-min))
+	 (when (search-forward "ORG-BABEL-OOPS")
+	   (user-error "Mismatched quotation marks in babel code; could not parse output."))
+	 (search-forward "# Org babel begin output")
+	 (delete-region (point-min) (1+ (point)))
+	 (goto-char (point-max))
+	 (search-backward "# Org babel end output")
+	 (delete-region (point) (point-max))
+	 (goto-char (point-min))
+	 (flush-lines "^$")
+	 (flush-lines "^# Org babel prompt$")
+	 (delete-trailing-whitespace)
+	 (buffer-string))))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
-- 
2.0.4


[-- Attachment #3: Type: text/plain, Size: 15 bytes --]

-- 
Aaron Ecay

Re: R code block produces only partial output

[Thomas S. Dye]

Aaron Ecay <aaronecay@gmail.com> writes:

> Well, I think that it’s going to be difficult to make babel a better
> literate programming solution for R if we restrict ourselves not to
> use the state-of-the-art R package for low-level literate programming
> support.  Org is full of features which one needs to install other
> software to use, and I’m comfortable with the idea that babel’s R
> support should require the evaluate package.  However, it’s difficult
> to argue this point of view when no one has spoken up about their own
> requirements, and a spirit of conservatism in the face of vague
> imagined difficulties persists.

As a regular user of babel, including ob-R, I do want to see it mature
into a state-of-the-art, productive literate programming environment.
I've followed babel development for a long time and my sense is that
responsible experimentation is the norm, backed up by the ability to
revert commits that end up causing undue headaches.  If Aaron's good
work on ob-R raises howls of protest from users with restrictive IT
managers, or the evaluate package somehow becomes a pariah, then at some
point ob-R would have to drop the evaluate package requirement.  If not,
then the rest of us can enjoy the benefits of Aaron's labor (hopefully
on master and not as a series of patches, which are a real pain for us
non-programmer types to maintain).

My $0.02.

Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: R code block produces only partial output

[Ista Zahn]

On Sat, Aug 23, 2014 at 2:35 PM, Thomas S. Dye <tsd@tsdye.com> wrote:
> Aaron Ecay <aaronecay@gmail.com> writes:
>
>> Well, I think that it’s going to be difficult to make babel a better
>> literate programming solution for R if we restrict ourselves not to
>> use the state-of-the-art R package for low-level literate programming
>> support.  Org is full of features which one needs to install other
>> software to use, and I’m comfortable with the idea that babel’s R
>> support should require the evaluate package.  However, it’s difficult
>> to argue this point of view when no one has spoken up about their own
>> requirements, and a spirit of conservatism in the face of vague
>> imagined difficulties persists.
>
> As a regular user of babel, including ob-R, I do want to see it mature
> into a state-of-the-art, productive literate programming environment.
> I've followed babel development for a long time and my sense is that
> responsible experimentation is the norm, backed up by the ability to
> revert commits that end up causing undue headaches.  If Aaron's good
> work on ob-R raises howls of protest from users with restrictive IT
> managers, or the evaluate package somehow becomes a pariah, then at some
> point ob-R would have to drop the evaluate package requirement.  If not,
> then the rest of us can enjoy the benefits of Aaron's labor (hopefully
> on master and not as a series of patches, which are a real pain for us
> non-programmer types to maintain).
>
> My $0.02.

+1.

R has considerable infrastructure to support litterarate programming,
and it makes little sense to avoid using it because a) some
hypothetical users will have difficulty installing the software or b)
because other languages supported by babel lack similar functionality.
We are currently missing out on some useful things available in other
literate programming systems for R, such as the ability to easily
return interleaved input and output, and I would really like to see R
support in babel take advantage of existing R features to provide a
first-class R literate programming environment.

Best,
Ista

>
> Tom
>
> --
> Thomas S. Dye
> http://www.tsdye.com
>

Re: R code block produces only partial output

[Charles C. Berry]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5044 bytes --]


Aaron,

Some comments inline.

A patch that modifies what you have done is attached. Apart from adding 
`org-babel-local-file-name' in two places, I think it is functionally 
that same as yours, but I find the R code easier to follow.


On Sat, 16 Aug 2014, Aaron Ecay wrote:

> Hi Chuck,
>
> Thanks for your feedback.
>
> 2014ko abuztuak 16an, "Charles C. Berry"-ek idatzi zuen:
>> Aaron,
>>
>> I think doing something along these lines makes sense. evaluate() obviates
>> the need for tryCatch() and capture.output(), and it makes customizing
>> error/warning/message stuff clean. Its use might also lead to cleaner code
>> for R graphics handling.
>>
>> But adding a dependency on the evaluate package is a significant step.
>> This could be a nuisance for users whose code runs just fine right now. It
>> would need to be installed anywhere R is executed, e.g. in remote
>> sessions. The user would need to install it in her private directory if it
>> is not on the system; some shops actually discourage this.
>                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> That’s ... special.  Do you have experience with such environments?

Not really. Just seen complaints...

>
>>
>> So, making it optional might be necessary.
>
> I don’t like this idea; it means that we’d have to support the old,
> hacky way of doing things indefinitely.  I’d like to hear more from
> people for whom local installation of R packages creates an issue.
>

The old hacky way works pretty well most of the time. I suspect that it 
will be hard to get folks to really test a new ob-R.el in advance of its 
moving to maint. ./lisp/test-ob-R.el only has one :session src block and 
a few other src blocks...

I was thinking of having a drop in replacement for the each of the two 
functions that are changed and maybe selecting which version to use with 
defalias. If there is a slicker way to enable a user to revert to an 
earlier version without having good git skills, fine. But I think the 
possibility of breaking something that someone needs and not noticing till 
the changes go to maint is high.

>>
>> ---
>>
>> I looked at the patch briefly. Some comments:
>>
>> - You can ditch tryCatch and capture.output. You might browse
>>    knitr:::block_exec to see how it uses evaluate().
>
> I’m not sure I see what you mean.  The tryCatch exists to ensure that
> the sentinel file is created, to signal emacs that the R code is done
> running.  It might not be strictly necessary, as long as the rest of
> babel’s injected code is error-free (evaluate takes care of catching
> errors in user code).  But emacs will busy-wait indefinitely if the
> creation of that file does not happen, so I have tried to play it safe.
>

Not sure I get why the sentinel file is needed, but the patch here uses 
on.exit(file.create(...)) to ensure that that file is created. One hiccup 
(not sure if it exists in master,too) is that starting a remote session 
and then trying to run src blocks from a buffer for a local file will hang 
(because a local temp file is used for sentinel). So there is still stuff 
to do.

> As for capture.output, the knitr function you reference is doing a lot
> of heavy lifting; I don’t understand it all.  I just need to get the
> result of evaluate() into a file somehow.  There are other ways of doing
> this than capture.output + replay, but it seems like they’d be just as
> complicated.

After looking more at evaluate and that knitr function, I am not keen to 
replace replay() to obviate sink() or capture.output(). I had thought that 
the output_handler arg offered more potential that it actually does. If 
you ever feel the need to revise how graphics handling is done, it might 
be worth working thru what Yihui does there.

>
>>
>> - Wrap the code in local() to keep objects you create from persisting
>>    where they might not be wanted. I think using
>>  	local({ res <- evaluate(input, envir=parent.frame(2),...); <...>})
>>    will get assignments from `input' properly placed.
>
> I omitted to rm() .org.eval.result for debugging purposes, but in the
> final patch I will do so.  I much prefer an explicit variable creation /
> rm() to environment-hacking.
>

Putting objects in userspace is considered poor practice. FWIW, CRAN 
disallows it. `environment-hacking' as you call it underlies much of R. If 
you must create and hold variables use the attach(NULL,name="org-vars") 
trick.


>>
>> - let stop_on_error, keep_warning, and keep_message args be customizable
>>    or depend on a header arg. (Then I can stop wrapping require() in
>>    suppressPackageStartupMessages() which I always misspell. Argh!)
>
> I don’t like this, since it would not carry over to the other 3/4
> cases (value results in a session; either type of result outside of a
> session).  I’d like to get this patch working first, and then focus on
> the others.  It may be that we want to move to using the evaluate
> package in all cases, in which case this suggestion would be workable.

Fair enough.

Best,

Chuck

[-- Attachment #2: modify AE's session-results-output patch --]
[-- Type: TEXT/PLAIN, Size: 2076 bytes --]

From e51c092cc455f9deee8d3d5b413493aff84b38c3 Mon Sep 17 00:00:00 2001
From: chasberry <ccberry@ucsd.edu>
Date: Sat, 23 Aug 2014 15:54:55 -0700
Subject: [PATCH 2/2] merge CCBs version

---
 lisp/ob-R.el | 33 +++++++++++++++------------------
 1 file changed, 15 insertions(+), 18 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 08ef9c2..f828bf2 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -383,6 +383,17 @@ last statement in BODY, as elisp."
 
 (defvar ess-eval-visibly-p)
 
+(defvar org-babel-R-session-cmd
+   "local({
+      on.exit(file.create(%S))
+      sink(%S)
+      replay(
+          Filter(Negate(is.source),
+                 evaluate(%S, envir=parent.frame(2), new_device = FALSE, 
+                          stop_on_error = 0L)))
+      sink()})"
+"format string for an `R :session :results output' regime.")
+
 (defun org-babel-R-evaluate-session
     (session body result-type result-params column-names-p row-names-p)
   "Evaluate BODY in SESSION.
@@ -419,24 +430,10 @@ last statement in BODY, as elisp."
        (org-babel-comint-eval-invisibly-and-wait-for-file
 	session sentinel-file
 	(format
-"
-tryCatch(.org.eval.result <- evaluate(%S, new_device = FALSE, stop_on_error = 0L),
-         finally = {
-             capture.output(
-                 withCallingHandlers(
-                     replay(Filter(function (x) !inherits(x, \"source\"),
-                                   .org.eval.result)),
-                     message = function (x) {
-                         cat(x$message);
-                         invokeRestart(\"muffleMessage\")
-                     }),
-                 file=%S)
-             file.create(%S)
-         })
-"
-		(org-babel-chomp body)
-		output-file
-		sentinel-file))
+	 org-babel-R-session-cmd
+	 (org-babel-local-file-name sentinel-file)		
+	 (org-babel-local-file-name output-file)
+	 (org-babel-chomp body)))
        (with-temp-buffer
 	 (insert-file-contents output-file)
 	 (goto-char (point-min))
-- 
1.8.5.2 (Apple Git-48)

Re: R code block produces only partial output

[Aaron Ecay]

Hi Chuck,

Thanks for your feedback.  The patch you’ve sent looks basically correct
to me.

2014ko abuztuak 23an, "Charles C. Berry"-ek idatzi zuen:

[...]

> The old hacky way works pretty well most of the time. I suspect that it 
> will be hard to get folks to really test a new ob-R.el in advance of its 
> moving to maint. ./lisp/test-ob-R.el only has one :session src block and 
> a few other src blocks...

Agreed, more tests are something I want to add to this patch before
landing it, and I’ll try to encourage that in the future (from myself
most of all!).

> 
> I was thinking of having a drop in replacement for the each of the two 
> functions that are changed and maybe selecting which version to use with 
> defalias. If there is a slicker way to enable a user to revert to an 
> earlier version without having good git skills, fine. But I think the 
> possibility of breaking something that someone needs and not noticing till 
> the changes go to maint is high.

I think that having two different implementations will in the long run
increase the surface area susceptible to bugs.  We should do our best
and then release what we have when we feel it’s stable.

(I admit to being a bit bewildered by org’s release process; I just use
whatever is in git and deal with the occasional breakage.  The big no-no
is letting a bug slip into an emacs-bundled version of Org, since that
will live for years.  But it seems like it takes a long time even after
reaching maint for code to make its way into emacs.)


[...]

> 
> Not sure I get why the sentinel file is needed, 

The execution of the R commands is asynchronous (from emacs’ POV); org
polls for the existence of this file to let it know that R has finished
its execution.

> but the patch here uses on.exit(file.create(...)) to ensure that that
> file is created. One hiccup (not sure if it exists in master,too) is
> that starting a remote session and then trying to run src blocks from
> a buffer for a local file will hang (because a local temp file is used
> for sentinel). So there is still stuff to do.

Hmm, OK.


[...]

> 
> Putting objects in userspace is considered poor practice. FWIW, CRAN 
> disallows it. `environment-hacking' as you call it underlies much of R. If 
> you must create and hold variables use the attach(NULL,name="org-vars") 
> trick.

I’ll defer to your judgment.

Thanks again for the feedback and revised patch.  I’ll try to get a few
tests written this weekend, and study the remote vs. local files
business.

-- 
Aaron Ecay

Re: R code block produces only partial output

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

Hi Chuck,

Attached is a new version of the patch, including a few more tests.  I
have one question:

2014ko abuztuak 28an, Aaron Ecay-ek idatzi zuen:
>> but the patch here uses on.exit(file.create(...)) to ensure that that
>> file is created. One hiccup (not sure if it exists in master,too) is
>> that starting a remote session and then trying to run src blocks from
>> a buffer for a local file will hang (because a local temp file is used
>> for sentinel). So there is still stuff to do.
> 
> Hmm, OK.

Can you give a recipe to reproduce this?  Everything seems to work for
me using the following test code, although there is an unexplained ~10s
delay while babel sits in the ‘(while (not (file-exists-p file)) ...)’
loop at the end of ‘org-babel-comint-eval-invisibly-and-wait-for-file’.
,----
| #+name: foo
| #+begin_src R :session *foo* :dir /ssh:aecay@foo:/home/aecay :results output
|   1+1
| #+end_src
| 
| #+RESULTS: foo
| : [1] 2
`----

Thanks,

-- 
Aaron Ecay

[-- Attachment #2: 0001-ob-R-overhaul-handling-of-output-results-type-in-a-s.patch --]
[-- Type: text/x-diff, Size: 7157 bytes --]

From 65aa82b71e709250717896ab9bc85bf144a8ee30 Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Sat, 16 Aug 2014 00:49:05 -0400
Subject: [PATCH] ob-R: overhaul handling of :output results type in a session
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/ob-R.el (org-babel-R-check-evaluate-package): New function.
(org-babel-R-initiate-session): Use it.
(org-babel-R-evaluate-session): Use the evaluate package to capture
session output.

This uses the “evaluate” R package[1] to capture the
output (incl. warnings, errors, and messages) from a babel R session.
This avoids the output showing up in the session buffer, and dodges
some previous issues with removing R prompts (>) when scraping the
output from the session buffer.

Thanks to Charles C. Berry for assistance with this code.

[1] <http://cran.r-project.org/web/packages/evaluate/index.html>
---
 lisp/ob-R.el              | 73 +++++++++++++++++++++++++++++++++++------------
 testing/lisp/test-ob-R.el | 61 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 115 insertions(+), 19 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 41b943c..81b3290 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -245,6 +245,22 @@ This function is called by `org-babel-execute-src-block'."
 	  ((stringp  value) (format "%s <- %S" name (org-no-properties value)))
 	  (t                (format "%s <- %S" name (prin1-to-string value))))))
 
+(defvar ess-execute-in-process-buffer)
+(defun org-babel-R-check-evaluate-package (&optional recursive)
+  (save-window-excursion
+    (let ((ess-execute-in-process-buffer nil)
+	  (r-buff (current-buffer)))
+      (ess-execute "library(evaluate)" nil "org-babel-R-auto")
+      (when (with-current-buffer "*org-babel-R-auto*"
+	      (goto-char (point-min))
+	      (search-forward "Error" nil t))
+	(if (and (not recursive)
+		 (y-or-n-p "Cannot load the evaluate package required for babel session support, would you like to install it from CRAN?"))
+	    (progn
+	      (message "Downloading and installing package (may take some time)")
+	      (ess-execute "install.packages(\"evaluate\")" nil "org-babel-R-auto")
+	      (org-babel-R-check-evaluate-package t))
+	  (user-error "R package evaluate is required, but not available."))))))
 
 (defvar ess-ask-for-ess-directory) ; dynamically scoped
 (defun org-babel-R-initiate-session (session params)
@@ -261,7 +277,9 @@ This function is called by `org-babel-execute-src-block'."
 	  (when (get-buffer session)
 	    ;; Session buffer exists, but with dead process
 	    (set-buffer session))
-	  (require 'ess) (R)
+	  (require 'ess)
+	  (R)
+	  (org-babel-R-check-evaluate-package)
 	  (rename-buffer
 	   (if (bufferp session)
 	       (buffer-name session)
@@ -365,8 +383,27 @@ last statement in BODY, as elisp."
 
 (defvar ess-eval-visibly-p)
 
+(defconst org-babel-R-session-cmd
+  "local({
+      on.exit(file.create(%S))
+      sink(%S)
+      withCallingHandlers(
+          replay(
+              Filter(Negate(is.source),
+                     evaluate(%S, envir=parent.frame(2), new_device = FALSE,
+                              stop_on_error = 0L))),
+          message = function (x) {
+              cat(x$message);
+              invokeRestart(\"muffleMessage\")
+          },
+          error = function (e) {
+              cat(e$message)
+          })
+      sink()})"
+"format string for an `R :session :results output' regime.")
+
 (defun org-babel-R-evaluate-session
-  (session body result-type result-params column-names-p row-names-p)
+    (session body result-type result-params column-names-p row-names-p)
   "Evaluate BODY in SESSION.
 If RESULT-TYPE equals 'output then return standard output as a
 string.  If RESULT-TYPE equals 'value then return the value of the
@@ -396,23 +433,21 @@ last statement in BODY, as elisp."
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
     (output
-     (mapconcat
-      'org-babel-chomp
-      (butlast
-       (delq nil
-	     (mapcar
-	      (lambda (line) (when (> (length line) 0) line))
-	      (mapcar
-	       (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		      "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
-		     (substring line (match-end 1))
-		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+     (let* ((output-file (org-babel-temp-file "R-"))
+	    (sentinel-file (concat output-file "-sentinel")))
+       (org-babel-comint-eval-invisibly-and-wait-for-file
+	session sentinel-file
+	(format
+	 org-babel-R-session-cmd
+	 (org-babel-local-file-name sentinel-file)
+	 (org-babel-local-file-name output-file)
+	 (org-babel-chomp body)))
+       (with-temp-buffer
+	 (insert-file-contents output-file)
+	 (goto-char (point-min))
+	 (flush-lines "^$")
+	 (delete-trailing-whitespace)
+	 (buffer-string))))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
diff --git a/testing/lisp/test-ob-R.el b/testing/lisp/test-ob-R.el
index e3f13f1..3d7ebef 100644
--- a/testing/lisp/test-ob-R.el
+++ b/testing/lisp/test-ob-R.el
@@ -79,6 +79,67 @@ x
     (should (equal '(("col") ("a") ("b"))
 		   (org-babel-execute-src-block)))))
 
+(ert-deftest test-ob-R/session-output ()
+  (org-test-with-temp-text "#+begin_src R :results output :session *foo*
+  1
+  message(\"hi\")
+  2
+  warning(\"hi2\")
+  3
+  stop(\"hi3\")
+  4
+#+end_src
+"
+			   (should (string= (org-babel-execute-src-block)
+					    "[1] 1
+hi
+[1] 2
+Warning message:
+hi2
+[1] 3
+Error: hi3
+[1] 4"))))
+
+(ert-deftest test-ob-R/session-output-stray-open-paren ()
+  (org-test-with-temp-text "#+begin_src R :results output :session *foo*
+  (
+#+end_src
+"
+			   (should (string= (org-babel-execute-src-block)
+					    "<text>:2:0: unexpected end of input
+1: (
+   ^"))))
+
+(ert-deftest test-ob-R/session-output-stray-close-paren ()
+  (org-test-with-temp-text "#+begin_src R :results output :session *foo*
+  )
+#+end_src
+"
+			   (should (string= (org-babel-execute-src-block)
+					    "<text>:1:1: unexpected ')'
+1: )
+    ^"))))
+
+(ert-deftest test-ob-R/session-output-stray-single-quote ()
+  (org-test-with-temp-text "#+begin_src R :results output :session *foo*
+  '
+#+end_src
+"
+			   (should (string= (org-babel-execute-src-block)
+					    "<text>:1:1: unexpected INCOMPLETE_STRING
+1: '
+    ^"))))
+
+(ert-deftest test-ob-R/session-output-stray-dbl-quote ()
+  (org-test-with-temp-text "#+begin_src R :results output :session *foo*
+  \"
+#+end_src
+"
+			   (should (string= (org-babel-execute-src-block)
+					    "<text>:1:1: unexpected INCOMPLETE_STRING
+1: \"
+    ^"))))
+
 (provide 'test-ob-R)
 
 ;;; test-ob-R.el ends here
-- 
2.1.0

Re: R code block produces only partial output

[Charles C. Berry]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3075 bytes --]

On Sun, 31 Aug 2014, Aaron Ecay wrote:

> Hi Chuck,
>
> Attached is a new version of the patch, including a few more tests.  I
> have one question:
>
> 2014ko abuztuak 28an, Aaron Ecay-ek idatzi zuen:
>>> but the patch here uses on.exit(file.create(...)) to ensure that that
>>> file is created. One hiccup (not sure if it exists in master,too) is
>>> that starting a remote session and then trying to run src blocks from
>>> a buffer for a local file will hang (because a local temp file is used
>>> for sentinel). So there is still stuff to do.
>>
>> Hmm, OK.
>
> Can you give a recipe to reproduce this?  Everything seems to work for
> me using the following test code, although there is an unexplained ~10s
> delay while babel sits in the ‘(while (not (file-exists-p file)) ...)’
> loop at the end of ‘org-babel-comint-eval-invisibly-and-wait-for-file’.
> ,----
> | #+name: foo
> | #+begin_src R :session *foo* :dir /ssh:aecay@foo:/home/aecay :results output
> |   1+1
> | #+end_src
> |
> | #+RESULTS: foo
> | : [1] 2
> `----


Aaron,

I installed your patch.

Two cases:

1) I open a connection to a remote server by opening a directory there.

From *Messages*:

Tramp: Opening connection for berry@<censored> using scpc...

Then start an R session, type 'library(evaluate)', then change to a local 
buffer with an R src block and C-c C-c, wait a long while then C-g, then 
change to the *R* buffer and quit the session. I get this in the *R* 
buffer:


,----
| > > options(STERM='iESS', str.dendrogram.last="'", editor='emacsclient', 
| show.error.locations=TRUE)
| > library(evaluate)
| >
| + + + + . + + + Error in file(file, ifelse(append, "a", "w")) (from #3) :
|   cannot open the connection
| In addition: Warning message:
| In file(file, ifelse(append, "a", "w")) :
|   cannot open file 
| '/var/folders/kb/2hchpbyj7lb6z76l0q73w_fh0000gn/T/babel-392ywI/R-392jUu': 
| No such file or directory
| > q()
| Save workspace image? [y/n/c]: n
| Warning message:
| In 
| file.create("/var/folders/kb/2hchpbyj7lb6z76l0q73w_fh0000gn/T/babel-392ywI/R-392jUu-sentinel") 
| :
|   cannot create file 
| '/var/folders/kb/2hchpbyj7lb6z76l0q73w_fh0000gn/T/babel-392ywI/R-392jUu-sentinel', 
| reason 'No such file or directory'
| 
| Process R finished at Mon Sep  1 08:31:19 2014
`----

This worked OK with master.


2) If I modify your 'foo' block to use  :dir /ssh:berry@<censored>/<censored>

and C-c C-c, I get an error message 'Invalid function: with-parsed-tramp-file-name'

In the *Messages* buffer, I see:

,----
| Tramp: Decoding region into remote file /ssh:berry@microb215.med.upenn.edu:/tmp/R-39288P...done
| org-babel-comint-eval-invisibly-and-wait-for-file: Invalid function: with-parsed-tramp-file-name
`----

FWIW, the emacs on that remote system is 22.1.

I thought maybe tramp does some magic that I do not grok, tries to
use the remote emacs and fails. If so, note:

On the remote system

C-h f with-parsed TAB RET shows

,----
| with-parsed-tramp-file-name is a Lisp macro in `tramp.el'.
| ...
`----

so it *does* have that function.

???

HTH,

Chuck

Re: R code block produces only partial output

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 3627 bytes --]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> On Wed, 6 Aug 2014, Aaron Ecay wrote:
>
>> Hi Eric,
>>
>> 2014ko abuztuak 6an, Eric Schulte-ek idatzi zuen:
>>
>> [...]
>>
>>> Perhaps you could begin with a patch for the regexp issue in this
>>> thread?
>>
>> I have pushed a patch which allows us to avoid the regex issue
>> entirely by using a native R method to capture the session output to a
>> file.
>>
>> This introduces the change that the output no longer appears in the
>> session buffer, but I think that’s actually an improvement: we were not
>> previously echoing the commands to the buffer, such that the output
>> would show up “out of the blue” without any indication of how it got
>> there.
>
> Hi Aaron,
>
> I like what you are trying to do, but ...
>
> 1) The change has at least one bug: Remote sessions are broken by this 
> change.
>
> 2) The behavior of :results output is modified in ways that might not be 
> desired. i.e. warnings and errors will not show up in the output.
>
> Can you revert this change until the bugs are sorted out and consensus 
> about the proper handling of cases like '2' is reached?
>
> Can I also suggest that in the future before a change is pushed, that the 
> patch is announced so we can try it out or at least eyeball it and discuss
> issues/bugs?

I think that would be a good idea, or create a separate branch for
testing these features.

>
> Details:
>
> Issue 1) ===========
>
> If I open a *.org file on a remote machine and C-c C-c on a src block that 
> has `:session :results output', after the usual session startup the src 
> block fails. The session buffer shows this
>
> ====
> Error in file(file, if (append) "a" else "w") :
>    cannot open the connection
> In addition: Warning message:
> In file(file, if (append) "a" else "w") :
>    cannot open file '/scpc:berry@<DELETED.URL>:/tmp/R-1155xWV': 
> No such file or directory
>> 
> ===
>
> The file '/tmp/R-1155xWV' was created.
>
> I think if the tramp file localname is used. it might work. I do not know 
> tramp, but maybe something like
>
>   (let output-file-localname
>    (if (tramp-tramp-file-p output-file)
>        (tramp-file-name-localname
>         (tramp-dissect-file-name output-file))
>      output-file))
>
> is good enough.
>
>
> Issue 2) ===========
>
> ECM:
>
> #+NAME: aa
> #+BEGIN_SRC R :session R2 :results output
> warning("this is a warning")
> 1+1
> #+END_SRC
>
> #+RESULTS: aa
> : [1] 2
>
> For some purposes having the warnings in the #+RESULTS: block is helpful.
>
> And when revising code, having the errors in the #+RESULTS helps - 
> especially if I have to put aside work in progress.

Instead of capture.output, one could possibly use sink() as it can also
capture error messages and warnings, as shown in the help:

#+begin_src R 
## capture all the output to a file.
zz <- file("all.Rout", open = "wt")
sink(zz)
sink(zz, type = "message")
try(log("a"))
## back to the console
sink(type = "message")
sink()
file.show("all.Rout")
#+end_src

This combined with a try() bloxk could work?

Cheers,

Rainer

>
> HTH,
>
> Chuck
>

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

[-- Attachment #2: Type: application/pgp-signature, Size: 494 bytes --]

Re: R code block produces only partial output

[Andreas Kiermeier]

[-- Attachment #1: Type: text/plain, Size: 1658 bytes --]

​
On 6 August 2014 03:35, Charles Berry <ccberry@ucsd.edu> wrote:
​[snip]​

> org-babel-R-evaluate-session uses
>
>  (string-match "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
>
> to find the start of R output in the session.
>
> This does not match the `          0', but matches the `         .6'
> in the output you show above, so if that had been in a session, all the
> output up to and including the '.' before the '6' would be clipped
> by the following
>
>   (substring line (match-end 1))
>
>
> as Andreas output showed.
>
> Deleting the "\\." fixes Andreas case, but what are the circumstances
> requiring the  "\\." ?
>

​
​
Thanks Charles for finding this.

I do use R quite a bit, and I can only think of ">
​ ​
" (at beginning of the line) as the command prompt, along with "+
​ ​
" (at beginning of the line) as a continuation prompt for multi-line
commands. I don't know what
 ​ ​
"\" or
​ ​
"." provide
​ (I don't think this has changed over the various R versions)​

​.

​A ​
complication might be that the user can change the command prompt (and
continuation prompt) to something else they like (
 ​see here: ​
http://cran.r-project.org/doc/manuals/r-release/R-intro.html#Customizing-the-environment
 ​)​.

​For the time being I've taken the "\\." out of my local version of ob-R.el
(then byte compiled and loaded) and it seems to work fine.

​Maybe this is a question for the ESS developers who use R extensively and
probably might be able to provide some more definitive guidance on this
than I can?

​Thanks,

Andreas​

[-- Attachment #2: Type: text/html, Size: 4375 bytes --]

Re: R code block produces only partial output

[Charles C. Berry]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2512 bytes --]

On Tue, 5 Aug 2014, Andreas Kiermeier wrote:

> ​
> On 6 August 2014 03:35, Charles Berry <ccberry@ucsd.edu> wrote:
> ​[snip]​
>
>> org-babel-R-evaluate-session uses
>>
>>  (string-match "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)" line)
>>
>> to find the start of R output in the session.
>>
[snip]

> ​
> Thanks Charles for finding this.
>
> I do use R quite a bit, and I can only think of ">
> ​ ​
> " (at beginning of the line) as the command prompt, along with "+
> ​ ​
> " (at beginning of the line) as a continuation prompt for multi-line
> commands. I don't know what
> ​ ​
> "\" or
> ​ ​
> "." provide
> ​ (I don't think this has changed over the various R versions)​
>
> ​.
>
> ​A ​
> complication might be that the user can change the command prompt (and
> continuation prompt) to something else they like (
> ​see here: ​
> http://cran.r-project.org/doc/manuals/r-release/R-intro.html#Customizing-the-environment
> ​)​.
>
> ​For the time being I've taken the "\\." out of my local version of ob-R.el
> (then byte compiled and loaded) and it seems to work fine.
>
> ​Maybe this is a question for the ESS developers who use R extensively and
> probably might be able to provide some more definitive guidance on this
> than I can?


Here is the docstring from ESS's inferior-S-prompt:

inferior-S-prompt is a variable defined in `ess-custom.el'.
Its value is "[]a-zA-Z0-9.[]*\\([>+.] \\)*> "

Documentation:
Regexp used in S and R inferior and transcript buffers for prompt 
navigation.
Customise it to make `comint-previous-prompt' quiqly navigate to
interesting portions of the buffer.


You can customize this variable.

----

Note that there is a space *required* after the [>+.], but the hard coded 
regex in `org-babel-R-evaluate-session' lets the space be optional.

I am guessing the a-zA-Z stuff is there to strip 'Browse[1]> ' prompts and 
such, which probably isn't an issue for babel.

Just before the defcustom for inferior-S-prompt, I see this comment
in ess-custom.el:

;; need to recognise  + + + > > >
;; and "+ . + " in tracebug prompt

Ha! The '.' seems to be for ess-tracebug. I am guessing that nobody will 
need to output tracebug sessions from babel. So maybe it is safe to delete 
the "." after all - or at least require the space after it.

Maybe the regex should be put in a defcustom.

p.s. There is also `inferior-ess-prompt' which on my system is the same as 
inferior-S-prompt (in an R buffer - its buffer local)



HTH,

Chuck

Re: R code block produces only partial output

[Aaron Ecay]

Hi Chuck,

2014ko abuztuak 5an, "Charles C. Berry"-ek idatzi zuen:

[...]

> 
> 
> Here is the docstring from ESS's inferior-S-prompt:
> 
> inferior-S-prompt is a variable defined in `ess-custom.el'.
> Its value is "[]a-zA-Z0-9.[]*\\([>+.] \\)*> "
> 
> Documentation:
> Regexp used in S and R inferior and transcript buffers for prompt 
> navigation.
> Customise it to make `comint-previous-prompt' quiqly navigate to
> interesting portions of the buffer.
> 
> 
> You can customize this variable.
> 
> ----
> 
> Note that there is a space *required* after the [>+.], but the hard coded 
> regex in `org-babel-R-evaluate-session' lets the space be optional.
> 
> I am guessing the a-zA-Z stuff is there to strip 'Browse[1]> ' prompts and 
> such, which probably isn't an issue for babel.

Why doesn’t org just use the value of inferior-S-prompt?  (which would
then need to have the bit concatenated to the end that matches
e.g. [1]).  Another way of putting this might be “does the space that
ESS requires ever turn up missing?”

> 
> Just before the defcustom for inferior-S-prompt, I see this comment
> in ess-custom.el:
> 
> ;; need to recognise  + + + > > >
> ;; and "+ . + " in tracebug prompt
> 
> Ha! The '.' seems to be for ess-tracebug. I am guessing that nobody will 
> need to output tracebug sessions from babel. So maybe it is safe to delete 
> the "." after all - or at least require the space after it.

Tracebug is on by default in ESS – C-h v ess-use-tracebug.  Tracebug
even has babel support: see e.g. line 148 in ess.tracebug.el
<https://github.com/emacs-ess/ESS/blob/4283f1304a54502c42707b6a4ba347703f0992dd/lisp/ess-tracebug.el#L148>.

> 
> Maybe the regex should be put in a defcustom.

Why?  The current mismatch between babel and ESS seems like a bug, but
once that’s fixed I don’t see the use case for configurability here.
(But maybe you have something in mind...)

-- 
Aaron Ecay

Re: R code block produces only partial output

[Charles C. Berry]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2826 bytes --]

On Tue, 5 Aug 2014, Aaron Ecay wrote:

> Hi Chuck,
>
> 2014ko abuztuak 5an, "Charles C. Berry"-ek idatzi zuen:
>
> [...]
>
>>
>>
>> Here is the docstring from ESS's inferior-S-prompt:
>>
>> inferior-S-prompt is a variable defined in `ess-custom.el'.
>> Its value is "[]a-zA-Z0-9.[]*\\([>+.] \\)*> "
[snip]
>>
>> ----
>>
>> Note that there is a space *required* after the [>+.], but the hard coded
>> regex in `org-babel-R-evaluate-session' lets the space be optional.
>>
>> I am guessing the a-zA-Z stuff is there to strip 'Browse[1]> ' prompts and
>> such, which probably isn't an issue for babel.
>
> Why doesn’t org just use the value of inferior-S-prompt?  (which would
> then need to have the bit concatenated to the end that matches
> e.g. [1]).  Another way of putting this might be “does the space that
> ESS requires ever turn up missing?”

I don't really know why.

Re the space, do the characters preceeding the [>+.] ever show up 
in Babel output?

>
>>
>> Just before the defcustom for inferior-S-prompt, I see this comment
>> in ess-custom.el:
>>
>> ;; need to recognise  + + + > > >
>> ;; and "+ . + " in tracebug prompt
>>
>> Ha! The '.' seems to be for ess-tracebug. I am guessing that nobody will
>> need to output tracebug sessions from babel. So maybe it is safe to delete
>> the "." after all - or at least require the space after it.
>
> Tracebug is on by default in ESS – C-h v ess-use-tracebug.  Tracebug
> even has babel support: see e.g. line 148 in ess.tracebug.el
> <https://github.com/emacs-ess/ESS/blob/4283f1304a54502c42707b6a4ba347703f0992dd/lisp/ess-tracebug.el#L148>.
>

When I said 'output tracebug sessions' I meant in the sense of using 
`:results output' to save the part of the session pertaining to tracebug 
in the *.org buffer, which seemed like an unusual use case.


>>
>> Maybe the regex should be put in a defcustom.
>
> Why?

options(prompt="$ ") # for example

Plus, if anything like the case that started this thread ever happens 
again it might be easier to figure out and offer workarounds for special 
cases that would be hazardous as a default setting.

> The current mismatch between babel and ESS seems like a bug, but
> once that’s fixed I don’t see the use case for configurability here.
> (But maybe you have something in mind...)

It looks like the babel regex was copied and intentionally modified for 
some reason --- suggesting the author knows something we do not, e.g. a 
case that would be broken by using `inferior-S-prompt'.

I don't have a handle on all the issues in `comint-use-prompt-regexp' and 
all the `inferior-*-prompt' variables, but those are customizable.

If the mismatch is a bug, why not use 'inferior-S-prompt' as the default 
and allow customization or at least use  `(defvar org-babel-R-prompt 
inferior-S-prompt)'?

HTH,

Chuck