[RFC] ob-reticulate: R+Python interface from Babel

[RFC] ob-reticulate: R+Python interface from Babel

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

Hi all,

Reticulate is an R package for interfacing between R and Python. It allows accessing objects in a Python session from R and vice versa. See https://rstudio.github.io/reticulate/ for more info about it.

I've written a small patch for using reticulate from org-babel. It allows creating a source block of lang "reticulate", which behaves as Python for font highlighting and editing, but is executed in an R session via reticulate.

I'm wondering whether this should go into org-mode, or whether to package this separately. I'm also curious whether this would be useful to anyone here. Any feedback is appreciated.

The main advantage of reticulate is being able to access Python objects directly from R and vice versa, without having to write them to a separate file or pass them through the ":var" header argument. For example, we could do the following:

#+begin_src reticulate :session
  import pandas as pd

  fib = [0, 1]
  for _ in range(10):
      fib.append(fib[-1] + fib[-2])

  df = pd.DataFrame({
      "i": list(range(len(fib))),
      "F_i": fib
  })
#+end_src

#+begin_src R :session :results graphics value file :file fig.png
  library(reticulate)
  with(py$df, plot(i, F_i))
#+end_src

Reticulate source blocks support both "value" and "output" results, and even supports graphics with matplotlib. It's primarily intended to be used in sessions, and the ":session" header argument should match between reticulate and R source blocks.

Cheers,
Jack


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-reticulate-Babel-source-lang-for-R-Python-reticul.patch --]
[-- Type: text/x-patch, Size: 2845 bytes --]

From 0f691a200cf088c72f93f7552d73caeafb8d588f Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Mon, 24 Aug 2020 08:02:17 -0700
Subject: [PATCH] ob-reticulate: Babel source lang for R+Python reticulate
 package

* lisp/ob-reticulate.el: New babel source block lang for
R's reticulate package for evaluating Python code.
* lisp/org-src.el (org-src-lang-modes): Add reticulate.
---
 lisp/ob-reticulate.el | 50 +++++++++++++++++++++++++++++++++++++++++++
 lisp/org-src.el       |  1 +
 2 files changed, 51 insertions(+)
 create mode 100644 lisp/ob-reticulate.el

diff --git a/lisp/ob-reticulate.el b/lisp/ob-reticulate.el
new file mode 100644
index 000000000..7da48681c
--- /dev/null
+++ b/lisp/ob-reticulate.el
@@ -0,0 +1,50 @@
+;;; ob-reticulate.el --- Babel Functions for reticulate -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2020 Free Software Foundation, Inc.
+
+;; Author: Jack Kamm
+;; Keywords: literate programming, reproducible research, R, statistics
+;; Homepage: https://orgmode.org
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Org-Babel support for the R package reticulate.
+
+;;; Code:
+
+(require 'ob-R)
+(require 'ob-python)
+
+(defalias 'org-babel-edit-prep:reticulate 'org-babel-edit-prep:R)
+
+(defun org-babel-execute:reticulate (body params)
+  (let* ((tmp-src-file (org-babel-temp-file "reticulate-"))
+	 (result-type (cdr (assq :result-type params))))
+    (with-temp-file tmp-src-file (insert body))
+    (org-babel-execute:R
+     (format (concat "reticulate::py_run_string(\"%s\")"
+		     (when (equal result-type 'value) "
+reticulate::py$`__org_babel_python_final`"))
+	     (format org-babel-python--eval-ast
+		     (org-babel-process-file-name
+		      tmp-src-file 'noquote)))
+     params)))
+
+(provide 'ob-reticulate)
+
+;;; ob-reticulate.el ends here
diff --git a/lisp/org-src.el b/lisp/org-src.el
index 28733d011..1b3d83f87 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -197,6 +197,7 @@ (defcustom org-src-lang-modes
     ("dot" . fundamental)
     ("elisp" . emacs-lisp)
     ("ocaml" . tuareg)
+    ("reticulate" . python)
     ("screen" . shell-script)
     ("shell" . sh)
     ("sqlite" . sql))
-- 
2.28.0

Re: [RFC] ob-reticulate: R+Python interface from Babel

[Kyle Andrews]

[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]

Hi Jack,

As a frequent reticulate user I am very excited to see this patch. I hope
others feel the same and it gets included into org mode as I cannot wait to
use it.

Best Regards,
Kyle

On Mon, Aug 24, 2020, 11:28 Jack Kamm <jackkamm@gmail.com> wrote:

> Hi all,
>
> Reticulate is an R package for interfacing between R and Python. It allows
> accessing objects in a Python session from R and vice versa. See
> https://rstudio.github.io/reticulate/ for more info about it.
>
> I've written a small patch for using reticulate from org-babel. It allows
> creating a source block of lang "reticulate", which behaves as Python for
> font highlighting and editing, but is executed in an R session via
> reticulate.
>
> I'm wondering whether this should go into org-mode, or whether to package
> this separately. I'm also curious whether this would be useful to anyone
> here. Any feedback is appreciated.
>
> The main advantage of reticulate is being able to access Python objects
> directly from R and vice versa, without having to write them to a separate
> file or pass them through the ":var" header argument. For example, we could
> do the following:
>
> #+begin_src reticulate :session
>   import pandas as pd
>
>   fib = [0, 1]
>   for _ in range(10):
>       fib.append(fib[-1] + fib[-2])
>
>   df = pd.DataFrame({
>       "i": list(range(len(fib))),
>       "F_i": fib
>   })
> #+end_src
>
> #+begin_src R :session :results graphics value file :file fig.png
>   library(reticulate)
>   with(py$df, plot(i, F_i))
> #+end_src
>
> Reticulate source blocks support both "value" and "output" results, and
> even supports graphics with matplotlib. It's primarily intended to be used
> in sessions, and the ":session" header argument should match between
> reticulate and R source blocks.
>
> Cheers,
> Jack
>
>

[-- Attachment #2: Type: text/html, Size: 2465 bytes --]

Re: [RFC] ob-reticulate: R+Python interface from Babel

[Kyle Meyer]

Jack Kamm writes:

> Hi all,
>
> Reticulate is an R package for interfacing between R and Python. It
> allows accessing objects in a Python session from R and vice
> versa. See https://rstudio.github.io/reticulate/ for more info about
> it.
>
> I've written a small patch for using reticulate from org-babel. It
> allows creating a source block of lang "reticulate", which behaves as
> Python for font highlighting and editing, but is executed in an R
> session via reticulate.

Neat, thanks for sharing.

> I'm wondering whether this should go into org-mode, or whether to
> package this separately. I'm also curious whether this would be useful
> to anyone here. Any feedback is appreciated.

It'd be good to hear from others, but in my view this would be fine to
add to Org proper.

I will say that in general new Babel libraries make me nervous.  My
impression is that Babel brings in a good number of bug reports, and a
new library is adding surface to that area [*] while not necessarily
adding eyes and hands.  Of course that worry doesn't apply here, as
you're already taking care of ob-python.

  [*] And that surface can be quite challenging to deal with because it
      is not just Elisp code; it brings in a whole outside language that
      other developers may have no clue about.

> +;;; Code:
> +
> +(require 'ob-R)
> +(require 'ob-python)
> +
> +(defalias 'org-babel-edit-prep:reticulate 'org-babel-edit-prep:R)
> +
> +(defun org-babel-execute:reticulate (body params)
> +  (let* ((tmp-src-file (org-babel-temp-file "reticulate-"))
> +	 (result-type (cdr (assq :result-type params))))
> +    (with-temp-file tmp-src-file (insert body))
> +    (org-babel-execute:R
> +     (format (concat "reticulate::py_run_string(\"%s\")"
> +		     (when (equal result-type 'value) "
> +reticulate::py$`__org_babel_python_final`"))
> +	     (format org-babel-python--eval-ast
> +		     (org-babel-process-file-name
> +		      tmp-src-file 'noquote)))
> +     params)))
> +
> +(provide 'ob-reticulate)

Well, that's pleasantly few lines of code :)

Re: [RFC] ob-reticulate: R+Python interface from Babel

[Bastien]

Hi Jack and Kyle,

Kyle Meyer <kyle@kyleam.com> writes:

>> I'm wondering whether this should go into org-mode, or whether to
>> package this separately. I'm also curious whether this would be useful
>> to anyone here. Any feedback is appreciated.
>
> It'd be good to hear from others, but in my view this would be fine to
> add to Org proper.

I think this would be a good addition to Org ecosystem (thanks Jack!)
but I also feel like we need to tidy things up a bit wrt Org Babel and
have stricter rules for library inclusion.

I will bring this as a separate discussion/proposal once 9.4 is out.

Thanks!

-- 
 Bastien

Re: [RFC] ob-reticulate: R+Python interface from Babel

[Jack Kamm]

Hi all,

I've put ob-reticulate.el on github so it can be more easily used:
https://github.com/jackkamm/ob-reticulate

I plan to submit it to MELPA or GNU ELPA over the coming weeks as well.

Cheers,
Jack

ob-reticulate: R+Python interface from Babel

[Jack Kamm]

Hi all,

ob-reticulate is now available on MELPA.

You can find more information here:
https://github.com/jackkamm/ob-reticulate

Cheers,
Jack

Re: ob-reticulate: R+Python interface from Babel

[Jeremie Juste]

Hello Jack,

Many thanks for this package. It seems like a better way all together
to manipulate python output.

I admit that my python foo is decreasing more and more but

#+BEGIN_SRC python :results output :session *Python*
  import pandas as pd
  df = pd.DataFrame({"X":[1,2,3], "Y":["a","b","c"]})
  df
#+end_src

#+RESULTS:

#+BEGIN_SRC python :results value  :session *Python*
  import pandas as pd
  df = pd.DataFrame({"X":[1,2,3], "Y":["a","b","c"]})
  df
#+end_src

#+RESULTS:
:    X  Y
: 0  1  a
: 1  2  b
: 2  3  c

compared to your [1] example of reticulate is easier to work with.
https://github.com/jackkamm/ob-reticulate

Best regards,
Jeremie


On Saturday, 27 Feb 2021 at 06:15, Jack Kamm wrote:
> Hi all,
>
> ob-reticulate is now available on MELPA.
>
> You can find more information here:
> https://github.com/jackkamm/ob-reticulate
>
> Cheers,
> Jack
>

-- 
Jeremie Juste

Re: ob-reticulate: R+Python interface from Babel

[Jack Kamm]

Hi Jeremie,

> Many thanks for this package. It seems like a better way all together
> to manipulate python output.

I don't disagree, at least when it comes to handling dataframes.

This is because ob-reticulate makes the block actually executed with
ob-R.el instead of ob-python.el. R has better built-in support for
manipulating dataframes, which ob-R.el can take advantage of when
handling output.

I tried before to improve ob-python handling of dataframes, plots, and
other results, but the code got a bit messy and I decided to put it on
ice. I may return to it, some day.

Cheers,
Jack