Re: graphing from org-tables

From: Dan Davison <davison@stats.ox.ac.uk>
To: Eric Schulte <schulte.eric@gmail.com>
Cc: Org-mode <emacs-orgmode@gnu.org>
Subject: Re: graphing from org-tables
Date: Sun, 21 Dec 2008 15:31:53 -0500	[thread overview]
Message-ID: <20081221203153.GB14389@stats.ox.ac.uk> (raw)
In-Reply-To: <20080728154152.GD21509@stats.ox.ac.uk>

On Mon, Jul 28, 2008 at 04:41:52PM +0100, Dan Davison wrote:
> On Mon, Jul 28, 2008 at 08:32:22AM -0700, Eric Schulte wrote:
> > Hi Dan,
> > 
> > One way around the sleep(60) hack may be to create the R graph using
> > an inferior R process.  See
> 
> Yep, I just this minute asked for help on the Emacs Speaks Statistics
> (ESS) mailing list! I'll report back if I make progress with that.
> 
> https://stat.ethz.ch/pipermail/ess-help/2008-July/004785.html

I think it's worth keeping this avenue alive, despite so much progress
having been made with gnuplot in the interim. So below is an initial
version of org-table-eval-R, which takes an arbitrary R function, and
applies it to an org table. As suggested above, this uses
inferior-ess-mode and therefore requires ess (emacs speaks statistics)
and R to be installed.

At it's simplest, R-function can be the name of a standard R
function. So with Eric's first example data set on Worg
(http://legito.net/worg/org-tutorials/org-plot.php) i.e. with 2
dependent variables

(org-table-eval-R "summary") produces (in a new buffer)

 first.dependent.var second.dependent.var
 Min.   :0.1560      Min.   :0.1808      
 1st Qu.:0.1773      1st Qu.:0.1985      
 Median :0.1900      Median :0.2146      
 Mean   :0.2110      Mean   :0.2262      
 3rd Qu.:0.2333      3rd Qu.:0.2375      
 Max.   :0.4250      Max.   :0.3750      

and (org-table-eval-R "matplot") produces a rough version of Eric's
gnuplot figure. (see below for a closer reproduction of the figure,
with the legend etc).

It would be simple to provide org-plot/R which would construct a
suitable plotting function and pass it to org-table-eval-R, and some
ready-made R code could be provided for standard plots -- histograms &
barplots, scatter plots, etc. I've given an initial version of
org-plot/R below, which gives the option to produce eps output rather
than a pop-up window (and it would be easy to use any of the R
graphics devices for output: jpeg, png, ps, pdf, tiff, X11, quartz,
etc)

But the power of this approach is that it provides not just plotting
functionality, but the ability to do arbitrary calculations and
statistical analyses of the org table, via the base R packages and the
hundreds of user-written packages.

It would have been better to have developed these ideas along with the
development of org-plot, but unfortunately I let that slip. But
org-table-eval-R provides a lot of power for anyone that's prepared to
write some R code. And if there's interest, then some more
user-friendly functionality could be provided for org users that want
something easier, but can install ess and R.

Dan

and here's a rough version of the "O R G" grid plot

(org-table-eval-R "(function (x) image(t(as.matrix(x))))" t)

Of course, these can be beautified to your heart's content with lots
of extra arguments to the R plotting functions. This comes a bit
closer to the two dependent-variables graph:

(org-table-eval-R "(function(x) { matplot(x, type=\"l\", ylab=\"\") ; legend(\"topright\", legend=c(\"dep var 1\"\,\"dep var 2\"), col=1:2, lty=1:2) })")

--------------------------------------------------------------
(defun org-table-eval-R (R-function &optional rownames-absent)
  "Apply an arbitrary R function to the org table at
point. R-FUNCTION is an R function (a string) taking one
argument, which is the R data frame representation of the org
table. It may be the name of an appropriate R
function (e.g. \"summary\", \"plot\"), or a user-defined
anonymous function of the form \"(function(data.frame)
{...})\". Unless ROWNAMES-ABSENT is non-nil the row names of the
data frame are taken from the first column of the org table"
  (interactive)
  (require 'ess)
  (let ((csv-file (make-temp-file "org-table-eval-R-")))
    (org-table-export csv-file "orgtbl-to-csv")
    (ess-execute (org-table-eval-R-make-expr 
		  R-function csv-file rownames-absent))
    (delete-file csv-file)))

(defun org-table-eval-R-make-expr (R-function csv-file rownames-absent)
  (concat R-function
	  "(read.csv(\"" csv-file "\""
	  (unless rownames-absent ", row.names=1") "))"))

(defun org-plot/R (&optional plot-function file rownames-absent)
  "Use R to create a plot using the org table at point. If FILE
is non-nil then the plot will be written to file (currently as
eps, in future the format could easily be determined by the file
extension.) Unless ROWNAMES-ABSENT is non-nil the row names will
be taken from the first column of the org table."
  (interactive)
  (require 'ess)
  (org-table-eval-R (or plot-function "plot") rownames-absent)
  (if file (ess-execute (concat "dev.copy2eps(\"" file "\")"))))

> 
> Dan
> 
> 
> 
> 
> > http://cran.r-project.org/doc/FAQ/R-FAQ.html#R-and-Emacs for
> > information on running R functions from inside of Emacs.
> > 
> > Best -- Eric
> > 
> > On Saturday, July 26, at 19:15, Dan Davison wrote:
> >  > R (www.r-project.org) is pretty good for data plotting and statistical
> >  > analyses. Here's my effort at the org-table-plot function, using
> >  > R. Since R contains a csv importer that can read from stdin, it's
> >  > pretty simple. I've tried to code it so that you can provide an
> >  > arbitrary R function as the optional argument, so in principle you can
> >  > do to your org-table anything that R is capable of in the realms of
> >  > data analysis and visualisation.
> >  > 
> >  > (defun org-table-R-plot (&optional R-function)
> >  > "Plot the current table using R. The table is transformed into a dataframe in R. Optional
> >  > argument R-function is a string which is either the name of an R
> >  > function, or an anonymous function definition of the form (function(d) {...}),
> >  > requiring a single argument (the dataframe). The default is to use
> >  > the R function 'plot' which produces scatter plots of all pairwise
> >  > combinations of columns. An example custom plotting function is:
> >  > * plot column 3 against column 1, adding least-squares linear regression fit in blue                                                                                                                               (function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") })                                                                                                                      "
> >  >   (interactive)
> >  >   (unless R-function (set 'R-function "plot"))
> >  >   (let ((file (make-temp-file "org-table-R-plot")))
> >  >     (org-table-export file "orgtbl-to-csv")
> >  >     (set-buffer (find-file-noselect file))
> >  >     (shell-command-on-region
> >  >      (point-min) (point-max)
> >  >      (concat "Rscript -e 'X11() ; " R-function "(read.csv(\"stdin\")) ; system(\"sleep 60\")'"))
> >  >     (delete-file file)))
> >  > 
> >  > 
> >  > R is at www.r-project.org 
> >  > (package r-base on ubuntu/debian)
> >  > 
> >  > Rscript is a command line non-interactive scripting utility that is
> >  > bundled automatically with the R installation. I reckon it'll be OK on
> >  > OSX but no idea about Windows.  My function doesn't have to be used
> >  > for plotting; the R-function argument can be any function operating on
> >  > the data from the org-table, producing numerical or graphical output.
> >  > 
> >  > There's several things that need to be sorted out with my function, e.g.
> >  > 
> >  > (i) I haven't worked out how to return control to the emacs process
> >  > while keeping the plot window there. I tried adding an & to the
> >  > shell-command, but that seemed to result in R receiving nothing on
> >  > stdin. So I've got that 'sleep 60' hack in there currently; use C-g if
> >  > you get bored of your plot.
> >  > 
> >  > (ii) If the R-function isn't doing graphics, then the call to X11()
> >  > gets in the way. X11() would only work on linux/mac OSX(?) anyway.
> >  > 
> >  > (iii) I'm afraid I don't even know yet how to pass the optional string
> >  > argument using M-x org-table-R-plot. Is it possible with some sort of
> >  > prefix argument, and an (interactive something) declaration? Anyway,
> >  > it seems to work if you evaluate e.g.
> >  > 
> >  > (org-table-R-plot "(function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") })")
> >  > 
> >  > Suggestions for improvements welcome!
> >  > 
> >  > Dan
> >  > 
> >  > 
> >  > 
> >  > On Fri, Jul 25, 2008 at 12:07:00PM -0700, Eric Schulte wrote:
> >  > > 
> >  > > I had some time waiting for things to execute, so I condensed your
> >  > > process into a single command (borrowing heavily from
> >  > > org-export-table).
> >  > > 
> >  > > (defun org-table/gnuplot (&optional x-col)
> >  > >   "Plot the current table using gnuplot.  Use a prefix argument
> >  > > to specify a column to use for the x-coordinates, to use the row
> >  > > number for the x-coordinates provide a prefix argument of 0."
> >  > >   (interactive "p")
> >  > >   (message (format "%S" x-col))
> >  > >   (unless (org-at-table-p)
> >  > >     (error "No table at point"))
> >  > >   (require 'org-exp)
> >  > >   (require 'gnuplot)
> >  > >   (org-table-align) ;; make sure we have everything we need
> >  > >   (let* ((beg (org-table-begin))
> >  > > 	 (end (org-table-end))
> >  > > 	 (cols (save-excursion
> >  > > 		 (goto-char end)
> >  > > 		 (backward-char 3)
> >  > > 		 (org-table-current-column)))
> >  > > 	 (data-beg (if (and 
> >  > > 			(goto-char beg)
> >  > > 			(re-search-forward org-table-dataline-regexp end t)
> >  > > 			(re-search-forward org-table-hline-regexp end t)
> >  > > 			(re-search-forward org-table-dataline-regexp end t))
> >  > > 		       (match-beginning 0)
> >  > > 		     beg))
> >  > > 	 (skip (- (line-number-at-pos data-beg) (line-number-at-pos beg)))
> >  > > 	 (exp-format (format "orgtbl-to-tsv :skip %d" skip))
> >  > > 	 (file (make-temp-file "org-table-plot")))
> >  > >     ;; export table
> >  > >     (org-table-export file exp-format)
> >  > >     (with-temp-buffer
> >  > >       ;; write script
> >  > >       (insert (org-table/gnuplot-script file x-col cols))
> >  > >       ;; graph table
> >  > >       (gnuplot-mode)
> >  > >       (gnuplot-send-buffer-to-gnuplot)
> >  > >       (bury-buffer (get-buffer "*gnuplot*")))
> >  > >     (delete-file file)))
> >  > > 
> >  > > (defun org-table/gnuplot-script (file x-col num-cols)
> >  > >   (let ((plot-str "'%s' using %s:%d with lines title '%d'");; "\\\n    ,"
> >  > > 	script)
> >  > >     (dotimes (col (+ 1 num-cols))
> >  > >       (unless (or (and x-col (equal col x-col)) (equal col 0))
> >  > > 	(setf script (cons (format plot-str file (or (and x-col (format "%d" x-col)) "") col col) script))))
> >  > >     (concat "plot " (mapconcat 'identity (reverse script) "\\\n    ,"))))
> >  > > 
> >  > > On Friday, July 25, at 17:25, James TD Smith wrote:
> >  > >  > On 2008-07-25 08:53:31(-0700), Eric Schulte wrote:
> >  > >  > > 
> >  > >  > > Any advice for quick graphing of a table in org-mode?
> >  > >  > > 
> >  > >  > 
> >  > >  > I have a setup for plotting data from tables. I'm not sure if it's exactly what
> >  > >  > you want, but yoy may find it useful.
> >  > >  > 
> >  > >  > 1. Add the following to your .emacs:
> >  > >  > 
> >  > >  > (defun ahkt-plot-table (script)
> >  > >  >   "util function to export and plot a table using the supplied
> >  > >  > gnuplot `script'"
> >  > >  >   (org-table-export)
> >  > >  >   (let ((cbuf (current-buffer))
> >  > >  > 	(cwin (selected-window)))
> >  > >  >     (save-restriction
> >  > >  >       (save-excursion
> >  > >  > 	(find-file script)
> >  > >  > 	(gnuplot-send-buffer-to-gnuplot)
> >  > >  > 	(bury-buffer)
> >  > >  > 	(bury-buffer (get-buffer "*gnuplot*"))))
> >  > >  >     (and (window-live-p cwin) (select-window cwin))
> >  > >  >     (switch-to-buffer cbuf)
> >  > >  >     (delete-other-windows)))
> >  > >  > 
> >  > >  > 2. Create a gnuplot script which plots data from a file.
> >  > >  > 
> >  > >  > 3. Add the following properties to the headline containing the table.
> >  > >  > TABLE_EXPORT_FILE <filename in the gnuploy script>
> >  > >  > TABLE_EXPORT_FORMAT orgtbl-to-generic :skip 4 :splice t :sep "\t"
> >  > >  > 
> >  > >  > 4. Add an org link in the table (it must be in the table otherwise the export
> >  > >  > doesn't work) as below:
> >  > >  > [[elisp:(ahkt-plot-table "<gnuplot script>")][plot table]] 
> >  > >  > 
> >  > >  > I suggest you put it at the top of the table.
> >  > >  > You will then need to adjust the 'skip' parameter in the export format depending
> >  > >  > on the number of lines at the top of the table which should not be exported
> >  > >  > (hlines, more than one plotting link etc). 
> >  > >  > 
> >  > >  > 5. You should then be able to open the link, and get a plot of the table
> >  > >  > contents.
> >  > >  > 
> >  > >  > 
> >  > >  > --
> >  > >  > |-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|
> >  > >  > 
> >  > >  > 
> >  > >  > _______________________________________________
> >  > >  > Emacs-orgmode mailing list
> >  > >  > Remember: use `Reply All' to send replies to the list.
> >  > >  > Emacs-orgmode@gnu.org
> >  > >  > http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> >  > > 
> >  > > -- 
> >  > > schulte
> >  > > 
> >  > > 
> >  > > _______________________________________________
> >  > > Emacs-orgmode mailing list
> >  > > Remember: use `Reply All' to send replies to the list.
> >  > > Emacs-orgmode@gnu.org
> >  > > http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> > 
> > -- 
> > schulte
> 
> 
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

-- 
http://www.stats.ox.ac.uk/~davison