Re: graphing from org-tables

From: Dan Davison <davison@stats.ox.ac.uk>
To: emacs-orgmode@gnu.org
Subject: Re: graphing from org-tables
Date: Thu, 22 Jan 2009 11:27:11 -0500	[thread overview]
Message-ID: <20090122162711.GA11250@stats.ox.ac.uk> (raw)
In-Reply-To: <87r640p2ld.fsf@gmail.com>

On Mon, Dec 22, 2008 at 12:00:14PM -0500, Eric Schulte wrote:
> Hi Dan,
> 

<...>

> 2) Also interesting is the idea of referencing a table from a block of R
>    code elsewhere in the org file.  I've worked some on processing
>    blocks of R code in org files in a manner similar to Sweave.
>    http://github.com/eschulte/org-contrib/tree/master/org-exp-blocks.el
>    If we could implement a simple means of referencing tables (either
>    whole tables, or cells/cols/rows of tables) from these R-code blocks
>    then that seems like it may be fairly powerful.

I agree with this. So in the case of my code, I think we would like to
be able to do something like

#+TBLR: data:<org-table-reference> columns:(1 2) action:tabulate

This would make the output of the analysis/plot of the table pointed
to by <org-table-reference> appear at the location in the file of the
#+TBLR line. Perhaps one design aim would be to have the same
mechanism work for referencing tables for processing by #TBLFM,
org-plot, org-table-R, org-exp-blocks, etc. What ideas to people have
about implementation (and syntax) for this sort of table referencing?

Dan

> 
> Thanks -- Eric
> 
> Dan Davison <davison@stats.ox.ac.uk> writes:
> 
> > On Mon, Jul 28, 2008 at 04:41:52PM +0100, Dan Davison wrote:
> >> On Mon, Jul 28, 2008 at 08:32:22AM -0700, Eric Schulte wrote:
> >> > Hi Dan,
> >> > 
> >> > One way around the sleep(60) hack may be to create the R graph using
> >> > an inferior R process.  See
> >> 
> >> Yep, I just this minute asked for help on the Emacs Speaks Statistics
> >> (ESS) mailing list! I'll report back if I make progress with that.
> >> 
> >> https://stat.ethz.ch/pipermail/ess-help/2008-July/004785.html
> >
> > I think it's worth keeping this avenue alive, despite so much progress
> > having been made with gnuplot in the interim. So below is an initial
> > version of org-table-eval-R, which takes an arbitrary R function, and
> > applies it to an org table. As suggested above, this uses
> > inferior-ess-mode and therefore requires ess (emacs speaks statistics)
> > and R to be installed.
> >
> > At it's simplest, R-function can be the name of a standard R
> > function. So with Eric's first example data set on Worg
> > (http://legito.net/worg/org-tutorials/org-plot.php) i.e. with 2
> > dependent variables
> >
> > (org-table-eval-R "summary") produces (in a new buffer)
> >
> >  first.dependent.var second.dependent.var
> >  Min.   :0.1560      Min.   :0.1808      
> >  1st Qu.:0.1773      1st Qu.:0.1985      
> >  Median :0.1900      Median :0.2146      
> >  Mean   :0.2110      Mean   :0.2262      
> >  3rd Qu.:0.2333      3rd Qu.:0.2375      
> >  Max.   :0.4250      Max.   :0.3750      
> >
> > and (org-table-eval-R "matplot") produces a rough version of Eric's
> > gnuplot figure. (see below for a closer reproduction of the figure,
> > with the legend etc).
> >
> > It would be simple to provide org-plot/R which would construct a
> > suitable plotting function and pass it to org-table-eval-R, and some
> > ready-made R code could be provided for standard plots -- histograms &
> > barplots, scatter plots, etc. I've given an initial version of
> > org-plot/R below, which gives the option to produce eps output rather
> > than a pop-up window (and it would be easy to use any of the R
> > graphics devices for output: jpeg, png, ps, pdf, tiff, X11, quartz,
> > etc)
> >
> > But the power of this approach is that it provides not just plotting
> > functionality, but the ability to do arbitrary calculations and
> > statistical analyses of the org table, via the base R packages and the
> > hundreds of user-written packages.
> >
> > It would have been better to have developed these ideas along with the
> > development of org-plot, but unfortunately I let that slip. But
> > org-table-eval-R provides a lot of power for anyone that's prepared to
> > write some R code. And if there's interest, then some more
> > user-friendly functionality could be provided for org users that want
> > something easier, but can install ess and R.
> >
> > Dan
> >
> > and here's a rough version of the "O R G" grid plot
> >
> > (org-table-eval-R "(function (x) image(t(as.matrix(x))))" t)
> >
> > Of course, these can be beautified to your heart's content with lots
> > of extra arguments to the R plotting functions. This comes a bit
> > closer to the two dependent-variables graph:
> >
> > (org-table-eval-R "(function(x) { matplot(x, type=\"l\", ylab=\"\") ; legend(\"topright\", legend=c(\"dep var 1\"\,\"dep var 2\"), col=1:2, lty=1:2) })")
> >
> >
> >
> > --------------------------------------------------------------
> > (defun org-table-eval-R (R-function &optional rownames-absent)
> >   "Apply an arbitrary R function to the org table at
> > point. R-FUNCTION is an R function (a string) taking one
> > argument, which is the R data frame representation of the org
> > table. It may be the name of an appropriate R
> > function (e.g. \"summary\", \"plot\"), or a user-defined
> > anonymous function of the form \"(function(data.frame)
> > {...})\". Unless ROWNAMES-ABSENT is non-nil the row names of the
> > data frame are taken from the first column of the org table"
> >   (interactive)
> >   (require 'ess)
> >   (let ((csv-file (make-temp-file "org-table-eval-R-")))
> >     (org-table-export csv-file "orgtbl-to-csv")
> >     (ess-execute (org-table-eval-R-make-expr 
> > 		  R-function csv-file rownames-absent))
> >     (delete-file csv-file)))
> >
> > (defun org-table-eval-R-make-expr (R-function csv-file rownames-absent)
> >   (concat R-function
> > 	  "(read.csv(\"" csv-file "\""
> > 	  (unless rownames-absent ", row.names=1") "))"))
> >
> >
> > (defun org-plot/R (&optional plot-function file rownames-absent)
> >   "Use R to create a plot using the org table at point. If FILE
> > is non-nil then the plot will be written to file (currently as
> > eps, in future the format could easily be determined by the file
> > extension.) Unless ROWNAMES-ABSENT is non-nil the row names will
> > be taken from the first column of the org table."
> >   (interactive)
> >   (require 'ess)
> >   (org-table-eval-R (or plot-function "plot") rownames-absent)
> >   (if file (ess-execute (concat "dev.copy2eps(\"" file "\")"))))
> >
> >
> >
> >
> >> 
> >> Dan
> >> 
> >> 
> >> 
> >> 
> >> > http://cran.r-project.org/doc/FAQ/R-FAQ.html#R-and-Emacs for
> >> > information on running R functions from inside of Emacs.
> >> > 
> >> > Best -- Eric
> >> > 
> >> > On Saturday, July 26, at 19:15, Dan Davison wrote:
> >> >  > R (www.r-project.org) is pretty good for data plotting and statistical
> >> >  > analyses. Here's my effort at the org-table-plot function, using
> >> >  > R. Since R contains a csv importer that can read from stdin, it's
> >> >  > pretty simple. I've tried to code it so that you can provide an
> >> >  > arbitrary R function as the optional argument, so in principle you can
> >> >  > do to your org-table anything that R is capable of in the realms of
> >> >  > data analysis and visualisation.
> >> >  > 
> >> >  > (defun org-table-R-plot (&optional R-function)
> >> >  > "Plot the current table using R. The table is transformed into a dataframe in R. Optional
> >> >  > argument R-function is a string which is either the name of an R
> >> >  > function, or an anonymous function definition of the form (function(d) {...}),
> >> >  > requiring a single argument (the dataframe). The default is to use
> >> >  > the R function 'plot' which produces scatter plots of all pairwise
> >> >  > combinations of columns. An example custom plotting function is:
> >> >  > * plot column 3 against column 1, adding least-squares linear regression fit in blue                                                                                                                               (function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") })                                                                                                                      "
> >> >  >   (interactive)
> >> >  >   (unless R-function (set 'R-function "plot"))
> >> >  >   (let ((file (make-temp-file "org-table-R-plot")))
> >> >  >     (org-table-export file "orgtbl-to-csv")
> >> >  >     (set-buffer (find-file-noselect file))
> >> >  >     (shell-command-on-region
> >> >  >      (point-min) (point-max)
> >> >  >      (concat "Rscript -e 'X11() ; " R-function "(read.csv(\"stdin\")) ; system(\"sleep 60\")'"))
> >> >  >     (delete-file file)))
> >> >  > 
> >> >  > 
> >> >  > R is at www.r-project.org 
> >> >  > (package r-base on ubuntu/debian)
> >> >  > 
> >> >  > Rscript is a command line non-interactive scripting utility that is
> >> >  > bundled automatically with the R installation. I reckon it'll be OK on
> >> >  > OSX but no idea about Windows.  My function doesn't have to be used
> >> >  > for plotting; the R-function argument can be any function operating on
> >> >  > the data from the org-table, producing numerical or graphical output.
> >> >  > 
> >> >  > There's several things that need to be sorted out with my function, e.g.
> >> >  > 
> >> >  > (i) I haven't worked out how to return control to the emacs process
> >> >  > while keeping the plot window there. I tried adding an & to the
> >> >  > shell-command, but that seemed to result in R receiving nothing on
> >> >  > stdin. So I've got that 'sleep 60' hack in there currently; use C-g if
> >> >  > you get bored of your plot.
> >> >  > 
> >> >  > (ii) If the R-function isn't doing graphics, then the call to X11()
> >> >  > gets in the way. X11() would only work on linux/mac OSX(?) anyway.
> >> >  > 
> >> >  > (iii) I'm afraid I don't even know yet how to pass the optional string
> >> >  > argument using M-x org-table-R-plot. Is it possible with some sort of
> >> >  > prefix argument, and an (interactive something) declaration? Anyway,
> >> >  > it seems to work if you evaluate e.g.
> >> >  > 
> >> >  > (org-table-R-plot "(function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") })")
> >> >  > 
> >> >  > Suggestions for improvements welcome!
> >> >  > 
> >> >  > Dan
> >> >  > 
> >> >  > 
> >> >  > 
> >> >  > On Fri, Jul 25, 2008 at 12:07:00PM -0700, Eric Schulte wrote:
> >> >  > > 
> >> >  > > I had some time waiting for things to execute, so I condensed your
> >> >  > > process into a single command (borrowing heavily from
> >> >  > > org-export-table).
> >> >  > > 
> >> >  > > (defun org-table/gnuplot (&optional x-col)
> >> >  > >   "Plot the current table using gnuplot.  Use a prefix argument
> >> >  > > to specify a column to use for the x-coordinates, to use the row
> >> >  > > number for the x-coordinates provide a prefix argument of 0."
> >> >  > >   (interactive "p")
> >> >  > >   (message (format "%S" x-col))
> >> >  > >   (unless (org-at-table-p)
> >> >  > >     (error "No table at point"))
> >> >  > >   (require 'org-exp)
> >> >  > >   (require 'gnuplot)
> >> >  > >   (org-table-align) ;; make sure we have everything we need
> >> >  > >   (let* ((beg (org-table-begin))
> >> >  > > 	 (end (org-table-end))
> >> >  > > 	 (cols (save-excursion
> >> >  > > 		 (goto-char end)
> >> >  > > 		 (backward-char 3)
> >> >  > > 		 (org-table-current-column)))
> >> >  > > 	 (data-beg (if (and 
> >> >  > > 			(goto-char beg)
> >> >  > > 			(re-search-forward org-table-dataline-regexp end t)
> >> >  > > 			(re-search-forward org-table-hline-regexp end t)
> >> >  > > 			(re-search-forward org-table-dataline-regexp end t))
> >> >  > > 		       (match-beginning 0)
> >> >  > > 		     beg))
> >> >  > > 	 (skip (- (line-number-at-pos data-beg) (line-number-at-pos beg)))
> >> >  > > 	 (exp-format (format "orgtbl-to-tsv :skip %d" skip))
> >> >  > > 	 (file (make-temp-file "org-table-plot")))
> >> >  > >     ;; export table
> >> >  > >     (org-table-export file exp-format)
> >> >  > >     (with-temp-buffer
> >> >  > >       ;; write script
> >> >  > >       (insert (org-table/gnuplot-script file x-col cols))
> >> >  > >       ;; graph table
> >> >  > >       (gnuplot-mode)
> >> >  > >       (gnuplot-send-buffer-to-gnuplot)
> >> >  > >       (bury-buffer (get-buffer "*gnuplot*")))
> >> >  > >     (delete-file file)))
> >> >  > > 
> >> >  > > (defun org-table/gnuplot-script (file x-col num-cols)
> >> >  > >   (let ((plot-str "'%s' using %s:%d with lines title '%d'");; "\\\n    ,"
> >> >  > > 	script)
> >> >  > >     (dotimes (col (+ 1 num-cols))
> >> >  > >       (unless (or (and x-col (equal col x-col)) (equal col 0))
> >> >  > > 	(setf script (cons (format plot-str file (or (and x-col (format "%d" x-col)) "") col col) script))))
> >> >  > >     (concat "plot " (mapconcat 'identity (reverse script) "\\\n    ,"))))
> >> >  > > 
> >> >  > > On Friday, July 25, at 17:25, James TD Smith wrote:
> >> >  > >  > On 2008-07-25 08:53:31(-0700), Eric Schulte wrote:
> >> >  > >  > > 
> >> >  > >  > > Any advice for quick graphing of a table in org-mode?
> >> >  > >  > > 
> >> >  > >  > 
> >> >  > >  > I have a setup for plotting data from tables. I'm not sure if it's exactly what
> >> >  > >  > you want, but yoy may find it useful.
> >> >  > >  > 
> >> >  > >  > 1. Add the following to your .emacs:
> >> >  > >  > 
> >> >  > >  > (defun ahkt-plot-table (script)
> >> >  > >  >   "util function to export and plot a table using the supplied
> >> >  > >  > gnuplot `script'"
> >> >  > >  >   (org-table-export)
> >> >  > >  >   (let ((cbuf (current-buffer))
> >> >  > >  > 	(cwin (selected-window)))
> >> >  > >  >     (save-restriction
> >> >  > >  >       (save-excursion
> >> >  > >  > 	(find-file script)
> >> >  > >  > 	(gnuplot-send-buffer-to-gnuplot)
> >> >  > >  > 	(bury-buffer)
> >> >  > >  > 	(bury-buffer (get-buffer "*gnuplot*"))))
> >> >  > >  >     (and (window-live-p cwin) (select-window cwin))
> >> >  > >  >     (switch-to-buffer cbuf)
> >> >  > >  >     (delete-other-windows)))
> >> >  > >  > 
> >> >  > >  > 2. Create a gnuplot script which plots data from a file.
> >> >  > >  > 
> >> >  > >  > 3. Add the following properties to the headline containing the table.
> >> >  > >  > TABLE_EXPORT_FILE <filename in the gnuploy script>
> >> >  > >  > TABLE_EXPORT_FORMAT orgtbl-to-generic :skip 4 :splice t :sep "\t"
> >> >  > >  > 
> >> >  > >  > 4. Add an org link in the table (it must be in the table otherwise the export
> >> >  > >  > doesn't work) as below:
> >> >  > >  > [[elisp:(ahkt-plot-table "<gnuplot script>")][plot table]] 
> >> >  > >  > 
> >> >  > >  > I suggest you put it at the top of the table.
> >> >  > >  > You will then need to adjust the 'skip' parameter in the export format depending
> >> >  > >  > on the number of lines at the top of the table which should not be exported
> >> >  > >  > (hlines, more than one plotting link etc). 
> >> >  > >  > 
> >> >  > >  > 5. You should then be able to open the link, and get a plot of the table
> >> >  > >  > contents.
> >> >  > >  > 
> >> >  > >  > 
> >> >  > >  > --
> >> >  > >  > |-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|
> >> >  > >  > 
> >> >  > >  > 
> >> >  > >  > _______________________________________________
> >> >  > >  > Emacs-orgmode mailing list
> >> >  > >  > Remember: use `Reply All' to send replies to the list.
> >> >  > >  > Emacs-orgmode@gnu.org
> >> >  > >  > http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> >> >  > > 
> >> >  > > -- 
> >> >  > > schulte
> >> >  > > 
> >> >  > > 
> >> >  > > _______________________________________________
> >> >  > > Emacs-orgmode mailing list
> >> >  > > Remember: use `Reply All' to send replies to the list.
> >> >  > > Emacs-orgmode@gnu.org
> >> >  > > http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> >> > 
> >> > -- 
> >> > schulte
> >> 
> >> 
> >> _______________________________________________
> >> Emacs-orgmode mailing list
> >> Remember: use `Reply All' to send replies to the list.
> >> Emacs-orgmode@gnu.org
> >> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> 
> 
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

-- 
http://www.stats.ox.ac.uk/~davison