Re: graphing from org-tables

From: "Eric Schulte" <schulte.eric@gmail.com>
To: Dan Davison <davison@stats.ox.ac.uk>
Cc: Org-mode <emacs-orgmode@gnu.org>
Subject: Re: graphing from org-tables
Date: Mon, 22 Dec 2008 12:00:14 -0500	[thread overview]
Message-ID: <87r640p2ld.fsf@gmail.com> (raw)
In-Reply-To: <20081221203153.GB14389@stats.ox.ac.uk> (Dan Davison's message of "Sun, 21 Dec 2008 15:31:53 -0500")

Hi Dan,

This looks very interesting, and I look forward to playing around with
these functions (when I have some/any free time).

After a quick glance it seems like these functions could be expanded in
two different directions...

1) Adapted to a simple calling mechanism like the one used for org-plot
   which relies solely on a couple of parameters stuffed into a #+PLOT
   line next to a table.  I think this would be a good option for org
   users who are more comfortable w/R than w/gnuplot.

2) Also interesting is the idea of referencing a table from a block of R
   code elsewhere in the org file.  I've worked some on processing
   blocks of R code in org files in a manner similar to Sweave.
   http://github.com/eschulte/org-contrib/tree/master/org-exp-blocks.el
   If we could implement a simple means of referencing tables (either
   whole tables, or cells/cols/rows of tables) from these R-code blocks
   then that seems like it may be fairly powerful.

Thanks -- Eric

Dan Davison <davison@stats.ox.ac.uk> writes:

> On Mon, Jul 28, 2008 at 04:41:52PM +0100, Dan Davison wrote:
>> On Mon, Jul 28, 2008 at 08:32:22AM -0700, Eric Schulte wrote:
>> > Hi Dan,
>> > 
>> > One way around the sleep(60) hack may be to create the R graph using
>> > an inferior R process.  See
>> 
>> Yep, I just this minute asked for help on the Emacs Speaks Statistics
>> (ESS) mailing list! I'll report back if I make progress with that.
>> 
>> https://stat.ethz.ch/pipermail/ess-help/2008-July/004785.html
>
> I think it's worth keeping this avenue alive, despite so much progress
> having been made with gnuplot in the interim. So below is an initial
> version of org-table-eval-R, which takes an arbitrary R function, and
> applies it to an org table. As suggested above, this uses
> inferior-ess-mode and therefore requires ess (emacs speaks statistics)
> and R to be installed.
>
> At it's simplest, R-function can be the name of a standard R
> function. So with Eric's first example data set on Worg
> (http://legito.net/worg/org-tutorials/org-plot.php) i.e. with 2
> dependent variables
>
> (org-table-eval-R "summary") produces (in a new buffer)
>
>  first.dependent.var second.dependent.var
>  Min.   :0.1560      Min.   :0.1808      
>  1st Qu.:0.1773      1st Qu.:0.1985      
>  Median :0.1900      Median :0.2146      
>  Mean   :0.2110      Mean   :0.2262      
>  3rd Qu.:0.2333      3rd Qu.:0.2375      
>  Max.   :0.4250      Max.   :0.3750      
>
> and (org-table-eval-R "matplot") produces a rough version of Eric's
> gnuplot figure. (see below for a closer reproduction of the figure,
> with the legend etc).
>
> It would be simple to provide org-plot/R which would construct a
> suitable plotting function and pass it to org-table-eval-R, and some
> ready-made R code could be provided for standard plots -- histograms &
> barplots, scatter plots, etc. I've given an initial version of
> org-plot/R below, which gives the option to produce eps output rather
> than a pop-up window (and it would be easy to use any of the R
> graphics devices for output: jpeg, png, ps, pdf, tiff, X11, quartz,
> etc)
>
> But the power of this approach is that it provides not just plotting
> functionality, but the ability to do arbitrary calculations and
> statistical analyses of the org table, via the base R packages and the
> hundreds of user-written packages.
>
> It would have been better to have developed these ideas along with the
> development of org-plot, but unfortunately I let that slip. But
> org-table-eval-R provides a lot of power for anyone that's prepared to
> write some R code. And if there's interest, then some more
> user-friendly functionality could be provided for org users that want
> something easier, but can install ess and R.
>
> Dan
>
> and here's a rough version of the "O R G" grid plot
>
> (org-table-eval-R "(function (x) image(t(as.matrix(x))))" t)
>
> Of course, these can be beautified to your heart's content with lots
> of extra arguments to the R plotting functions. This comes a bit
> closer to the two dependent-variables graph:
>
> (org-table-eval-R "(function(x) { matplot(x, type=\"l\", ylab=\"\") ; legend(\"topright\", legend=c(\"dep var 1\"\,\"dep var 2\"), col=1:2, lty=1:2) })")
>
>
>
> --------------------------------------------------------------
> (defun org-table-eval-R (R-function &optional rownames-absent)
>   "Apply an arbitrary R function to the org table at
> point. R-FUNCTION is an R function (a string) taking one
> argument, which is the R data frame representation of the org
> table. It may be the name of an appropriate R
> function (e.g. \"summary\", \"plot\"), or a user-defined
> anonymous function of the form \"(function(data.frame)
> {...})\". Unless ROWNAMES-ABSENT is non-nil the row names of the
> data frame are taken from the first column of the org table"
>   (interactive)
>   (require 'ess)
>   (let ((csv-file (make-temp-file "org-table-eval-R-")))
>     (org-table-export csv-file "orgtbl-to-csv")
>     (ess-execute (org-table-eval-R-make-expr 
> 		  R-function csv-file rownames-absent))
>     (delete-file csv-file)))
>
> (defun org-table-eval-R-make-expr (R-function csv-file rownames-absent)
>   (concat R-function
> 	  "(read.csv(\"" csv-file "\""
> 	  (unless rownames-absent ", row.names=1") "))"))
>
>
> (defun org-plot/R (&optional plot-function file rownames-absent)
>   "Use R to create a plot using the org table at point. If FILE
> is non-nil then the plot will be written to file (currently as
> eps, in future the format could easily be determined by the file
> extension.) Unless ROWNAMES-ABSENT is non-nil the row names will
> be taken from the first column of the org table."
>   (interactive)
>   (require 'ess)
>   (org-table-eval-R (or plot-function "plot") rownames-absent)
>   (if file (ess-execute (concat "dev.copy2eps(\"" file "\")"))))
>
>
>
>
>> 
>> Dan
>> 
>> 
>> 
>> 
>> > http://cran.r-project.org/doc/FAQ/R-FAQ.html#R-and-Emacs for
>> > information on running R functions from inside of Emacs.
>> > 
>> > Best -- Eric
>> > 
>> > On Saturday, July 26, at 19:15, Dan Davison wrote:
>> >  > R (www.r-project.org) is pretty good for data plotting and statistical
>> >  > analyses. Here's my effort at the org-table-plot function, using
>> >  > R. Since R contains a csv importer that can read from stdin, it's
>> >  > pretty simple. I've tried to code it so that you can provide an
>> >  > arbitrary R function as the optional argument, so in principle you can
>> >  > do to your org-table anything that R is capable of in the realms of
>> >  > data analysis and visualisation.
>> >  > 
>> >  > (defun org-table-R-plot (&optional R-function)
>> >  > "Plot the current table using R. The table is transformed into a dataframe in R. Optional
>> >  > argument R-function is a string which is either the name of an R
>> >  > function, or an anonymous function definition of the form (function(d) {...}),
>> >  > requiring a single argument (the dataframe). The default is to use
>> >  > the R function 'plot' which produces scatter plots of all pairwise
>> >  > combinations of columns. An example custom plotting function is:
>> >  > * plot column 3 against column 1, adding least-squares linear regression fit in blue                                                                                                                               (function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") })                                                                                                                      "
>> >  >   (interactive)
>> >  >   (unless R-function (set 'R-function "plot"))
>> >  >   (let ((file (make-temp-file "org-table-R-plot")))
>> >  >     (org-table-export file "orgtbl-to-csv")
>> >  >     (set-buffer (find-file-noselect file))
>> >  >     (shell-command-on-region
>> >  >      (point-min) (point-max)
>> >  >      (concat "Rscript -e 'X11() ; " R-function "(read.csv(\"stdin\")) ; system(\"sleep 60\")'"))
>> >  >     (delete-file file)))
>> >  > 
>> >  > 
>> >  > R is at www.r-project.org 
>> >  > (package r-base on ubuntu/debian)
>> >  > 
>> >  > Rscript is a command line non-interactive scripting utility that is
>> >  > bundled automatically with the R installation. I reckon it'll be OK on
>> >  > OSX but no idea about Windows.  My function doesn't have to be used
>> >  > for plotting; the R-function argument can be any function operating on
>> >  > the data from the org-table, producing numerical or graphical output.
>> >  > 
>> >  > There's several things that need to be sorted out with my function, e.g.
>> >  > 
>> >  > (i) I haven't worked out how to return control to the emacs process
>> >  > while keeping the plot window there. I tried adding an & to the
>> >  > shell-command, but that seemed to result in R receiving nothing on
>> >  > stdin. So I've got that 'sleep 60' hack in there currently; use C-g if
>> >  > you get bored of your plot.
>> >  > 
>> >  > (ii) If the R-function isn't doing graphics, then the call to X11()
>> >  > gets in the way. X11() would only work on linux/mac OSX(?) anyway.
>> >  > 
>> >  > (iii) I'm afraid I don't even know yet how to pass the optional string
>> >  > argument using M-x org-table-R-plot. Is it possible with some sort of
>> >  > prefix argument, and an (interactive something) declaration? Anyway,
>> >  > it seems to work if you evaluate e.g.
>> >  > 
>> >  > (org-table-R-plot "(function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") })")
>> >  > 
>> >  > Suggestions for improvements welcome!
>> >  > 
>> >  > Dan
>> >  > 
>> >  > 
>> >  > 
>> >  > On Fri, Jul 25, 2008 at 12:07:00PM -0700, Eric Schulte wrote:
>> >  > > 
>> >  > > I had some time waiting for things to execute, so I condensed your
>> >  > > process into a single command (borrowing heavily from
>> >  > > org-export-table).
>> >  > > 
>> >  > > (defun org-table/gnuplot (&optional x-col)
>> >  > >   "Plot the current table using gnuplot.  Use a prefix argument
>> >  > > to specify a column to use for the x-coordinates, to use the row
>> >  > > number for the x-coordinates provide a prefix argument of 0."
>> >  > >   (interactive "p")
>> >  > >   (message (format "%S" x-col))
>> >  > >   (unless (org-at-table-p)
>> >  > >     (error "No table at point"))
>> >  > >   (require 'org-exp)
>> >  > >   (require 'gnuplot)
>> >  > >   (org-table-align) ;; make sure we have everything we need
>> >  > >   (let* ((beg (org-table-begin))
>> >  > > 	 (end (org-table-end))
>> >  > > 	 (cols (save-excursion
>> >  > > 		 (goto-char end)
>> >  > > 		 (backward-char 3)
>> >  > > 		 (org-table-current-column)))
>> >  > > 	 (data-beg (if (and 
>> >  > > 			(goto-char beg)
>> >  > > 			(re-search-forward org-table-dataline-regexp end t)
>> >  > > 			(re-search-forward org-table-hline-regexp end t)
>> >  > > 			(re-search-forward org-table-dataline-regexp end t))
>> >  > > 		       (match-beginning 0)
>> >  > > 		     beg))
>> >  > > 	 (skip (- (line-number-at-pos data-beg) (line-number-at-pos beg)))
>> >  > > 	 (exp-format (format "orgtbl-to-tsv :skip %d" skip))
>> >  > > 	 (file (make-temp-file "org-table-plot")))
>> >  > >     ;; export table
>> >  > >     (org-table-export file exp-format)
>> >  > >     (with-temp-buffer
>> >  > >       ;; write script
>> >  > >       (insert (org-table/gnuplot-script file x-col cols))
>> >  > >       ;; graph table
>> >  > >       (gnuplot-mode)
>> >  > >       (gnuplot-send-buffer-to-gnuplot)
>> >  > >       (bury-buffer (get-buffer "*gnuplot*")))
>> >  > >     (delete-file file)))
>> >  > > 
>> >  > > (defun org-table/gnuplot-script (file x-col num-cols)
>> >  > >   (let ((plot-str "'%s' using %s:%d with lines title '%d'");; "\\\n    ,"
>> >  > > 	script)
>> >  > >     (dotimes (col (+ 1 num-cols))
>> >  > >       (unless (or (and x-col (equal col x-col)) (equal col 0))
>> >  > > 	(setf script (cons (format plot-str file (or (and x-col (format "%d" x-col)) "") col col) script))))
>> >  > >     (concat "plot " (mapconcat 'identity (reverse script) "\\\n    ,"))))
>> >  > > 
>> >  > > On Friday, July 25, at 17:25, James TD Smith wrote:
>> >  > >  > On 2008-07-25 08:53:31(-0700), Eric Schulte wrote:
>> >  > >  > > 
>> >  > >  > > Any advice for quick graphing of a table in org-mode?
>> >  > >  > > 
>> >  > >  > 
>> >  > >  > I have a setup for plotting data from tables. I'm not sure if it's exactly what
>> >  > >  > you want, but yoy may find it useful.
>> >  > >  > 
>> >  > >  > 1. Add the following to your .emacs:
>> >  > >  > 
>> >  > >  > (defun ahkt-plot-table (script)
>> >  > >  >   "util function to export and plot a table using the supplied
>> >  > >  > gnuplot `script'"
>> >  > >  >   (org-table-export)
>> >  > >  >   (let ((cbuf (current-buffer))
>> >  > >  > 	(cwin (selected-window)))
>> >  > >  >     (save-restriction
>> >  > >  >       (save-excursion
>> >  > >  > 	(find-file script)
>> >  > >  > 	(gnuplot-send-buffer-to-gnuplot)
>> >  > >  > 	(bury-buffer)
>> >  > >  > 	(bury-buffer (get-buffer "*gnuplot*"))))
>> >  > >  >     (and (window-live-p cwin) (select-window cwin))
>> >  > >  >     (switch-to-buffer cbuf)
>> >  > >  >     (delete-other-windows)))
>> >  > >  > 
>> >  > >  > 2. Create a gnuplot script which plots data from a file.
>> >  > >  > 
>> >  > >  > 3. Add the following properties to the headline containing the table.
>> >  > >  > TABLE_EXPORT_FILE <filename in the gnuploy script>
>> >  > >  > TABLE_EXPORT_FORMAT orgtbl-to-generic :skip 4 :splice t :sep "\t"
>> >  > >  > 
>> >  > >  > 4. Add an org link in the table (it must be in the table otherwise the export
>> >  > >  > doesn't work) as below:
>> >  > >  > [[elisp:(ahkt-plot-table "<gnuplot script>")][plot table]] 
>> >  > >  > 
>> >  > >  > I suggest you put it at the top of the table.
>> >  > >  > You will then need to adjust the 'skip' parameter in the export format depending
>> >  > >  > on the number of lines at the top of the table which should not be exported
>> >  > >  > (hlines, more than one plotting link etc). 
>> >  > >  > 
>> >  > >  > 5. You should then be able to open the link, and get a plot of the table
>> >  > >  > contents.
>> >  > >  > 
>> >  > >  > 
>> >  > >  > --
>> >  > >  > |-<James TD Smith>-<email/ahktenzero@mohorovi.cc>-|
>> >  > >  > 
>> >  > >  > 
>> >  > >  > _______________________________________________
>> >  > >  > Emacs-orgmode mailing list
>> >  > >  > Remember: use `Reply All' to send replies to the list.
>> >  > >  > Emacs-orgmode@gnu.org
>> >  > >  > http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>> >  > > 
>> >  > > -- 
>> >  > > schulte
>> >  > > 
>> >  > > 
>> >  > > _______________________________________________
>> >  > > Emacs-orgmode mailing list
>> >  > > Remember: use `Reply All' to send replies to the list.
>> >  > > Emacs-orgmode@gnu.org
>> >  > > http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>> > 
>> > -- 
>> > schulte
>> 
>> 
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Remember: use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode