From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Davison Subject: Re: graphing from org-tables Date: Thu, 22 Jan 2009 11:27:11 -0500 Message-ID: <20090122162711.GA11250@stats.ox.ac.uk> References: <74FDB779-BB93-414F-ABD3-DBA6DE6C528E@gmail.com> <87r640p2ld.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1LQ2Om-0002pd-6Y for emacs-orgmode@gnu.org; Thu, 22 Jan 2009 11:27:20 -0500 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1LQ2Ol-0002oX-6W for emacs-orgmode@gnu.org; Thu, 22 Jan 2009 11:27:19 -0500 Received: from [199.232.76.173] (port=53997 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1LQ2Ok-0002oG-Tv for emacs-orgmode@gnu.org; Thu, 22 Jan 2009 11:27:19 -0500 Received: from markov.stats.ox.ac.uk ([163.1.210.1]:53309) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1LQ2Ok-0008Up-9A for emacs-orgmode@gnu.org; Thu, 22 Jan 2009 11:27:18 -0500 Content-Disposition: inline In-Reply-To: <87r640p2ld.fsf@gmail.com> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: emacs-orgmode@gnu.org On Mon, Dec 22, 2008 at 12:00:14PM -0500, Eric Schulte wrote: > Hi Dan, > <...> > 2) Also interesting is the idea of referencing a table from a block of R > code elsewhere in the org file. I've worked some on processing > blocks of R code in org files in a manner similar to Sweave. > http://github.com/eschulte/org-contrib/tree/master/org-exp-blocks.el > If we could implement a simple means of referencing tables (either > whole tables, or cells/cols/rows of tables) from these R-code blocks > then that seems like it may be fairly powerful. I agree with this. So in the case of my code, I think we would like to be able to do something like #+TBLR: data: columns:(1 2) action:tabulate This would make the output of the analysis/plot of the table pointed to by appear at the location in the file of the #+TBLR line. Perhaps one design aim would be to have the same mechanism work for referencing tables for processing by #TBLFM, org-plot, org-table-R, org-exp-blocks, etc. What ideas to people have about implementation (and syntax) for this sort of table referencing? Dan > > Thanks -- Eric > > Dan Davison writes: > > > On Mon, Jul 28, 2008 at 04:41:52PM +0100, Dan Davison wrote: > >> On Mon, Jul 28, 2008 at 08:32:22AM -0700, Eric Schulte wrote: > >> > Hi Dan, > >> > > >> > One way around the sleep(60) hack may be to create the R graph using > >> > an inferior R process. See > >> > >> Yep, I just this minute asked for help on the Emacs Speaks Statistics > >> (ESS) mailing list! I'll report back if I make progress with that. > >> > >> https://stat.ethz.ch/pipermail/ess-help/2008-July/004785.html > > > > I think it's worth keeping this avenue alive, despite so much progress > > having been made with gnuplot in the interim. So below is an initial > > version of org-table-eval-R, which takes an arbitrary R function, and > > applies it to an org table. As suggested above, this uses > > inferior-ess-mode and therefore requires ess (emacs speaks statistics) > > and R to be installed. > > > > At it's simplest, R-function can be the name of a standard R > > function. So with Eric's first example data set on Worg > > (http://legito.net/worg/org-tutorials/org-plot.php) i.e. with 2 > > dependent variables > > > > (org-table-eval-R "summary") produces (in a new buffer) > > > > first.dependent.var second.dependent.var > > Min. :0.1560 Min. :0.1808 > > 1st Qu.:0.1773 1st Qu.:0.1985 > > Median :0.1900 Median :0.2146 > > Mean :0.2110 Mean :0.2262 > > 3rd Qu.:0.2333 3rd Qu.:0.2375 > > Max. :0.4250 Max. :0.3750 > > > > and (org-table-eval-R "matplot") produces a rough version of Eric's > > gnuplot figure. (see below for a closer reproduction of the figure, > > with the legend etc). > > > > It would be simple to provide org-plot/R which would construct a > > suitable plotting function and pass it to org-table-eval-R, and some > > ready-made R code could be provided for standard plots -- histograms & > > barplots, scatter plots, etc. I've given an initial version of > > org-plot/R below, which gives the option to produce eps output rather > > than a pop-up window (and it would be easy to use any of the R > > graphics devices for output: jpeg, png, ps, pdf, tiff, X11, quartz, > > etc) > > > > But the power of this approach is that it provides not just plotting > > functionality, but the ability to do arbitrary calculations and > > statistical analyses of the org table, via the base R packages and the > > hundreds of user-written packages. > > > > It would have been better to have developed these ideas along with the > > development of org-plot, but unfortunately I let that slip. But > > org-table-eval-R provides a lot of power for anyone that's prepared to > > write some R code. And if there's interest, then some more > > user-friendly functionality could be provided for org users that want > > something easier, but can install ess and R. > > > > Dan > > > > and here's a rough version of the "O R G" grid plot > > > > (org-table-eval-R "(function (x) image(t(as.matrix(x))))" t) > > > > Of course, these can be beautified to your heart's content with lots > > of extra arguments to the R plotting functions. This comes a bit > > closer to the two dependent-variables graph: > > > > (org-table-eval-R "(function(x) { matplot(x, type=\"l\", ylab=\"\") ; legend(\"topright\", legend=c(\"dep var 1\"\,\"dep var 2\"), col=1:2, lty=1:2) })") > > > > > > > > -------------------------------------------------------------- > > (defun org-table-eval-R (R-function &optional rownames-absent) > > "Apply an arbitrary R function to the org table at > > point. R-FUNCTION is an R function (a string) taking one > > argument, which is the R data frame representation of the org > > table. It may be the name of an appropriate R > > function (e.g. \"summary\", \"plot\"), or a user-defined > > anonymous function of the form \"(function(data.frame) > > {...})\". Unless ROWNAMES-ABSENT is non-nil the row names of the > > data frame are taken from the first column of the org table" > > (interactive) > > (require 'ess) > > (let ((csv-file (make-temp-file "org-table-eval-R-"))) > > (org-table-export csv-file "orgtbl-to-csv") > > (ess-execute (org-table-eval-R-make-expr > > R-function csv-file rownames-absent)) > > (delete-file csv-file))) > > > > (defun org-table-eval-R-make-expr (R-function csv-file rownames-absent) > > (concat R-function > > "(read.csv(\"" csv-file "\"" > > (unless rownames-absent ", row.names=1") "))")) > > > > > > (defun org-plot/R (&optional plot-function file rownames-absent) > > "Use R to create a plot using the org table at point. If FILE > > is non-nil then the plot will be written to file (currently as > > eps, in future the format could easily be determined by the file > > extension.) Unless ROWNAMES-ABSENT is non-nil the row names will > > be taken from the first column of the org table." > > (interactive) > > (require 'ess) > > (org-table-eval-R (or plot-function "plot") rownames-absent) > > (if file (ess-execute (concat "dev.copy2eps(\"" file "\")")))) > > > > > > > > > >> > >> Dan > >> > >> > >> > >> > >> > http://cran.r-project.org/doc/FAQ/R-FAQ.html#R-and-Emacs for > >> > information on running R functions from inside of Emacs. > >> > > >> > Best -- Eric > >> > > >> > On Saturday, July 26, at 19:15, Dan Davison wrote: > >> > > R (www.r-project.org) is pretty good for data plotting and statistical > >> > > analyses. Here's my effort at the org-table-plot function, using > >> > > R. Since R contains a csv importer that can read from stdin, it's > >> > > pretty simple. I've tried to code it so that you can provide an > >> > > arbitrary R function as the optional argument, so in principle you can > >> > > do to your org-table anything that R is capable of in the realms of > >> > > data analysis and visualisation. > >> > > > >> > > (defun org-table-R-plot (&optional R-function) > >> > > "Plot the current table using R. The table is transformed into a dataframe in R. Optional > >> > > argument R-function is a string which is either the name of an R > >> > > function, or an anonymous function definition of the form (function(d) {...}), > >> > > requiring a single argument (the dataframe). The default is to use > >> > > the R function 'plot' which produces scatter plots of all pairwise > >> > > combinations of columns. An example custom plotting function is: > >> > > * plot column 3 against column 1, adding least-squares linear regression fit in blue (function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") }) " > >> > > (interactive) > >> > > (unless R-function (set 'R-function "plot")) > >> > > (let ((file (make-temp-file "org-table-R-plot"))) > >> > > (org-table-export file "orgtbl-to-csv") > >> > > (set-buffer (find-file-noselect file)) > >> > > (shell-command-on-region > >> > > (point-min) (point-max) > >> > > (concat "Rscript -e 'X11() ; " R-function "(read.csv(\"stdin\")) ; system(\"sleep 60\")'")) > >> > > (delete-file file))) > >> > > > >> > > > >> > > R is at www.r-project.org > >> > > (package r-base on ubuntu/debian) > >> > > > >> > > Rscript is a command line non-interactive scripting utility that is > >> > > bundled automatically with the R installation. I reckon it'll be OK on > >> > > OSX but no idea about Windows. My function doesn't have to be used > >> > > for plotting; the R-function argument can be any function operating on > >> > > the data from the org-table, producing numerical or graphical output. > >> > > > >> > > There's several things that need to be sorted out with my function, e.g. > >> > > > >> > > (i) I haven't worked out how to return control to the emacs process > >> > > while keeping the plot window there. I tried adding an & to the > >> > > shell-command, but that seemed to result in R receiving nothing on > >> > > stdin. So I've got that 'sleep 60' hack in there currently; use C-g if > >> > > you get bored of your plot. > >> > > > >> > > (ii) If the R-function isn't doing graphics, then the call to X11() > >> > > gets in the way. X11() would only work on linux/mac OSX(?) anyway. > >> > > > >> > > (iii) I'm afraid I don't even know yet how to pass the optional string > >> > > argument using M-x org-table-R-plot. Is it possible with some sort of > >> > > prefix argument, and an (interactive something) declaration? Anyway, > >> > > it seems to work if you evaluate e.g. > >> > > > >> > > (org-table-R-plot "(function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") })") > >> > > > >> > > Suggestions for improvements welcome! > >> > > > >> > > Dan > >> > > > >> > > > >> > > > >> > > On Fri, Jul 25, 2008 at 12:07:00PM -0700, Eric Schulte wrote: > >> > > > > >> > > > I had some time waiting for things to execute, so I condensed your > >> > > > process into a single command (borrowing heavily from > >> > > > org-export-table). > >> > > > > >> > > > (defun org-table/gnuplot (&optional x-col) > >> > > > "Plot the current table using gnuplot. Use a prefix argument > >> > > > to specify a column to use for the x-coordinates, to use the row > >> > > > number for the x-coordinates provide a prefix argument of 0." > >> > > > (interactive "p") > >> > > > (message (format "%S" x-col)) > >> > > > (unless (org-at-table-p) > >> > > > (error "No table at point")) > >> > > > (require 'org-exp) > >> > > > (require 'gnuplot) > >> > > > (org-table-align) ;; make sure we have everything we need > >> > > > (let* ((beg (org-table-begin)) > >> > > > (end (org-table-end)) > >> > > > (cols (save-excursion > >> > > > (goto-char end) > >> > > > (backward-char 3) > >> > > > (org-table-current-column))) > >> > > > (data-beg (if (and > >> > > > (goto-char beg) > >> > > > (re-search-forward org-table-dataline-regexp end t) > >> > > > (re-search-forward org-table-hline-regexp end t) > >> > > > (re-search-forward org-table-dataline-regexp end t)) > >> > > > (match-beginning 0) > >> > > > beg)) > >> > > > (skip (- (line-number-at-pos data-beg) (line-number-at-pos beg))) > >> > > > (exp-format (format "orgtbl-to-tsv :skip %d" skip)) > >> > > > (file (make-temp-file "org-table-plot"))) > >> > > > ;; export table > >> > > > (org-table-export file exp-format) > >> > > > (with-temp-buffer > >> > > > ;; write script > >> > > > (insert (org-table/gnuplot-script file x-col cols)) > >> > > > ;; graph table > >> > > > (gnuplot-mode) > >> > > > (gnuplot-send-buffer-to-gnuplot) > >> > > > (bury-buffer (get-buffer "*gnuplot*"))) > >> > > > (delete-file file))) > >> > > > > >> > > > (defun org-table/gnuplot-script (file x-col num-cols) > >> > > > (let ((plot-str "'%s' using %s:%d with lines title '%d'");; "\\\n ," > >> > > > script) > >> > > > (dotimes (col (+ 1 num-cols)) > >> > > > (unless (or (and x-col (equal col x-col)) (equal col 0)) > >> > > > (setf script (cons (format plot-str file (or (and x-col (format "%d" x-col)) "") col col) script)))) > >> > > > (concat "plot " (mapconcat 'identity (reverse script) "\\\n ,")))) > >> > > > > >> > > > On Friday, July 25, at 17:25, James TD Smith wrote: > >> > > > > On 2008-07-25 08:53:31(-0700), Eric Schulte wrote: > >> > > > > > > >> > > > > > Any advice for quick graphing of a table in org-mode? > >> > > > > > > >> > > > > > >> > > > > I have a setup for plotting data from tables. I'm not sure if it's exactly what > >> > > > > you want, but yoy may find it useful. > >> > > > > > >> > > > > 1. Add the following to your .emacs: > >> > > > > > >> > > > > (defun ahkt-plot-table (script) > >> > > > > "util function to export and plot a table using the supplied > >> > > > > gnuplot `script'" > >> > > > > (org-table-export) > >> > > > > (let ((cbuf (current-buffer)) > >> > > > > (cwin (selected-window))) > >> > > > > (save-restriction > >> > > > > (save-excursion > >> > > > > (find-file script) > >> > > > > (gnuplot-send-buffer-to-gnuplot) > >> > > > > (bury-buffer) > >> > > > > (bury-buffer (get-buffer "*gnuplot*")))) > >> > > > > (and (window-live-p cwin) (select-window cwin)) > >> > > > > (switch-to-buffer cbuf) > >> > > > > (delete-other-windows))) > >> > > > > > >> > > > > 2. Create a gnuplot script which plots data from a file. > >> > > > > > >> > > > > 3. Add the following properties to the headline containing the table. > >> > > > > TABLE_EXPORT_FILE > >> > > > > TABLE_EXPORT_FORMAT orgtbl-to-generic :skip 4 :splice t :sep "\t" > >> > > > > > >> > > > > 4. Add an org link in the table (it must be in the table otherwise the export > >> > > > > doesn't work) as below: > >> > > > > [[elisp:(ahkt-plot-table "")][plot table]] > >> > > > > > >> > > > > I suggest you put it at the top of the table. > >> > > > > You will then need to adjust the 'skip' parameter in the export format depending > >> > > > > on the number of lines at the top of the table which should not be exported > >> > > > > (hlines, more than one plotting link etc). > >> > > > > > >> > > > > 5. You should then be able to open the link, and get a plot of the table > >> > > > > contents. > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > |---| > >> > > > > > >> > > > > > >> > > > > _______________________________________________ > >> > > > > Emacs-orgmode mailing list > >> > > > > Remember: use `Reply All' to send replies to the list. > >> > > > > Emacs-orgmode@gnu.org > >> > > > > http://lists.gnu.org/mailman/listinfo/emacs-orgmode > >> > > > > >> > > > -- > >> > > > schulte > >> > > > > >> > > > > >> > > > _______________________________________________ > >> > > > Emacs-orgmode mailing list > >> > > > Remember: use `Reply All' to send replies to the list. > >> > > > Emacs-orgmode@gnu.org > >> > > > http://lists.gnu.org/mailman/listinfo/emacs-orgmode > >> > > >> > -- > >> > schulte > >> > >> > >> _______________________________________________ > >> Emacs-orgmode mailing list > >> Remember: use `Reply All' to send replies to the list. > >> Emacs-orgmode@gnu.org > >> http://lists.gnu.org/mailman/listinfo/emacs-orgmode > > > _______________________________________________ > Emacs-orgmode mailing list > Remember: use `Reply All' to send replies to the list. > Emacs-orgmode@gnu.org > http://lists.gnu.org/mailman/listinfo/emacs-orgmode -- http://www.stats.ox.ac.uk/~davison