From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Eric Schulte" Subject: Re: graphing from org-tables Date: Mon, 22 Dec 2008 12:00:14 -0500 Message-ID: <87r640p2ld.fsf@gmail.com> References: <4889f6f8.29578c0a.3e24.ffff8395@mx.google.com> <20080725162527.GA54528@yog-sothoth.mohorovi.cc> <488a2450.1e078e0a.2bc0.181f@mx.google.com> <20080726181524.GA10786@stats.ox.ac.uk> <488de67a.1abd600a.16a2.09e7@mx.google.com> <20080728154152.GD21509@stats.ox.ac.uk> <20081221203153.GB14389@stats.ox.ac.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1LEoA9-0000Tf-I3 for emacs-orgmode@gnu.org; Mon, 22 Dec 2008 12:01:49 -0500 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1LEoA7-0000SP-Hy for emacs-orgmode@gnu.org; Mon, 22 Dec 2008 12:01:48 -0500 Received: from [199.232.76.173] (port=48225 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1LEoA6-0000S0-SK for emacs-orgmode@gnu.org; Mon, 22 Dec 2008 12:01:47 -0500 Received: from qw-out-1920.google.com ([74.125.92.148]:37436) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1LEoA6-00029I-5O for emacs-orgmode@gnu.org; Mon, 22 Dec 2008 12:01:46 -0500 Received: by qw-out-1920.google.com with SMTP id 4so773912qwk.24 for ; Mon, 22 Dec 2008 09:01:45 -0800 (PST) In-Reply-To: <20081221203153.GB14389@stats.ox.ac.uk> (Dan Davison's message of "Sun, 21 Dec 2008 15:31:53 -0500") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Dan Davison Cc: Org-mode Hi Dan, This looks very interesting, and I look forward to playing around with these functions (when I have some/any free time). After a quick glance it seems like these functions could be expanded in two different directions... 1) Adapted to a simple calling mechanism like the one used for org-plot which relies solely on a couple of parameters stuffed into a #+PLOT line next to a table. I think this would be a good option for org users who are more comfortable w/R than w/gnuplot. 2) Also interesting is the idea of referencing a table from a block of R code elsewhere in the org file. I've worked some on processing blocks of R code in org files in a manner similar to Sweave. http://github.com/eschulte/org-contrib/tree/master/org-exp-blocks.el If we could implement a simple means of referencing tables (either whole tables, or cells/cols/rows of tables) from these R-code blocks then that seems like it may be fairly powerful. Thanks -- Eric Dan Davison writes: > On Mon, Jul 28, 2008 at 04:41:52PM +0100, Dan Davison wrote: >> On Mon, Jul 28, 2008 at 08:32:22AM -0700, Eric Schulte wrote: >> > Hi Dan, >> > >> > One way around the sleep(60) hack may be to create the R graph using >> > an inferior R process. See >> >> Yep, I just this minute asked for help on the Emacs Speaks Statistics >> (ESS) mailing list! I'll report back if I make progress with that. >> >> https://stat.ethz.ch/pipermail/ess-help/2008-July/004785.html > > I think it's worth keeping this avenue alive, despite so much progress > having been made with gnuplot in the interim. So below is an initial > version of org-table-eval-R, which takes an arbitrary R function, and > applies it to an org table. As suggested above, this uses > inferior-ess-mode and therefore requires ess (emacs speaks statistics) > and R to be installed. > > At it's simplest, R-function can be the name of a standard R > function. So with Eric's first example data set on Worg > (http://legito.net/worg/org-tutorials/org-plot.php) i.e. with 2 > dependent variables > > (org-table-eval-R "summary") produces (in a new buffer) > > first.dependent.var second.dependent.var > Min. :0.1560 Min. :0.1808 > 1st Qu.:0.1773 1st Qu.:0.1985 > Median :0.1900 Median :0.2146 > Mean :0.2110 Mean :0.2262 > 3rd Qu.:0.2333 3rd Qu.:0.2375 > Max. :0.4250 Max. :0.3750 > > and (org-table-eval-R "matplot") produces a rough version of Eric's > gnuplot figure. (see below for a closer reproduction of the figure, > with the legend etc). > > It would be simple to provide org-plot/R which would construct a > suitable plotting function and pass it to org-table-eval-R, and some > ready-made R code could be provided for standard plots -- histograms & > barplots, scatter plots, etc. I've given an initial version of > org-plot/R below, which gives the option to produce eps output rather > than a pop-up window (and it would be easy to use any of the R > graphics devices for output: jpeg, png, ps, pdf, tiff, X11, quartz, > etc) > > But the power of this approach is that it provides not just plotting > functionality, but the ability to do arbitrary calculations and > statistical analyses of the org table, via the base R packages and the > hundreds of user-written packages. > > It would have been better to have developed these ideas along with the > development of org-plot, but unfortunately I let that slip. But > org-table-eval-R provides a lot of power for anyone that's prepared to > write some R code. And if there's interest, then some more > user-friendly functionality could be provided for org users that want > something easier, but can install ess and R. > > Dan > > and here's a rough version of the "O R G" grid plot > > (org-table-eval-R "(function (x) image(t(as.matrix(x))))" t) > > Of course, these can be beautified to your heart's content with lots > of extra arguments to the R plotting functions. This comes a bit > closer to the two dependent-variables graph: > > (org-table-eval-R "(function(x) { matplot(x, type=\"l\", ylab=\"\") ; legend(\"topright\", legend=c(\"dep var 1\"\,\"dep var 2\"), col=1:2, lty=1:2) })") > > > > -------------------------------------------------------------- > (defun org-table-eval-R (R-function &optional rownames-absent) > "Apply an arbitrary R function to the org table at > point. R-FUNCTION is an R function (a string) taking one > argument, which is the R data frame representation of the org > table. It may be the name of an appropriate R > function (e.g. \"summary\", \"plot\"), or a user-defined > anonymous function of the form \"(function(data.frame) > {...})\". Unless ROWNAMES-ABSENT is non-nil the row names of the > data frame are taken from the first column of the org table" > (interactive) > (require 'ess) > (let ((csv-file (make-temp-file "org-table-eval-R-"))) > (org-table-export csv-file "orgtbl-to-csv") > (ess-execute (org-table-eval-R-make-expr > R-function csv-file rownames-absent)) > (delete-file csv-file))) > > (defun org-table-eval-R-make-expr (R-function csv-file rownames-absent) > (concat R-function > "(read.csv(\"" csv-file "\"" > (unless rownames-absent ", row.names=1") "))")) > > > (defun org-plot/R (&optional plot-function file rownames-absent) > "Use R to create a plot using the org table at point. If FILE > is non-nil then the plot will be written to file (currently as > eps, in future the format could easily be determined by the file > extension.) Unless ROWNAMES-ABSENT is non-nil the row names will > be taken from the first column of the org table." > (interactive) > (require 'ess) > (org-table-eval-R (or plot-function "plot") rownames-absent) > (if file (ess-execute (concat "dev.copy2eps(\"" file "\")")))) > > > > >> >> Dan >> >> >> >> >> > http://cran.r-project.org/doc/FAQ/R-FAQ.html#R-and-Emacs for >> > information on running R functions from inside of Emacs. >> > >> > Best -- Eric >> > >> > On Saturday, July 26, at 19:15, Dan Davison wrote: >> > > R (www.r-project.org) is pretty good for data plotting and statistical >> > > analyses. Here's my effort at the org-table-plot function, using >> > > R. Since R contains a csv importer that can read from stdin, it's >> > > pretty simple. I've tried to code it so that you can provide an >> > > arbitrary R function as the optional argument, so in principle you can >> > > do to your org-table anything that R is capable of in the realms of >> > > data analysis and visualisation. >> > > >> > > (defun org-table-R-plot (&optional R-function) >> > > "Plot the current table using R. The table is transformed into a dataframe in R. Optional >> > > argument R-function is a string which is either the name of an R >> > > function, or an anonymous function definition of the form (function(d) {...}), >> > > requiring a single argument (the dataframe). The default is to use >> > > the R function 'plot' which produces scatter plots of all pairwise >> > > combinations of columns. An example custom plotting function is: >> > > * plot column 3 against column 1, adding least-squares linear regression fit in blue (function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") }) " >> > > (interactive) >> > > (unless R-function (set 'R-function "plot")) >> > > (let ((file (make-temp-file "org-table-R-plot"))) >> > > (org-table-export file "orgtbl-to-csv") >> > > (set-buffer (find-file-noselect file)) >> > > (shell-command-on-region >> > > (point-min) (point-max) >> > > (concat "Rscript -e 'X11() ; " R-function "(read.csv(\"stdin\")) ; system(\"sleep 60\")'")) >> > > (delete-file file))) >> > > >> > > >> > > R is at www.r-project.org >> > > (package r-base on ubuntu/debian) >> > > >> > > Rscript is a command line non-interactive scripting utility that is >> > > bundled automatically with the R installation. I reckon it'll be OK on >> > > OSX but no idea about Windows. My function doesn't have to be used >> > > for plotting; the R-function argument can be any function operating on >> > > the data from the org-table, producing numerical or graphical output. >> > > >> > > There's several things that need to be sorted out with my function, e.g. >> > > >> > > (i) I haven't worked out how to return control to the emacs process >> > > while keeping the plot window there. I tried adding an & to the >> > > shell-command, but that seemed to result in R receiving nothing on >> > > stdin. So I've got that 'sleep 60' hack in there currently; use C-g if >> > > you get bored of your plot. >> > > >> > > (ii) If the R-function isn't doing graphics, then the call to X11() >> > > gets in the way. X11() would only work on linux/mac OSX(?) anyway. >> > > >> > > (iii) I'm afraid I don't even know yet how to pass the optional string >> > > argument using M-x org-table-R-plot. Is it possible with some sort of >> > > prefix argument, and an (interactive something) declaration? Anyway, >> > > it seems to work if you evaluate e.g. >> > > >> > > (org-table-R-plot "(function(df) { plot(x=df[,1], y=df[,3]) ; abline(lm(df[,3] ~ df[,1]), col=\"blue\") })") >> > > >> > > Suggestions for improvements welcome! >> > > >> > > Dan >> > > >> > > >> > > >> > > On Fri, Jul 25, 2008 at 12:07:00PM -0700, Eric Schulte wrote: >> > > > >> > > > I had some time waiting for things to execute, so I condensed your >> > > > process into a single command (borrowing heavily from >> > > > org-export-table). >> > > > >> > > > (defun org-table/gnuplot (&optional x-col) >> > > > "Plot the current table using gnuplot. Use a prefix argument >> > > > to specify a column to use for the x-coordinates, to use the row >> > > > number for the x-coordinates provide a prefix argument of 0." >> > > > (interactive "p") >> > > > (message (format "%S" x-col)) >> > > > (unless (org-at-table-p) >> > > > (error "No table at point")) >> > > > (require 'org-exp) >> > > > (require 'gnuplot) >> > > > (org-table-align) ;; make sure we have everything we need >> > > > (let* ((beg (org-table-begin)) >> > > > (end (org-table-end)) >> > > > (cols (save-excursion >> > > > (goto-char end) >> > > > (backward-char 3) >> > > > (org-table-current-column))) >> > > > (data-beg (if (and >> > > > (goto-char beg) >> > > > (re-search-forward org-table-dataline-regexp end t) >> > > > (re-search-forward org-table-hline-regexp end t) >> > > > (re-search-forward org-table-dataline-regexp end t)) >> > > > (match-beginning 0) >> > > > beg)) >> > > > (skip (- (line-number-at-pos data-beg) (line-number-at-pos beg))) >> > > > (exp-format (format "orgtbl-to-tsv :skip %d" skip)) >> > > > (file (make-temp-file "org-table-plot"))) >> > > > ;; export table >> > > > (org-table-export file exp-format) >> > > > (with-temp-buffer >> > > > ;; write script >> > > > (insert (org-table/gnuplot-script file x-col cols)) >> > > > ;; graph table >> > > > (gnuplot-mode) >> > > > (gnuplot-send-buffer-to-gnuplot) >> > > > (bury-buffer (get-buffer "*gnuplot*"))) >> > > > (delete-file file))) >> > > > >> > > > (defun org-table/gnuplot-script (file x-col num-cols) >> > > > (let ((plot-str "'%s' using %s:%d with lines title '%d'");; "\\\n ," >> > > > script) >> > > > (dotimes (col (+ 1 num-cols)) >> > > > (unless (or (and x-col (equal col x-col)) (equal col 0)) >> > > > (setf script (cons (format plot-str file (or (and x-col (format "%d" x-col)) "") col col) script)))) >> > > > (concat "plot " (mapconcat 'identity (reverse script) "\\\n ,")))) >> > > > >> > > > On Friday, July 25, at 17:25, James TD Smith wrote: >> > > > > On 2008-07-25 08:53:31(-0700), Eric Schulte wrote: >> > > > > > >> > > > > > Any advice for quick graphing of a table in org-mode? >> > > > > > >> > > > > >> > > > > I have a setup for plotting data from tables. I'm not sure if it's exactly what >> > > > > you want, but yoy may find it useful. >> > > > > >> > > > > 1. Add the following to your .emacs: >> > > > > >> > > > > (defun ahkt-plot-table (script) >> > > > > "util function to export and plot a table using the supplied >> > > > > gnuplot `script'" >> > > > > (org-table-export) >> > > > > (let ((cbuf (current-buffer)) >> > > > > (cwin (selected-window))) >> > > > > (save-restriction >> > > > > (save-excursion >> > > > > (find-file script) >> > > > > (gnuplot-send-buffer-to-gnuplot) >> > > > > (bury-buffer) >> > > > > (bury-buffer (get-buffer "*gnuplot*")))) >> > > > > (and (window-live-p cwin) (select-window cwin)) >> > > > > (switch-to-buffer cbuf) >> > > > > (delete-other-windows))) >> > > > > >> > > > > 2. Create a gnuplot script which plots data from a file. >> > > > > >> > > > > 3. Add the following properties to the headline containing the table. >> > > > > TABLE_EXPORT_FILE >> > > > > TABLE_EXPORT_FORMAT orgtbl-to-generic :skip 4 :splice t :sep "\t" >> > > > > >> > > > > 4. Add an org link in the table (it must be in the table otherwise the export >> > > > > doesn't work) as below: >> > > > > [[elisp:(ahkt-plot-table "")][plot table]] >> > > > > >> > > > > I suggest you put it at the top of the table. >> > > > > You will then need to adjust the 'skip' parameter in the export format depending >> > > > > on the number of lines at the top of the table which should not be exported >> > > > > (hlines, more than one plotting link etc). >> > > > > >> > > > > 5. You should then be able to open the link, and get a plot of the table >> > > > > contents. >> > > > > >> > > > > >> > > > > -- >> > > > > |---| >> > > > > >> > > > > >> > > > > _______________________________________________ >> > > > > Emacs-orgmode mailing list >> > > > > Remember: use `Reply All' to send replies to the list. >> > > > > Emacs-orgmode@gnu.org >> > > > > http://lists.gnu.org/mailman/listinfo/emacs-orgmode >> > > > >> > > > -- >> > > > schulte >> > > > >> > > > >> > > > _______________________________________________ >> > > > Emacs-orgmode mailing list >> > > > Remember: use `Reply All' to send replies to the list. >> > > > Emacs-orgmode@gnu.org >> > > > http://lists.gnu.org/mailman/listinfo/emacs-orgmode >> > >> > -- >> > schulte >> >> >> _______________________________________________ >> Emacs-orgmode mailing list >> Remember: use `Reply All' to send replies to the list. >> Emacs-orgmode@gnu.org >> http://lists.gnu.org/mailman/listinfo/emacs-orgmode