plotting tables

plotting tables

[Eric Abrahamsen]

I'm trying to get my head around plotting data from org tables, for use
in LaTeX exports. I've been looking at these two pages:

http://orgmode.org/worg/org-tutorials/org-plot.html
http://orgmode.org/worg/org-contrib/babel/languages/ob-doc-gnuplot.html

As far as I can tell, there are two general approaches. One is an org
table with some #+PLOT declarations, and when I call `org-plot/gnuplot'
on it, gnuplot processes the table and shows it to me. Presumably I can
make it save to a file instead, I haven't investigated that yet.

The other approach seems to be using a named table, and then a gnuplot
source block that accepts the table data as a variable. In this case
you're writing an actual gnuplot script, rather than feeding gnuplot
options into a #+PLOT line.

My questions:

1. Are these really mutually incompatible approaches, as they appear to
be?

2. What's my best option if I want the following scenario: I start with
an org file containing an org table, call `org-latex-export-to-pdf' on
that file, get a coffee, and come back to find a nice pdf containing
just the plotted graph (no table). Can someone show me just the barest
example?

3. I've been learning the tikz LaTeX package and am very impressed. Has
anyone used tikz/pgfplots with org?


In return, I promise to add a very hand-holdy explanation to worg,
provided that there isn't already one there that I missed.

Thanks,
Eric

Re: plotting tables

[Dieter Wilhelm]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> 3. I've been learning the tikz LaTeX package and am very impressed. Has
> anyone used tikz/pgfplots with org?

So far I only used these packages within literal latex blocks
(#+begin_latex ...  #+end_latex).

I also used gnuplot, besides others, but in my opinion with
tikz/pgfplots you are able to get the most visually pleasing results for
many technical graphics (especially when integrating the outcome within
LaTeX ;-)).

And besides gnuplot is still too restricted for handling data for my
needs, it has not a proper programming languages.  Nowadays I'm using R
to import, prepare and often also graphing data (within org-mode source
blocks) and directly exporting them to latex documents.

-- 
Best wishes

H. Dieter Wilhelm
Darmstadt
Germany

Re: plotting tables

[John Hendy]

On Sun, Mar 24, 2013 at 1:44 AM, Eric Abrahamsen
<eric@ericabrahamsen.net> wrote:
> I'm trying to get my head around plotting data from org tables, for use
> in LaTeX exports. I've been looking at these two pages:
>
> http://orgmode.org/worg/org-tutorials/org-plot.html
> http://orgmode.org/worg/org-contrib/babel/languages/ob-doc-gnuplot.html
>
> As far as I can tell, there are two general approaches. One is an org
> table with some #+PLOT declarations, and when I call `org-plot/gnuplot'
> on it, gnuplot processes the table and shows it to me. Presumably I can
> make it save to a file instead, I haven't investigated that yet.
>
> The other approach seems to be using a named table, and then a gnuplot
> source block that accepts the table data as a variable. In this case
> you're writing an actual gnuplot script, rather than feeding gnuplot
> options into a #+PLOT line.
>
> My questions:
>
> 1. Are these really mutually incompatible approaches, as they appear to
> be?
>

What do you mean by incompatible? I think if you used them both, you'd
just get two plots. It looks like #+plot just allows one to put some
gnuplot commands in the header above a table and get a plot out. Using
the org-babel approach lets you write straight gnuplot code and call a
data table from elsewhere.

I wrote ob-doc-gnuplot, long ago that I didn't really remember why.
- http://www.mail-archive.com/emacs-orgmode@gnu.org/msg28988.html

It looks like I was running into limitations with the #+plot syntax
and switched to using #+begin_src gnuplot because of that.

If you had a #+plot header and #+begin_src gnuplot with the same code,
both calling the table, if you had the correct :exports and :results
options in the block, you'd probably get to graphs.

> 2. What's my best option if I want the following scenario: I start with
> an org file containing an org table, call `org-latex-export-to-pdf' on
> that file, get a coffee, and come back to find a nice pdf containing
> just the plotted graph (no table). Can someone show me just the barest
> example?

Nice PDF of the plot and nothing else whatsoever, or nice PDF
containing other stuff from your org file and just not the header? If
you just want a standalone pdf, you can just use the file option
(assuming gnuplot will write to a pdf) in your code block:

#+begin_src gnuplot :file plot.pdf :exports results :results output

Something like that?

Otherwise, you could just hide the table in a headline with the :noexport: tag:

* Heading 1

Contains stuff you want in a resultant pdf file.

#+begin_src gnuplot :var data=foo

#+end_src gnuplot

** Data table          :noexport:

#+tblname: foo
| data table here |

That would keep the table hidden in a non-exported headline.

> 3. I've been learning the tikz LaTeX package and am very impressed. Has
> anyone used tikz/pgfplots with org?
>

Yes! Though as with Dieter, I've also migrated to R now. Gnuplot
worked fairly well for simple plotting, but at the end of the day,
most plotting (for me, at least) falls into some sort of workflow:
- get some data
- re-arrange, combine, pre-process some data
- get that data accessible to some program (e.g. gnuplot, R)
- plot results
- interpret results
- present plots in context of some project/endeavor

Sure you can just start feeding gnuplot from a file instead of having
big tables in Org-mode tables (which I find gets extremely cumbersome
for more than maybe 20 or so rows and 10 cols), but for most things
you actually need to *do* other things with that data.

If you're just learning... I'd recommend just switching to learning to
plot with R. Ditch other R stuff if it's intimidating at the moment --
just manipulate your data in Excel or LibreOffice, and teach yourself
the equivalent of plotting in gnuplot with R. Instead of =plot data
using 1:3=, it might be =plot(data[, 1], data[, 3])= in R.

The benefit is that as you learn, you can "grow into R," whereas if
you attempt some level of gnuplot mastery, you'll still be stuck
needing a bunch of other tools for all but the more simply plotting
needs.

For matching fonts/styles in your LaTeX output, there's the handy
=tikzDevice= R library: ETA: gasp! It's been removed from CRAN. You
can still get it, though:
- https://www.nesono.com/node/403

Here's the documentation:
- https://r-forge.r-project.org/scm/viewvc.php/*checkout*/pkg/inst/doc/tikzDevice.pdf?revision=35&root=tikzdevice

You can add frequently used options in ~/.Rprofile (just create it if
it doesn't exist). Of particular interest is setting the
tikzLatexPackages variable. The following would set your document to
use the =mathpazo= font family instead of the default. You could
replace with whatever you wanted (I use lmodern a lot).

options( tikzLatexPackages = c(
getOption( "tikzLatexPackages" ),
"\\usepackage{mathpazo}"
))


From there, you'd do something like so:

#+begin_src R

[any general R code]

tikz(file = "file.tex", width = n, height = n, standAlone = T)

[commands that generate the plot in R]

dev.off()

#+end_src

Now you'll have a .tex file of your plot and can just compile it.
Better yet, just add this after =dev.off()= above:

#+begin_src R

tools::texi2dvi("file.tex", pdf=T)

#+end_src

That would compile the resultant file. I tend to get the output in
some other format to make sure things look good and then do the output
to .tex -> pdf as the final step.

As a bonus to this process, if there's ever any re-arrangement or more
complex annotation to be done, you can add it to the .tex file before
compiling. This is fairly manually intensive, but for some things when
you just couldn't specify it based on the data to have it
automatically plotted, you'd end up doing the same amount of work in R
to add text here and there anyway. By looking at the tikz code, you
can generally figure out the min/max coordinates of the plot and add
whatever additional graphics or text you want. You also get fine-tuned
color/shape changing abilities.


Hope that helps!
John



>
> In return, I promise to add a very hand-holdy explanation to worg,
> provided that there isn't already one there that I missed.
>
> Thanks,
> Eric
>
>

Re: plotting tables

[Eric Abrahamsen]

John Hendy <jw.hendy@gmail.com> writes:

Wow, thank you for this comprehensive response!

> On Sun, Mar 24, 2013 at 1:44 AM, Eric Abrahamsen
> <eric@ericabrahamsen.net> wrote:

[...]

>> 1. Are these really mutually incompatible approaches, as they appear to
>> be?
>>
>
> What do you mean by incompatible? I think if you used them both, you'd
> just get two plots. It looks like #+plot just allows one to put some
> gnuplot commands in the header above a table and get a plot out. Using
> the org-babel approach lets you write straight gnuplot code and call a
> data table from elsewhere.

Right, "incompatible" was the wrong word, I probably meant "orthogonal".
I just meant you'd either use one or the other, not both.

>> 2. What's my best option if I want the following scenario: I start with
>> an org file containing an org table, call `org-latex-export-to-pdf' on
>> that file, get a coffee, and come back to find a nice pdf containing
>> just the plotted graph (no table). Can someone show me just the barest
>> example?
>
> Nice PDF of the plot and nothing else whatsoever, or nice PDF
> containing other stuff from your org file and just not the header?

Sorry, that wasn't clear: there's lots of stuff in the file and I want
all the rest of it, just not the table. A noexport tag is a fine solution to
keeping it out. But I'm still not sure how the resulting plot gets
included: surely I'd need a link to a file in here somewhere?

[...]

>> 3. I've been learning the tikz LaTeX package and am very impressed. Has
>> anyone used tikz/pgfplots with org?
>>
>
> Yes! Though as with Dieter, I've also migrated to R now. Gnuplot
> worked fairly well for simple plotting, but at the end of the day,
> most plotting (for me, at least) falls into some sort of workflow:
> - get some data
> - re-arrange, combine, pre-process some data
> - get that data accessible to some program (e.g. gnuplot, R)
> - plot results
> - interpret results
> - present plots in context of some project/endeavor

Over the past couple of days I think I've decided to move away from
gnuplot. What you and Dieter said is right: pgfplots does everything I
need it to, and it integrates perfectly into my existing latex
documents.

I'm reluctant to learn R: frankly my brain is full right now, and I'm
also not sure I need it. My data is actually very simple and doesn't
require any massaging that the org tables themselves can't handle. I do
have a lot of small tables, though, will need to produce these documents
fairly frequently, and need them to look good.

So in your setup below, you're using R to actually create the plots, but
tikz to style them? Or is R feeding data to tikz/pgfplots, and the
actually plotting is going on there? I found this a little confusing...

What I believe I'm looking for is something like this:

#+tblname: tabul
|  Dec |  Nov |  Oct | Sept |
+------+------+------+------|
| 1575 | 3430 |  332 | 2201 |
| 3118 | 3002 | 2334 | 1053 |

#+begin_src latex :var table=tabul
\begin{tikzpicture}
  \begin{axis}[blah]
  \addplot {table};
  \end{axis}
\end{tikzpicture}
#+end_src

Except that I'll need an intermediate step, so that table headings go
into separate vars and are put in the axis declaration, and the actual
table data comes out in a format that pgfplots can understand (right now
it's a sexp).

Maybe that's where R comes in?

Thanks again for all this information!

E

> Sure you can just start feeding gnuplot from a file instead of having
> big tables in Org-mode tables (which I find gets extremely cumbersome
> for more than maybe 20 or so rows and 10 cols), but for most things
> you actually need to *do* other things with that data.
>
> If you're just learning... I'd recommend just switching to learning to
> plot with R. Ditch other R stuff if it's intimidating at the moment --
> just manipulate your data in Excel or LibreOffice, and teach yourself
> the equivalent of plotting in gnuplot with R. Instead of =plot data
> using 1:3=, it might be =plot(data[, 1], data[, 3])= in R.
>
> The benefit is that as you learn, you can "grow into R," whereas if
> you attempt some level of gnuplot mastery, you'll still be stuck
> needing a bunch of other tools for all but the more simply plotting
> needs.
>
> For matching fonts/styles in your LaTeX output, there's the handy
> =tikzDevice= R library: ETA: gasp! It's been removed from CRAN. You
> can still get it, though:
> - https://www.nesono.com/node/403
>
> Here's the documentation:
> - https://r-forge.r-project.org/scm/viewvc.php/*checkout*/pkg/inst/doc/tikzDevice.pdf?revision=35&root=tikzdevice
>
> You can add frequently used options in ~/.Rprofile (just create it if
> it doesn't exist). Of particular interest is setting the
> tikzLatexPackages variable. The following would set your document to
> use the =mathpazo= font family instead of the default. You could
> replace with whatever you wanted (I use lmodern a lot).
>
> options( tikzLatexPackages = c(
> getOption( "tikzLatexPackages" ),
> "\\usepackage{mathpazo}"
> ))
>
>
> From there, you'd do something like so:
>
> #+begin_src R
>
> [any general R code]
>
> tikz(file = "file.tex", width = n, height = n, standAlone = T)
>
> [commands that generate the plot in R]
>
> dev.off()
>
> #+end_src
>
> Now you'll have a .tex file of your plot and can just compile it.
> Better yet, just add this after =dev.off()= above:
>
> #+begin_src R
>
> tools::texi2dvi("file.tex", pdf=T)
>
> #+end_src
>
> That would compile the resultant file. I tend to get the output in
> some other format to make sure things look good and then do the output
> to .tex -> pdf as the final step.
>
> As a bonus to this process, if there's ever any re-arrangement or more
> complex annotation to be done, you can add it to the .tex file before
> compiling. This is fairly manually intensive, but for some things when
> you just couldn't specify it based on the data to have it
> automatically plotted, you'd end up doing the same amount of work in R
> to add text here and there anyway. By looking at the tikz code, you
> can generally figure out the min/max coordinates of the plot and add
> whatever additional graphics or text you want. You also get fine-tuned
> color/shape changing abilities.
>
>
> Hope that helps!
> John
>
>
>
>>
>> In return, I promise to add a very hand-holdy explanation to worg,
>> provided that there isn't already one there that I missed.
>>
>> Thanks,
>> Eric
>>
>>

Re: plotting tables

[John Hendy]

On Mon, Mar 25, 2013 at 10:09 PM, Eric Abrahamsen
<eric@ericabrahamsen.net> wrote:
> John Hendy <jw.hendy@gmail.com> writes:
>
> Wow, thank you for this comprehensive response!
>

No problem. I get excited about this stuff :)

>> On Sun, Mar 24, 2013 at 1:44 AM, Eric Abrahamsen
>> <eric@ericabrahamsen.net> wrote:
>
> [...]
>
>>> 1. Are these really mutually incompatible approaches, as they appear to
>>> be?
>>>
>>
>> What do you mean by incompatible? I think if you used them both, you'd
>> just get two plots. It looks like #+plot just allows one to put some
>> gnuplot commands in the header above a table and get a plot out. Using
>> the org-babel approach lets you write straight gnuplot code and call a
>> data table from elsewhere.
>
> Right, "incompatible" was the wrong word, I probably meant "orthogonal".
> I just meant you'd either use one or the other, not both.

I'd say so. I think of them as a minimalist or full-fledged pair of options.

>
>>> 2. What's my best option if I want the following scenario: I start with
>>> an org file containing an org table, call `org-latex-export-to-pdf' on
>>> that file, get a coffee, and come back to find a nice pdf containing
>>> just the plotted graph (no table). Can someone show me just the barest
>>> example?
>>
>> Nice PDF of the plot and nothing else whatsoever, or nice PDF
>> containing other stuff from your org file and just not the header?
>
> Sorry, that wasn't clear: there's lots of stuff in the file and I want
> all the rest of it, just not the table. A noexport tag is a fine solution to
> keeping it out. But I'm still not sure how the resulting plot gets
> included: surely I'd need a link to a file in here somewhere?
>

The plot gets included via your babel block heading. If I do this:

#+begin_src R :exports results :results output graphics  :file plot.png

x <- 1:10
y <- x^2

plot(x,y)

#+end_src

If I then execute the block with C-c C-c, I'll get a line like this:

#+RESULTS:
[[file:plot.png]]

That inputs it into the PDF upon export.

If you name your block code block, you can fiddle with the output via
typical #+attr_latex settings. Set a name in the block like so (also
moving the :file option for tidyness):

#+name: square
#+header: :file plot.png
#+begin_src R :exports results :results output graphics

This changes the #+RESULTS block to this:

#+RESULTS: square
[[file:plot.png]]

Without a named result, if there's anything immediately before the
#+RESULTS, Babel won't recognize the pre-existing output and will add
another block. With named blocks, it figures it out and you can then
do this:

#+begin_center
#+attr_latex :width 0.75\textwidth
#+RESULTS: square
[[file:plot.png]]
#+end_center

If you change the contents of the =square= src block (say, changing
the file output name), it will be replaced in that block while leaving
your image settings alone.

> [...]
>
>>> 3. I've been learning the tikz LaTeX package and am very impressed. Has
>>> anyone used tikz/pgfplots with org?
>>>
>>
>> Yes! Though as with Dieter, I've also migrated to R now. Gnuplot
>> worked fairly well for simple plotting, but at the end of the day,
>> most plotting (for me, at least) falls into some sort of workflow:
>> - get some data
>> - re-arrange, combine, pre-process some data
>> - get that data accessible to some program (e.g. gnuplot, R)
>> - plot results
>> - interpret results
>> - present plots in context of some project/endeavor
>
> Over the past couple of days I think I've decided to move away from
> gnuplot. What you and Dieter said is right: pgfplots does everything I
> need it to, and it integrates perfectly into my existing latex
> documents.
>
> I'm reluctant to learn R: frankly my brain is full right now, and I'm
> also not sure I need it. My data is actually very simple and doesn't
> require any massaging that the org tables themselves can't handle. I do
> have a lot of small tables, though, will need to produce these documents
> fairly frequently, and need them to look good.
>

You know your data best; if the solution works for you and is
efficient... go for it!

> So in your setup below, you're using R to actually create the plots, but
> tikz to style them? Or is R feeding data to tikz/pgfplots, and the
> actually plotting is going on there? I found this a little confusing...
>

Yup -- tikzDevice just takes the R graphics and draws it with tikz.

#+begin_src R :results silent

# uncomment if you don't have these installed
# install.packages("filehash")
# install.packages("tikzDevice", repos="http://R-Forge.R-project.org")

# you also need the LaTeX preview package
# I have this in .Rprofile:

### begin .Rprofile ###
# options (tikzDefaultEngine = 'pdftex')
# options (tikzDocumentDeclaration = "\\documentclass{article}")
# options (tikzFooter = c("\\end{document}"))

# options (tikzLatexPackages = c(
#   "\\usepackage{tikz}",
#   "\\usepackage[pdftex,active,tightpage]{preview}",
#   "\\setlength\\PreviewBorder{0pt}",
#   "\\PreviewEnvironment{pgfpicture}",
#   "\\usepackage{lmodern}",
#   "\\renewcommand{\\familydefault}{\\sfdefault}")
# )

# options (tikzMetricPackages = c(
#   "\\usepackage[utf8]{inputenc}",
#   "\\usepackage[T1]{fontenc}",
#   "\\usetikzlibrary{calc}")
# )
### end .Rprofile ###

library(tikzDevice)
library(ggplot2)

p <- ggplot(mtcars, aes(mpg, wt, colour = factor(gear))) + geom_point()
p <- p + facet_grid(. ~ cyl) + theme_bw()

tikz("plot.tex", width=9, height=6, standAlone = T)
p
dev.off()

tools::texi2pdf("plot.tex", pdf = T)
#+end_src

Take a look at the .tex generated from that.

> What I believe I'm looking for is something like this:
>
> #+tblname: tabul
> |  Dec |  Nov |  Oct | Sept |
> +------+------+------+------|
> | 1575 | 3430 |  332 | 2201 |
> | 3118 | 3002 | 2334 | 1053 |
>
> #+begin_src latex :var table=tabul
> \begin{tikzpicture}
>   \begin{axis}[blah]
>   \addplot {table};
>   \end{axis}
> \end{tikzpicture}
> #+end_src
>
> Except that I'll need an intermediate step, so that table headings go
> into separate vars and are put in the axis declaration, and the actual
> table data comes out in a format that pgfplots can understand (right now
> it's a sexp).
>

I don't have enough experience with this. I got fairly into tikz for a
while (even got an example submitted to the TikZ/PDF examples repo!
[1]), but my issue with using it for plotting is that, in my opinion,
it has an even higher learning curve than R. The other issue, at least
from my experience, is that even if you generally know what you're
doing, it's *unbelievably* easy to mess up syntax somewhere, miss a
comma, goof up some combo of ={}= and/or semicolons... I've spent
massive amounts of time trying to debug my TikZ creations.

It's extremely well documented... but that doesn't mean you even know
what to search for in the documentation

From looking at pgfplots, it looks much easier to generate than hand
plotting with TikZ. Compare the code between these:
- Straight TikZ: http://www.texample.net/tikz/examples/feature/plotting/
- pgfplots: http://pgfplots.sourceforge.net/gallery.html

Anyway, again, your call... I just find the bulk output with
R/tikzDevice and then minor tweaking in actual TikZ code to work
fairly well. Then again, I'm back to editing a full-on TikZ file vs.
the syntax of pgfplots which does, indeed, look relatively simple, at
least for some of their examples.

> Maybe that's where R comes in?
>
> Thanks again for all this information!
>

Good luck in your decision and I hope you find something that works
well for you!

John

[1] http://www.texample.net/tikz/examples/bayes/

> E
>
>> Sure you can just start feeding gnuplot from a file instead of having
>> big tables in Org-mode tables (which I find gets extremely cumbersome
>> for more than maybe 20 or so rows and 10 cols), but for most things
>> you actually need to *do* other things with that data.
>>
>> If you're just learning... I'd recommend just switching to learning to
>> plot with R. Ditch other R stuff if it's intimidating at the moment --
>> just manipulate your data in Excel or LibreOffice, and teach yourself
>> the equivalent of plotting in gnuplot with R. Instead of =plot data
>> using 1:3=, it might be =plot(data[, 1], data[, 3])= in R.
>>
>> The benefit is that as you learn, you can "grow into R," whereas if
>> you attempt some level of gnuplot mastery, you'll still be stuck
>> needing a bunch of other tools for all but the more simply plotting
>> needs.
>>
>> For matching fonts/styles in your LaTeX output, there's the handy
>> =tikzDevice= R library: ETA: gasp! It's been removed from CRAN. You
>> can still get it, though:
>> - https://www.nesono.com/node/403
>>
>> Here's the documentation:
>> - https://r-forge.r-project.org/scm/viewvc.php/*checkout*/pkg/inst/doc/tikzDevice.pdf?revision=35&root=tikzdevice
>>
>> You can add frequently used options in ~/.Rprofile (just create it if
>> it doesn't exist). Of particular interest is setting the
>> tikzLatexPackages variable. The following would set your document to
>> use the =mathpazo= font family instead of the default. You could
>> replace with whatever you wanted (I use lmodern a lot).
>>
>> options( tikzLatexPackages = c(
>> getOption( "tikzLatexPackages" ),
>> "\\usepackage{mathpazo}"
>> ))
>>
>>
>> From there, you'd do something like so:
>>
>> #+begin_src R
>>
>> [any general R code]
>>
>> tikz(file = "file.tex", width = n, height = n, standAlone = T)
>>
>> [commands that generate the plot in R]
>>
>> dev.off()
>>
>> #+end_src
>>
>> Now you'll have a .tex file of your plot and can just compile it.
>> Better yet, just add this after =dev.off()= above:
>>
>> #+begin_src R
>>
>> tools::texi2dvi("file.tex", pdf=T)
>>
>> #+end_src
>>
>> That would compile the resultant file. I tend to get the output in
>> some other format to make sure things look good and then do the output
>> to .tex -> pdf as the final step.
>>
>> As a bonus to this process, if there's ever any re-arrangement or more
>> complex annotation to be done, you can add it to the .tex file before
>> compiling. This is fairly manually intensive, but for some things when
>> you just couldn't specify it based on the data to have it
>> automatically plotted, you'd end up doing the same amount of work in R
>> to add text here and there anyway. By looking at the tikz code, you
>> can generally figure out the min/max coordinates of the plot and add
>> whatever additional graphics or text you want. You also get fine-tuned
>> color/shape changing abilities.
>>
>>
>> Hope that helps!
>> John
>>
>>
>>
>>>
>>> In return, I promise to add a very hand-holdy explanation to worg,
>>> provided that there isn't already one there that I missed.
>>>
>>> Thanks,
>>> Eric
>>>
>>>
>
>