org-plot (generating graphs from org-mode)

org-plot (generating graphs from org-mode)

[Eric Schulte]

Hi,

org-plot enables generation of graphs from data stored in org tables
(maybe eventually elsewhere in org files).

It is available on github at
http://github.com/eschulte/org-plot/tree/master or with

  git clone git://github.com/eschulte/org-plot.git

Included with org-plot.el is an example.org file which should quickly
show off it's features, options and abilities.

While this is not completed, it is functional, and I figure better to
get early feedback then wait until it's all polished.  Please give it
a look and let me know what you think.

Thanks -- Eric

Re: org-plot (generating graphs from org-mode)

[Manish]

  On Tue, Aug 26, 2008 at 7:23 AM, Eric Schulte wrote:
  > Hi,
  >
  > org-plot enables generation of graphs from data stored in org tables
  > (maybe eventually elsewhere in org files).
  >
  > It is available on github at
  > http://github.com/eschulte/org-plot/tree/master or with
  >
  >  git clone git://github.com/eschulte/org-plot.git
  >
  > Included with org-plot.el is an example.org file which should quickly
  > show off it's features, options and abilities.
  >
  > While this is not completed, it is functional, and I figure better to
  > get early feedback then wait until it's all polished.  Please give it
  > a look and let me know what you think.

Awesome!  This has wonderful possibilities for my work.  I just wish I
had this a couple of months ago.  But I hope I am going to have an
opportunity to use this again soon.

Works great on GNU Emacs 22.1.1 with Org 6.06b and gnuplot 4.2
patchlevel 3.  Got a "current buffer has no process" when tried from
Windows Native GNU Emacs 22.2.1 but I am sure that's something I need
to correct at my end.

A couple of questions (or feature requests):

1. How can I save the image?  I think the question how to use a
   different device like "png" and have charts saved on disk.

2. How to use custom labels?

3. How can I pick and choose which columns to plot like in gnuplot's
   plot command?

4. How can I cycle through all tables and plot them all?

Thanks for writing and sharing this.

Regards,
-- Manish

Re: org-plot (generating graphs from org-mode)

[Sebastian Rose]

Hi Eric,

It's beautiful!

...but in emacs 23 only the first table works.

C-M-g on the other tables gives me the following message:

'Not in table data field'


emacs 23, CVS last week, Debian testing.


But, hey, this is wonderful!


  - Sebastian


Eric Schulte wrote:
> Hi,
> 
> org-plot enables generation of graphs from data stored in org tables
> (maybe eventually elsewhere in org files).
> 
> It is available on github at
> http://github.com/eschulte/org-plot/tree/master or with
> 
>   git clone git://github.com/eschulte/org-plot.git
> 
> Included with org-plot.el is an example.org file which should quickly
> show off it's features, options and abilities.
> 
> While this is not completed, it is functional, and I figure better to
> get early feedback then wait until it's all polished.  Please give it
> a look and let me know what you think.
> 
> Thanks -- Eric
> 
> 
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
> 

Re: org-plot (generating graphs from org-mode)

[Sebastian Rose]

Hi again,


still emacs 23:


On the table with data:

   The first try works. Then I have to pres C-M-g twice to get
   the plot window.


On the empty tables:

   with (setq debug-on-error t) this is in *Messages*:


(:plot-type grid :with lines :ind 0 :title "org mode" :map nil) [3 times]
Mark set [14 times]
Mapping tables: 100%
Mark set [2 times]
Mapping tables: done
signal: Wrong number of arguments: (lambda (el) (message (format "%S" 
el)) el), 2



The buffer is modified after C-M-g and emacs asks if I want to save it.


  - Sebastian

Re: org-plot (generating graphs from org-mode)

[Eric Schulte]

On Tuesday, August 26, at 14:36, Sebastian Rose wrote:
 > Hi again,
 > 
 > 
 > still emacs 23:

on Debian (Xubuntu) I'm using
- cvs emacs23 (from a couple of weeks ago)
- gnuplot 4.2 patchlevel 2
- gnuplot.el 0.6.0

 > On the table with data:
 > 
 >    The first try works. Then I have to pres C-M-g twice to get
 >    the plot window.

I've noticed this as well, but for me it only happens when doing 3d or
grid plots.  I think the gnuplot process may be changing state, I may
just start restarting this process before every graph.

 > On the empty tables:
 > 
 >    with (setq debug-on-error t) this is in *Messages*:

How have I never seen `debug-on-error' before! thanks

 > (:plot-type grid :with lines :ind 0 :title "org mode" :map nil) [3 times]
 > Mark set [14 times]
 > Mapping tables: 100%
 > Mark set [2 times]
 > Mapping tables: done
 > signal: Wrong number of arguments: (lambda (el) (message (format "%S" 
 > el)) el), 2
 > 
 > 
 > 
 > The buffer is modified after C-M-g and emacs asks if I want to save it.

That is certainly something to figure out.

I've added all these issues to my TODO for the project, and will
hopefully start resolving them soon, I'll update here.

Thanks for the testing/debugging -- Eric

 >   - Sebastian
 > 
 > 
 > 
 > 
 > _______________________________________________
 > Emacs-orgmode mailing list
 > Remember: use `Reply All' to send replies to the list.
 > Emacs-orgmode@gnu.org
 > http://lists.gnu.org/mailman/listinfo/emacs-orgmode

-- 
schulte

Re: org-plot (generating graphs from org-mode)

[Eric Schulte]

On Tuesday, August 26, at 15:32, Manish wrote:
 >   On Tue, Aug 26, 2008 at 7:23 AM, Eric Schulte wrote:
 >   > Hi,
 >   >
 >   > org-plot enables generation of graphs from data stored in org tables
 >   > (maybe eventually elsewhere in org files).
 >   >
 >   > It is available on github at
 >   > http://github.com/eschulte/org-plot/tree/master or with
 >   >
 >   >  git clone git://github.com/eschulte/org-plot.git
 >   >
 >   > Included with org-plot.el is an example.org file which should quickly
 >   > show off it's features, options and abilities.
 >   >
 >   > While this is not completed, it is functional, and I figure better to
 >   > get early feedback then wait until it's all polished.  Please give it
 >   > a look and let me know what you think.
 > 
 > Awesome!  This has wonderful possibilities for my work.  I just wish I
 > had this a couple of months ago.  But I hope I am going to have an
 > opportunity to use this again soon.

Great to hear, hopefully this will be more robust by then :)

 > Works great on GNU Emacs 22.1.1 with Org 6.06b and gnuplot 4.2
 > patchlevel 3.  Got a "current buffer has no process" when tried from
 > Windows Native GNU Emacs 22.2.1 but I am sure that's something I need
 > to correct at my end.

I'll take a look and see if there are any unix-only commands being
used, I have no access to windows machines so I may not be able to
reproduce.

 > A couple of questions (or feature requests):
 > 
 > 1. How can I save the image?  I think the question how to use a
 >    different device like "png" and have charts saved on disk.

you can save the image by adding a

 file:"path-to-image.jpg"

option to the #+PLOT: line above the graph, which should set gnuplot's
`output' and `term' options appropriately, if you want to do fancy
things you can set these values manually by adding something like the
following to the #+PLOT: line

 set:"terminal png transparent nocrop enhanced font arial 8"
 set:"output 'my-output-file.png'"

 > 2. How to use custom labels?

there is an option intended for this purpose labels:(col1 col2) which
should have this effect, however I will probably need to change the
way the regexp parses options first, I'll put this on my TODO list and
let you know when it's working.

 > 3. How can I pick and choose which columns to plot like in gnuplot's
 >    plot command?

you select the column to serve as the dependent variable (x-axis)
using the dep: command with the number of the column as the sole
argument (dep:1 to use the first column).  To specify which cols
should be plotted use the inds: options passing a list of column
numbers, like inds:(2 3)

 > 4. How can I cycle through all tables and plot them all?

I have no idea what the best method would be for cycling through
tables, I agree this would be a useful feature (picturing all graphs
being re-generated upon export to latex/html), however I don't know of
any commands for cycling through tables.  I've been thinking of
defining a [[plot:]] type link which may make this easier.  Also, I
may be able to copy an answer from the exporting code.  If anyone has
any ideas I'd love to hear them.

 > Thanks for writing and sharing this.

Thank you, I'm happy it may be useful. -- Eric

 > Regards,
 > -- Manish

-- 
schulte

Re: Emacs-orgmode Digest, Vol 30, Issue 39

[Stephan Schmitt]

Eric Schulte wrote:
> On Tuesday, August 26, at 14:36, Sebastian Rose wrote:
>  > The buffer is modified after C-M-g and emacs asks if I want to save it.
> 
> That is certainly something to figure out.
> 

This happens already if you press tab with the cursor 
located inside a table, so this is probably an org issue.

Eric, did you know that Carsten provided a function called 
org-table-to-lisp in org-6.06b?  I think with this you can 
refactor your code to be more modular and thus better 
maintainable and extensible.

Regards,
	Stephan

Re: Emacs-orgmode Digest, Vol 30, Issue 39

[Eric Schulte]

On Tuesday, August 26, at 19:57, Stephan Schmitt wrote:
 > Eric Schulte wrote:
 > > On Tuesday, August 26, at 14:36, Sebastian Rose wrote:
 > >  > The buffer is modified after C-M-g and emacs asks if I want to save it.
 > > 
 > > That is certainly something to figure out.
 > > 
 > 
 > This happens already if you press tab with the cursor 
 > located inside a table, so this is probably an org issue.

I had been using tab `org-cycle' in some cases to position the cursor
on selected columns, so that is probably the cause.

 > Eric, did you know that Carsten provided a function called 
 > org-table-to-lisp in org-6.06b?  I think with this you can 
 > refactor your code to be more modular and thus better 
 > maintainable and extensible.

I was not aware of that function.  It should certainly help, both in
cleaning up the code, and hopefully removing the reliance on
`org-cycle'.  Using `org-table-to-lisp' should also eliminate the 'Not
in table data field' error that Sebastian mentioned.  Thanks -- Eric

 > Regards,
 > 	Stephan

-- 
schulte

org-plot (generating graphs from org-mode)

[Eric Schulte]

On Tuesday, August 26, at 11:08, Eric Schulte wrote:
 > On Tuesday, August 26, at 19:57, Stephan Schmitt wrote:
 >  > Eric, did you know that Carsten provided a function called 
 >  > org-table-to-lisp in org-6.06b?  I think with this you can 
 >  > refactor your code to be more modular and thus better 
 >  > maintainable and extensible.

I have switched over to using `org-table-to-lisp' for all table
manipulation, and (at least in my environment), it seems to have made
the graphing functions faster, and much more reliable (no longer do I
have to run a function twice to generate a graph).

I believe this should fix most of the bugs mentioned previously.

Also, I have finished support for 3d graphing.  The updated version is
up on my git repo at

http://github.com/eschulte/org-plot/tree/master

or

git clone git://github.com/eschulte/org-plot.git

Of course if anyone has any comments, problems, or suggestions please
let me know.

Thanks -- Eric

-- 
schulte

Re: org-plot (generating graphs from org-mode)

[Sebastian Rose]

Hi Eric,


Great! This version fixes ALL the bugs I mentioned in my previous
email on emacs 23 !!!

I.e.

   - It always renders the plot after the first C-M-g
   - All three tables work

And it's much faster indeed!


Thank's, it's fantastic :-)


  - Sebastian


Eric Schulte wrote:
> On Tuesday, August 26, at 11:08, Eric Schulte wrote:
>  > On Tuesday, August 26, at 19:57, Stephan Schmitt wrote:
>  >  > Eric, did you know that Carsten provided a function called 
>  >  > org-table-to-lisp in org-6.06b?  I think with this you can 
>  >  > refactor your code to be more modular and thus better 
>  >  > maintainable and extensible.
> 
> I have switched over to using `org-table-to-lisp' for all table
> manipulation, and (at least in my environment), it seems to have made
> the graphing functions faster, and much more reliable (no longer do I
> have to run a function twice to generate a graph).
> 
> I believe this should fix most of the bugs mentioned previously.
> 
> Also, I have finished support for 3d graphing.  The updated version is
> up on my git repo at
> 
> http://github.com/eschulte/org-plot/tree/master
> 
> or
> 
> git clone git://github.com/eschulte/org-plot.git
> 
> Of course if anyone has any comments, problems, or suggestions please
> let me know.
> 
> Thanks -- Eric
> 

Re: org-plot (generating graphs from org-mode)

[Sebastian Rose]

Hi again,


Forgot to say: buffer-modified-p problem is gone too!!!


Woweeee!

Eric Schulte wrote:
> On Tuesday, August 26, at 11:08, Eric Schulte wrote:
>  > On Tuesday, August 26, at 19:57, Stephan Schmitt wrote:
>  >  > Eric, did you know that Carsten provided a function called 
>  >  > org-table-to-lisp in org-6.06b?  I think with this you can 
>  >  > refactor your code to be more modular and thus better 
>  >  > maintainable and extensible.
> 
> I have switched over to using `org-table-to-lisp' for all table
> manipulation, and (at least in my environment), it seems to have made
> the graphing functions faster, and much more reliable (no longer do I
> have to run a function twice to generate a graph).
> 
> I believe this should fix most of the bugs mentioned previously.
> 
> Also, I have finished support for 3d graphing.  The updated version is
> up on my git repo at
> 
> http://github.com/eschulte/org-plot/tree/master
> 
> or
> 
> git clone git://github.com/eschulte/org-plot.git
> 
> Of course if anyone has any comments, problems, or suggestions please
> let me know.
> 
> Thanks -- Eric
> 

Re: org-plot (generating graphs from org-mode)

[Eric Schulte]

On Wednesday, August 27, at 11:30, Sebastian Rose wrote:
 > Hi Eric,
 > 
 > 
 > Great! This version fixes ALL the bugs I mentioned in my previous
 > email on emacs 23 !!!
 > 

Wonderful!! thanks for testing this out.

-- 
schulte

Re: org-plot (generating graphs from org-mode)

[Samuel Wales]

Good stuff.

Possibilities:

1.  For a selected column, it can be nice to draw a histogram.  Long
ago, I developed the following formula for choosing a bin size.  Seems
to work.

        5*(max-min)/n

You can use something like bin*round*($1 * (1.0/bin)) to bin.  Note
that cl-extra.el's (round*) might be more consistent and
mathematically vetted than (round) (though you might not be allowed to
use cl-extra.el).

I haven't looked at gnuplot in a decade, so maybe it already has this.

2.  It might be nice to show descriptive statistics (I would show
mean, median, geometric mean, standard deviation, min, max, range)
somewhere while showing the plot, if you don't already do this.

Re: org-plot (generating graphs from org-mode)

[Eric Schulte]

On Wednesday, August 27, at 08:27, Samuel Wales wrote:
 > Good stuff.
 > 
 > Possibilities:
 > 
 > 1.  For a selected column, it can be nice to draw a histogram.  

There is a 'with histograms' option in gnuplot.  I made a small change
to org-plot.el so that this option now works as expected, for an
example of plotting a histogram (of one or more columns) using
org-plot see the example.org file in my git repo.

 > 2.  It might be nice to show descriptive statistics (I would show
 > mean, median, geometric mean, standard deviation, min, max, range)
 > somewhere while showing the plot, if you don't already do this.

That does seem like it would be useful (maybe accessible through
another optional argument), I've added it to the project's TODO file.

Thanks -- Eric

Re: org-plot (generating graphs from org-mode)

[William Henney]

Hi Eric

On Tue, Aug 26, 2008 at 6:09 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
> Of course if anyone has any comments, problems, or suggestions please
> let me know.

Org-plot is great! Thank you very much for writing this. I just found
a couple of small problems, as detailed below.

Cheers

Will

** Bugs in org-plot
*** [2008-08-28 Thu] Finding the table to plot
    =org-plot/goto-nearest-table= searches backward first, which means
    that if point is located in an options line for a following table,
    then it will find the previous table instead. This is usually
    undesired.

*** [2008-08-28 Thu] Columns with multi-word text
    Consider the following table:

#+PLOT: title:"Citas" ind:0 deps:(3) type:2d with:histograms set:"yrange [0:]"
    | Sede      | Max cites | H-index |
    |-----------+-----------+---------|
    | Chile     |    257.72 |   21.39 |
    | Leeds     |    165.77 |   19.68 |
    | São Paolo |     71.00 |   11.50 |
    | Stockholm |    134.19 |   14.33 |
    | Morelia   |    257.56 |   17.67 |

    In theory, this should plot the third column. However, the data
    point for São Paolo comes from the second column
    instead. Presumably this is because of the embedded space in "Sõo
    Paolo". If I replace it with a unicode non-breaking space,
    "São Paolo", then it works correctly.


-- 

 Dr William Henney, Centro de Radioastronomía y Astrofísica,
 Universidad Nacional Autónoma de México, Campus Morelia

Re: org-plot (generating graphs from org-mode)

[Eric Schulte]

Thanks for the feedback, much appreciated.  A new version of org-plot is
up on the git repo http://github.com/eschulte/org-plot/tree/master

detailed comments below...

"William Henney" <whenney@gmail.com> writes:
> *** [2008-08-28 Thu] Finding the table to plot
>     =org-plot/goto-nearest-table= searches backward first, which means
>     that if point is located in an options line for a following table,
>     then it will find the previous table instead. This is usually
>     undesired.

I changed this so it no longer looks backwards

> *** [2008-08-28 Thu] Columns with multi-word text
>     Consider the following table:
>
> #+PLOT: title:"Citas" ind:0 deps:(3) type:2d with:histograms set:"yrange [0:]"
>     | Sede      | Max cites | H-index |
>     |-----------+-----------+---------|
>     | Chile     |    257.72 |   21.39 |
>     | Leeds     |    165.77 |   19.68 |
>     | São Paolo |     71.00 |   11.50 |
>     | Stockholm |    134.19 |   14.33 |
>     | Morelia   |    257.56 |   17.67 |
>
>     In theory, this should plot the third column. However, the data
>     point for São Paolo comes from the second column
>     instead. Presumably this is because of the embedded space in "Sõo
>     Paolo". If I replace it with a unicode non-breaking space,
>     "São Paolo", then it works correctly.

you're right, I am now quoting non-text columns on export which takes
care of the problem you mentioned here.  Also, org-plot is now using to
gnuplot `xticlabel' command to use the ind columns values as labels in
the case that they are text.  (for an example plot your table from above
with the ind:1 option).  If you don't mind, I would like to add the
table you sent above to the example.org file in org-plot to demonstrate
this feature.

Thanks -- Eric

Re: org-plot (generating graphs from org-mode)

[Eric Schulte]

Thanks for the feedback, much appreciated.  A new version of org-plot is
up on the git repo http://github.com/eschulte/org-plot/tree/master

detailed comments below...

"William Henney" <whenney@gmail.com> writes:
> *** [2008-08-28 Thu] Finding the table to plot
>     =org-plot/goto-nearest-table= searches backward first, which means
>     that if point is located in an options line for a following table,
>     then it will find the previous table instead. This is usually
>     undesired.

I changed this so it no longer looks backwards

> *** [2008-08-28 Thu] Columns with multi-word text
>     Consider the following table:
>
> #+PLOT: title:"Citas" ind:0 deps:(3) type:2d with:histograms set:"yrange [0:]"
>     | Sede      | Max cites | H-index |
>     |-----------+-----------+---------|
>     | Chile     |    257.72 |   21.39 |
>     | Leeds     |    165.77 |   19.68 |
>     | São Paolo |     71.00 |   11.50 |
>     | Stockholm |    134.19 |   14.33 |
>     | Morelia   |    257.56 |   17.67 |
>
>     In theory, this should plot the third column. However, the data
>     point for São Paolo comes from the second column
>     instead. Presumably this is because of the embedded space in "Sõo
>     Paolo". If I replace it with a unicode non-breaking space,
>     "São Paolo", then it works correctly.

you're right, I am now quoting non-text columns on export which takes
care of the problem you mentioned here.  Also, org-plot is now using to
gnuplot `xticlabel' command to use the `ind' column's values as labels
in the case that they are text.  (to demonstrate, plot your table from
above with the ind:1 option).  If you don't mind, I would like to add
the table you sent above to the example.org file in org-plot to
demonstrate this feature.

Thanks -- Eric

Re: org-plot (generating graphs from org-mode)

[Carsten Dominik]

Hi Eric, you have done an awesome job.  I am very excited to try this  
out.

- Carsten

On Aug 26, 2008, at 5:10 PM, Eric Schulte wrote:
>> 4. How can I cycle through all tables and plot them all?
>
> I have no idea what the best method would be for cycling through
> tables, I agree this would be a useful feature (picturing all graphs
> being re-generated upon export to latex/html), however I don't know of
> any commands for cycling through tables.  I've been thinking of
> defining a [[plot:]] type link which may make this easier.  Also, I
> may be able to copy an answer from the exporting code.  If anyone has
> any ideas I'd love to hear them.


To call a function at each table, you can use org-table-map-tables,  
defined in org.el.

- Carsten

Re: org-plot (generating graphs from org-mode)

[Seweryn Kokot]

Hi Eric,

I just want to ask if it is necessary to redefine 'debug' function in
org-plot.el? Because when I put the following lines into .emacs file:

(require 'org-plot)
(add-hook 'org-mode-hook
		  (lambda ()
			(local-set-key (kbd "\M-\C-g")
						   'org-plot/gnuplot)))

whenever I invoke an error, instead of getting *Backtrace* buffer I
get a strange message in the echo line.

-- 
regards,
Seweryn Kokot

Re: Re: org-plot (generating graphs from org-mode)

[Eric Schulte]

oh, not at all,

I was using it for convenience during my own debugging, and didn't
realize I was overwriting an existing function.  Please feel free to
remove the `debug' definition from org-plot.el (I will remove it from
the version in my git repo right now).

Thanks -- Eric

Re: org-plot (generating graphs from org-mode)

[Thomas Baumann]

Awesome, thanks.

Thomas

Re: org-plot (generating graphs from org-mode)

[William Henney]

Hi Eric

On Sun, Aug 31, 2008 at 1:14 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>
> Thanks for the feedback, much appreciated.  A new version of org-plot is
> up on the git repo http://github.com/eschulte/org-plot/tree/master
>

I can confirm that the new version fixes both the problems that I
reported. Furthermore, the new option to use a column of text labels
for the x-axis is wonderful. Thanks!

> If you don't mind, I would like to add the
> table you sent above to the example.org file in org-plot to demonstrate
> this feature.
>

Yes, by all means, go ahead.

I've run into a couple more minor problems that might be worth looking
into if you have the time.

**** Writing plot to file
     Two issues here:
     1. The suffix of the output file must exactly match the name of a
        gnuplot terminal type. Thus file:"foo.jpeg" works, but
        file:"foo.jpg" doesn't. Similarly, one needs to use
        ".postscript" instead of ".ps".
     2. Some terminals seem to work, but some don't (there is no error
        message and the ouput file is created, but it is empty):
	- Works :: jpeg, png, svg, postscript, mp
	- Fails :: pdf (OK, this seems to be the only one - weird)
**** Non-ascii characters in strings
     These tend not to work. However, the results vary according to
     the output terminal in use, so I suspect that this may be due to
     bugs in gnuplot. My version is
:	G N U P L O T:
:	Version 4.2 patchlevel 3
:	last modified Mar 2008
:	System: Darwin 9.4.0
     One strange thing is that, even if the .org file is utf8, the
     graphics file seems to get written as latin-1.

Cheers

Will

-- 

 Dr William Henney, Centro de Radioastronomía y Astrofísica,
 Universidad Nacional Autónoma de México, Campus Morelia