using gnuplot's "splot" and "every" commands on org-mode table data

using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Hello everyone,

I'd be grateful if someone would offer me advice on using gnuplot's
"splot" and "every" commands when plotting data from a table within
org-mode.

As far as I can tell, these gnuplot commands do not work properly
because org-mode exports empty fields in tables as "" and gnuplot's
"every" and splot commands expect the data file to be formatted as a
datablock with blank lines marking the boundaries between datablocks.
(For the definition of a datablock, type "help glossary" at the
gnuplot prompt.)

I'm using org-mode version 8.0.2 and emacs version 24.2.1 on a
Fedora-18 system


To illustrate my point, consider a blocked datafile called "block.dat"
containing the following:

  1       1       2
  1       2       5
  1       3       10

  2       1       5
  2       2       8
  2       3       13

  3       1       10
  3       2       13
  3       3       18

For this file the gnuplot command

   #+begin_src gnuplot :var d="block.dat" :results silent
     plot "$d" u 2:3 ev :::0::0, "" u 2:3 ev :::1::1, "" u 2:3 ev :::2::2
   #+end_src

shows three separate lines of different colours as gnuplot
recognises the datafile as blocked data.

Also, the following command produces a surface plot

   #+begin_src gnuplot :var d="block.dat" :results silent
     splot "$d" u 1:2:3
   #+end_src

However, if I put the same data in the table below

   #+tblname: data
   | 1 | 1 |  2 |
   | 1 | 2 |  5 |
   | 1 | 3 | 10 |
   |   |   |    |
   | 2 | 1 |  5 |
   | 2 | 2 |  8 |
   | 2 | 3 | 13 |
   |   |   |    |
   | 3 | 1 | 10 |
   | 3 | 2 | 13 |
   | 3 | 3 | 18 |

and use the following plot command

   #+begin_src gnuplot :var d=data :results silent
     plot "$d" u 2:3 ev :::0::0, "" u 2:3 ev :::1::1, "" u 2:3 ev :::2::2
   #+end_src

the result is a plot of a single line of the same colour as gnuplot
joins all of the points in the data file.  This seems to be because
org-mode exports the table as

  "x"     "y"     "z"
  1       1       2
  1       2       5
  1       3       10
  ""      ""      ""
  2       1       5
  2       2       8
  2       3       13
  ""      ""      ""
  3       1       10
  3       2       13
  3       3       18

and gnuplot does not recognise this as a blocked data file because it
contains no blank lines.

The same problem occurs for

   #+begin_src gnuplot :var d=data :results silent
     splot "$d" u 1:2:3
   #+end_src

which does not produce a gridded surface plot.

Thanks for your help,

Paul

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 2974 bytes --]

Paul Stansell <paulstansell@gmail.com> writes:

> Hello everyone,
>
> I'd be grateful if someone would offer me advice on using gnuplot's
> "splot" and "every" commands when plotting data from a table within
> org-mode.
>
> As far as I can tell, these gnuplot commands do not work properly
> because org-mode exports empty fields in tables as "" and gnuplot's
> "every" and splot commands expect the data file to be formatted as a
> datablock with blank lines marking the boundaries between datablocks.
> (For the definition of a datablock, type "help glossary" at the
> gnuplot prompt.)
>
> I'm using org-mode version 8.0.2 and emacs version 24.2.1 on a
> Fedora-18 system
>
>
> To illustrate my point, consider a blocked datafile called "block.dat"
> containing the following:
>
>   1       1       2
>   1       2       5
>   1       3       10
>
>   2       1       5
>   2       2       8
>   2       3       13
>
>   3       1       10
>   3       2       13
>   3       3       18
>
> For this file the gnuplot command
>
>    #+begin_src gnuplot :var d="block.dat" :results silent
>      plot "$d" u 2:3 ev :::0::0, "" u 2:3 ev :::1::1, "" u 2:3 ev :::2::2
>    #+end_src
>
> shows three separate lines of different colours as gnuplot
> recognises the datafile as blocked data.
>
> Also, the following command produces a surface plot
>
>    #+begin_src gnuplot :var d="block.dat" :results silent
>      splot "$d" u 1:2:3
>    #+end_src
>
> However, if I put the same data in the table below
>
>    #+tblname: data
>    | 1 | 1 |  2 |
>    | 1 | 2 |  5 |
>    | 1 | 3 | 10 |
>    |   |   |    |
>    | 2 | 1 |  5 |
>    | 2 | 2 |  8 |
>    | 2 | 3 | 13 |
>    |   |   |    |
>    | 3 | 1 | 10 |
>    | 3 | 2 | 13 |
>    | 3 | 3 | 18 |
>
> and use the following plot command
>
>    #+begin_src gnuplot :var d=data :results silent
>      plot "$d" u 2:3 ev :::0::0, "" u 2:3 ev :::1::1, "" u 2:3 ev :::2::2
>    #+end_src
>
> the result is a plot of a single line of the same colour as gnuplot
> joins all of the points in the data file.  This seems to be because
> org-mode exports the table as
>
>   "x"     "y"     "z"
>   1       1       2
>   1       2       5
>   1       3       10
>   ""      ""      ""
>   2       1       5
>   2       2       8
>   2       3       13
>   ""      ""      ""
>   3       1       10
>   3       2       13
>   3       3       18
>
> and gnuplot does not recognise this as a blocked data file because it
> contains no blank lines.
>
> The same problem occurs for
>
>    #+begin_src gnuplot :var d=data :results silent
>      splot "$d" u 1:2:3
>    #+end_src
>
> which does not produce a gridded surface plot.
>
> Thanks for your help,
>

Hi Paul,

While I can't claim to fully follow your gnuplot examples, i would
recommend using an intervening shell code block to parse the Org-mode
table data into something that gnuplot will ingest.

If I understand your use case correctly, then something like the
following should work.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: example.org --]
[-- Type: text/x-org, Size: 399 bytes --]

#+name: data
| 1 | 1 |  2 |
| 1 | 2 |  5 |
| 1 | 3 | 10 |
|   |   |    |
| 2 | 1 |  5 |
| 2 | 2 |  8 |
| 2 | 3 | 13 |
|   |   |    |
| 3 | 1 | 10 |
| 3 | 2 | 13 |
| 3 | 3 | 18 |

#+name: clean
#+begin_src sh :var data=data :results file :file /tmp/data.gnuplot
  echo "$data" > /tmp/data.gnuplot
#+end_src

#+begin_src gnuplot :var data=clean(data) :results silent
  splot "$data" u 1:2:3
#+end_src

[-- Attachment #3: Type: text/plain, Size: 225 bytes --]


If you really wanted to be fancy, gnuplot will let you specify shell
transformations as part of the plotting command which would allow you to
forego the intermediate code block.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Achim Gratz]

Eric Schulte writes:
> If you really wanted to be fancy, gnuplot will let you specify shell
> transformations as part of the plotting command which would allow you to
> forego the intermediate code block.

As a side-note, if we'd drop the convention that the first separator
defines the heading of the table and introduced a proper heading
separator, then the following would be a table that Babel languages
could interpret more easily:

| a | b | c  |
|~~~+~~~+~~~~|
| 1 | 1 |  2 |
| 1 | 2 |  5 |
| 1 | 3 | 10 |
|---+---+----|
| 2 | 1 |  5 |
| 2 | 2 |  8 |
| 2 | 3 | 13 |
|---+---+----|
| 3 | 1 | 10 |
| 3 | 2 | 13 |
| 3 | 3 | 18 |


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric S Fraga]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte writes:
>> If you really wanted to be fancy, gnuplot will let you specify shell
>> transformations as part of the plotting command which would allow you to
>> forego the intermediate code block.
>
> As a side-note, if we'd drop the convention that the first separator
> defines the heading of the table and introduced a proper heading
> separator, then the following would be a table that Babel languages
> could interpret more easily:
>
> | a | b | c  |
> |~~~+~~~+~~~~|
> | 1 | 1 |  2 |
> | 1 | 2 |  5 |
> | 1 | 3 | 10 |
> |---+---+----|
> | 2 | 1 |  5 |
> | 2 | 2 |  8 |
> | 2 | 3 | 13 |
> |---+---+----|
> | 3 | 1 | 10 |
> | 3 | 2 | 13 |
> | 3 | 3 | 18 |
>
>
> Regards,
> Achim.

I would like to second this and actually possibly generalise the idea a
little further.

One of the problems with org tables, at the moment, is that the line
separators have two distinct functions:

1. they are used for presentation, both within org mode and for export
2. they provide key points for cell references in calculations

The two functions often coincide but not always.  For instance, the
following table is a reduced extract from a recent paper I wrote:

#+begin_src org
* Example table with multiple lines

  |   | Strategy | Average performance |
  |---+----------+---------------------|
  | / |          |                8.69 |
  | / |          |                9.72 |
  | / |          |                9.03 |
  |---+----------+---------------------|
  |   |        1 |                9.15 |
  |---+----------+---------------------|
  | / |          |                9.15 |
  | / |          |                7.60 |
  | / |          |                7.46 |
  |---+----------+---------------------|
  |   |        2 |                8.07 |
  |---+----------+---------------------|
  #+TBLFM: @5$3=vmean(@-I..@-II);%.2f::@9$3=vmean(@-I..@-II);%.2f
#+end_src

The (text) export consists of a three line table with lines separating
every line and one at the bottom as well:

   Strategy  Average performance 
  -------------------------------
          1                 9.15 
  -------------------------------
          2                 8.07 
  -------------------------------

Interesting aside: the export has only three lines whereas the original
table had 5 so the exporter is doing some intelligent work here!

Nevertheless, what I would like as the output would be

   Strategy  Average performance 
  -------------------------------
          1                 9.15 
          2                 8.07 

Unfortunately, because of the use of line separators to make
calculations easier to define, the exported table has more lines than I
would want.

At present, there is no mechanism for selective deletion of these line
separators upon export (that I know of: I would be happy to be corrected
on this!).  For the paper I submitted last week for publication, the
only post-org editing I had to do was delete a number \hline
specifications in the exported LaTeX file.  Not a major problem,
obviously, but it would be nice to not have to do even that!

Maybe there is a need for three different line separators: ~ for headers
(to be used as Achim indicates above but possibly helpful for exporters
as well), = for lines that should appear in exported output and - for
those that will not appear.  Header lines would typically export as
well.  All three would be used for formula definitions (i.e. @I
references) but would solely differ in how they are processed by the
exporters.

For upwards compatibility, should that be desired, the meaning of = and
- in the previous paragraph could be exchanged but would not be as
visually appealing (in my opinion).

Anyway, I will stop ruminating here and will get back to work...

Thanks,
eric
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_8.0.2-68-g40635b

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Achim Gratz]

Eric S Fraga writes:
> At present, there is no mechanism for selective deletion of these line
> separators upon export (that I know of: I would be happy to be corrected
> on this!).  For the paper I submitted last week for publication, the
> only post-org editing I had to do was delete a number \hline
> specifications in the exported LaTeX file.  Not a major problem,
> obviously, but it would be nice to not have to do even that!

Seems to me that one of the variants below would do what you wanted:

--8<---------------cut here---------------start------------->8---
* Example table with multiple lines

** Data :noexport:
  #+NAME: raw-table
  |   | Strategy | Average performance |
  |---+----------+---------------------|
  | / |          |                8.69 |
  | / |          |                9.72 |
  | / |          |                9.03 |
  |---+----------+---------------------|
  |   |        1 |                9.15 |
  |---+----------+---------------------|
  | / |          |                9.15 |
  | / |          |                7.60 |
  | / |          |                7.46 |
  |---+----------+---------------------|
  |   |        2 |                8.07 |
  |---+----------+---------------------|
  #+TBLFM: @5$3=vmean(@-I..@-II);%.2f::@9$3=vmean(@-I..@-II);%.2f

** Results

#+BEGIN_SRC elisp :var table=raw-table :colnames yes :exports results
table
#+END_SRC

#+RESULTS:
|   | Strategy | Average performance |
|---+----------+---------------------|
| / |          |                8.69 |
| / |          |                9.72 |
| / |          |                9.03 |
|   |        1 |                9.15 |
| / |          |                9.15 |
| / |          |                 7.6 |
| / |          |                7.46 |
|   |        2 |                8.07 |

#+BEGIN_SRC elisp :var table=raw-table :hlines yes :exports results
  (delq nil
        (let (hl)
          (mapcar (lambda (tl)
                    (if (listp tl)
                        (unless (equal (car tl) "/")
                          (cdr tl))
                      (unless hl
                        (setq hl t)
                        tl)))
                      table)))
#+END_SRC

#+RESULTS:
| Strategy | Average performance |
|----------+---------------------|
|        1 |                9.15 |
|        2 |                8.07 |

--8<---------------cut here---------------end--------------->8---


> Maybe there is a need for three different line separators: ~ for headers
> (to be used as Achim indicates above but possibly helpful for exporters
> as well), = for lines that should appear in exported output and - for
> those that will not appear.  Header lines would typically export as
> well.  All three would be used for formula definitions (i.e. @I
> references) but would solely differ in how they are processed by the
> exporters.

We should keep "---" as lines to be exported and perhaps use ":::" for
those that are just introducing convenience for table calculations as
"===" looks far too heavy to me for that purpose.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric S Fraga]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric S Fraga writes:
>> At present, there is no mechanism for selective deletion of these line
>> separators upon export (that I know of: I would be happy to be corrected
>
>> on this!).  For the paper I submitted last week for publication, the
>> only post-org editing I had to do was delete a number \hline
>> specifications in the exported LaTeX file.  Not a major problem,
>> obviously, but it would be nice to not have to do even that!
>
> Seems to me that one of the variants below would do what you wanted:

Yes, thanks for these!  There are other solutions using table references
as well.  But they are all clumsy, in my opinion.

[...]

> We should keep "---" as lines to be exported and perhaps use ":::" for
> those that are just introducing convenience for table calculations as
> "===" looks far too heavy to me for that purpose.

Sure, I can see how === may be too heavy.
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_8.0.2-101-gce5988

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Achim Gratz]

Eric S Fraga writes:
>> We should keep "---" as lines to be exported and perhaps use ":::" for
>> those that are just introducing convenience for table calculations as
>> "===" looks far too heavy to me for that purpose.
>
> Sure, I can see how === may be too heavy.

If you want to play around with this a bit, pull from my tableheadings
branch on http://repo.or.cz/w/org-mode/org-tableheadings.git (it lists
the git or http URLS to pull from on the page)

I've implemented "calculation separators" that register as hlines for
table formulas, but are ignored when row groups are constructed for
export.  Your example table would then look like this:

--8<---------------cut here---------------start------------->8---
  |   | Strategy | Average performance |
  |---+----------+---------------------|
  | / |          |                8.69 |
  | / |          |                9.72 |
  | / |          |                9.03 |
  |:::+::::::::::+:::::::::::::::::::::|
  |   |        1 |                9.15 |
  |:::+::::::::::+:::::::::::::::::::::|
  | / |          |                9.15 |
  | / |          |                7.60 |
  | / |          |                7.46 |
  |:::+::::::::::+:::::::::::::::::::::|
  |   |        2 |                8.07 |
  |---+----------+---------------------|
  #+TBLFM: @5$3=vmean(@-I..@-II);%.2f::@9$3=vmean(@-I..@-II);%.2f
--8<---------------cut here---------------end--------------->8---

This should then export the way you wanted without any fiddling.  Let me
know if you find any bugs (the test suite is clean, but test coverage
with tables is sketchy).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Rick Frankel]

On Sat, May 11, 2013 at 12:39:55PM +0200, Achim Gratz wrote:
> Eric S Fraga writes:
> >> We should keep "---" as lines to be exported and perhaps use ":::" for
> >> those that are just introducing convenience for table calculations as
> >> "===" looks far too heavy to me for that purpose.
> >
> > Sure, I can see how === may be too heavy.

My 2 cents: great idea, but i find : too visually heavly also. How
about periods (.), or just the first character of the hline (i like
the periods). Here's a comparison:


  |   | Strategy | Average performance |
  |---+----------+---------------------|
  | / |          |                8.69 |
  | / |          |                9.72 |
  | / |          |                9.03 |
  |...+..........+.....................|
  |   |        1 |                9.15 |
  |:--+----------+---------------------|
  | / |          |                9.15 |
  | / |          |                7.60 |
  | / |          |                7.46 |
  |:::+::::::::::+:::::::::::::::::::::|
  |   |        2 |                8.07 |
  |---+----------+---------------------|

rick

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric S Fraga]

Achim Gratz <Stromeko@nexgo.de> writes:

[...]

> I've implemented "calculation separators" that register as hlines for
> table formulas, but are ignored when row groups are constructed for
> export.  Your example table would then look like this:

Achim,

just to apologise for not getting back to you earlier on this.  I've
been swamped with work.  I'll try to have a test of your table updates
this week!

thanks,
eric
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_8.0.3-144-gf1b99a

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Eric,

What you suggest is indeed a good solution, so thank you very much for that!

I think it would still be nice, however, if someday the blank lines in
the org table would be exported as blank lines in the temporary text
file instead of as "" (which seems to be the root of the problem).

Kind regards,

Paul

> Hi Paul,
>
> While I can't claim to fully follow your gnuplot examples, i would
> recommend using an intervening shell code block to parse the Org-mode
> table data into something that gnuplot will ingest.
>
> If I understand your use case correctly, then something like the
> following should work.
>
>
>
> If you really wanted to be fancy, gnuplot will let you specify shell
> transformations as part of the plotting command which would allow you to
> forego the intermediate code block.
>
> --
> Eric Schulte
> http://cs.unm.edu/~eschulte
>

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

If the removal of ""'s will *always* be preferable for gnuplot, then we
should be able make that conversion the default behavior.  However,
before such a conversion we'd need to ensure that the new format is more
generally useful than the existing export format.

Both `org-babel-gnuplot-quote-tsv-field' and `org-babel-gnuplot-table-to-data'
could fairly easily be customized to support different table formatting.

Paul Stansell <paulstansell@gmail.com> writes:

> Eric,
>
> What you suggest is indeed a good solution, so thank you very much for that!
>
> I think it would still be nice, however, if someday the blank lines in
> the org table would be exported as blank lines in the temporary text
> file instead of as "" (which seems to be the root of the problem).
>
> Kind regards,
>
> Paul
>
>> Hi Paul,
>>
>> While I can't claim to fully follow your gnuplot examples, i would
>> recommend using an intervening shell code block to parse the Org-mode
>> table data into something that gnuplot will ingest.
>>
>> If I understand your use case correctly, then something like the
>> following should work.
>>
>>
>>
>> If you really wanted to be fancy, gnuplot will let you specify shell
>> transformations as part of the plotting command which would allow you to
>> forego the intermediate code block.
>>
>> --
>> Eric Schulte
>> http://cs.unm.edu/~eschulte
>>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Hi Eric,

My view is that the removal of '""'s will always be preferable for
gnuplot as it uses what it calls datablocks which are delimited by
blank lines (see, for example,
http://gnuplot.sourceforge.net/docs_4.2/node121.html and
http://gnuplot.sourceforge.net/docs_4.2/node61.html).

Paul

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 441 bytes --]

Paul Stansell <paulstansell@gmail.com> writes:

> Hi Eric,
>
> My view is that the removal of '""'s will always be preferable for
> gnuplot as it uses what it calls datablocks which are delimited by
> blank lines (see, for example,
> http://gnuplot.sourceforge.net/docs_4.2/node121.html and
> http://gnuplot.sourceforge.net/docs_4.2/node61.html).
>
> Paul

Hi Paul,

I think examples like the following may be more common than data
blocks.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnuplot.org --]
[-- Type: text/x-org, Size: 202 bytes --]

#+name: data
| 1 |  2 |
| 2 |  4 |
| 3 |  8 |
|   | 16 |
|   |    |
| 6 | 16 |
| 7 |  8 |
|   |  4 |
| 9 |  2 |

#+begin_src gnuplot :var data=data :results silent
  plot data u 0:1, '' u 0:2
#+end_src

[-- Attachment #3: Type: text/plain, Size: 131 bytes --]


Replacing ""s with blank space in the following breaks the resulting
plot.

Cheers,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Hi Eric,

In your last example are you illustrating the case of a data file with
missing data?  Replacing ""s with blank space in your example doesn't
really break the plot, it's just that gnuplot interprets the first
column of data as

1
2
3
16

6
7
4
9

and the second as

2
4
8


16
8
4
2

Gnuplot has a specific setting for handling missing values.  It's
common in gnuplot to use '?' to signify a missing value (type "help
missing" at the gnuplot prompt).  If, in your last example, you
replace the missing values with '?' (without the quotes) and then type
in gnuplot

  set datafile missing "?"

and then plot the data with

  plot "data.txt" u 0:1, '' u 0:2

you'll see that gnuplot handles it properly.  Putting '?' in the
blanks cells of the org mode table in your last attachment and
plotting from within org mode gives a temporary data file containing
'"?"', which gnuplot also handles correctly as it must recognise "?"
(with the quotes) as being a missing value.

But org mode's behaviour of inserting "" in blank cells (instead of
actual blanks) does break the use of "every" in gnuplot which is
required by splot when plotting surfaces from data files (unless the
data files are in "matrix" format).  Also, inserting "" breaks the use
of gnuplot's "index" which expects two blank lines to separate data
sets in a single file, but I've never used this facility.

I noticed that org mode writes table values of 'nan' and 'NaN' without
the double quotes, but, for example, 'MaM' is written with double
quotes, (as ''"MaM"') in the temporary data file.

Also, I think it's generally better to remove the double quotes from
around all strings exported by org mode for plotting by gnuplot as
they're generally not needed (exceptions are strings which contain
spaces which can be quoted so they are treated as a single entity by
gnuplot).  For example, there is a gnuplot demonstration script called
datastrings.dem which plots data from a file called ctg-y2.dat.  This
data file has unquoted month names in the first column that are
plotted as labels on the x-axis.

Paul

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

Paul Stansell <paulstansell@gmail.com> writes:

> Hi Eric,
>
> In your last example are you illustrating the case of a data file with
> missing data?

Yes.

> Replacing ""s with blank space in your example doesn't really break
> the plot, it's just that gnuplot interprets the first column of data
> as
>
> 1
> 2
> 3
> 16
>
> 6
> 7
> 4
> 9
>
> and the second as
>
> 2
> 4
> 8
>
>
> 16
> 8
> 4
> 2
>

I would consider this breaking the plot, in that gnuplot no longer lines
up the two columns of data.

>
> Gnuplot has a specific setting for handling missing values.  It's
> common in gnuplot to use '?' to signify a missing value (type "help
> missing" at the gnuplot prompt).  If, in your last example, you
> replace the missing values with '?' (without the quotes) and then type
> in gnuplot
>
>   set datafile missing "?"
>
> and then plot the data with
>
>   plot "data.txt" u 0:1, '' u 0:2
>
> you'll see that gnuplot handles it properly.  Putting '?' in the
> blanks cells of the org mode table in your last attachment and
> plotting from within org mode gives a temporary data file containing
> '"?"', which gnuplot also handles correctly as it must recognise "?"
> (with the quotes) as being a missing value.
>
> But org mode's behaviour of inserting "" in blank cells (instead of
> actual blanks) does break the use of "every" in gnuplot which is
> required by splot when plotting surfaces from data files (unless the
> data files are in "matrix" format).  Also, inserting "" breaks the use
> of gnuplot's "index" which expects two blank lines to separate data
> sets in a single file, but I've never used this facility.
>
> I noticed that org mode writes table values of 'nan' and 'NaN' without
> the double quotes, but, for example, 'MaM' is written with double
> quotes, (as ''"MaM"') in the temporary data file.
>

I didn't know that '?' was the standard gnuplot missing value.  In that
case I think the `org-babel-gnuplot-table-to-data' should be updated to
replace missing values with '?' by default.  If we find a common need to
specialize this value then we can add a gnuplot header argument which
can specify missing values for particular code blocks.

>
> Also, I think it's generally better to remove the double quotes from
> around all strings exported by org mode for plotting by gnuplot as
> they're generally not needed (exceptions are strings which contain
> spaces which can be quoted so they are treated as a single entity by
> gnuplot).  For example, there is a gnuplot demonstration script called
> datastrings.dem which plots data from a file called ctg-y2.dat.  This
> data file has unquoted month names in the first column that are
> plotted as labels on the x-axis.
>

I agree with your reasoning here.  This and the above should be fairly
straightforward changes to the `org-babel-gnuplot-table-to-data'
function.  If I find any free time I'll make these changes.  Although I
certainly wouldn't mind if someone else gets to them first.

Best regards (and thanks for the clear explanations),

>
> Paul

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

>
> I would consider this breaking the plot, in that gnuplot no longer lines
> up the two columns of data.
>

My fault, gnuplot does indeed still line up the data as appropriate when
missing values are simply replaced with blank space rather than "".

I've just pushed up changes to gnuplot code blocks which add a :missing
header argument which may be used to specify a missing value, and which
replace missing values with blank space (rather than "") when the
missing header argument is not supplied.

The attached org-mode file demonstrates both.  Press C-c C-v v on each
code block, and then open up the data file to see the effects of the
header argument.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnuplot.org --]
[-- Type: text/x-org, Size: 357 bytes --]

#+Property: :results silent

#+name: data
| 1 | 2 |
| 2 |   |
| 3 | 4 |
| 4 | 5 |

#+begin_src gnuplot :var data=data
  set key outside
  set xrange [-1:4]
  set yrange [0:6]
  plot data u 0:1, '' u 0:2
#+end_src

#+begin_src gnuplot :var data=data :missing "?"
  set key outside
  set xrange [-1:4]
  set yrange [0:6]
  plot data u 0:1, '' u 0:2
#+end_src

[-- Attachment #3: Type: text/plain, Size: 78 bytes --]


Thanks for pushing this through
-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]

Hi Eric,

Thanks for your work on this.  I've tried your latest changes and I
think they're a big improvement.

I've attached another org-mode file for you to try.  It illustrates
the use of double blank lines in the first table (called block_data)
to inform gnuplot that there're two separate data sets in the single
data file.  The first code block (gnuplot_1) plots these two data sets
as red and green surfaces.  It's really nice that gnuplot can now be
used in this way from within org-mode.

One slight drawback, however, is that there still doesn't seem to be a
way to specify missing values in the org table in the way they're
frequently used by gnuplot.  An illustration of what I mean by this is
given in the table block_data_missing in the attached org file.
You'll see that I've inserted two '?' characters for missing z-values.
 If you execute the second and third code blocks (gnuplot_2 and
gnuplot_3) you'll see that neither give the desired result.

If the block_data_missing table were exported exactly as it is except
for the removal of the '|' characters and "set datafile missing '?'"
were specified in the gnuplot script this would enable gnuplot to plot
the red and green surfaces with the missing values.  To see this try
executing the last code block (gnuplot_4) which plots the data which
is first "cleaned" by the shell command you suggested in a previous
post.  This would also remove the need to have the :missing header
argument.

Kind regards,

Paul

[-- Attachment #2: gnuplot2.org --]
[-- Type: application/octet-stream, Size: 1690 bytes --]

#+Property: :results silent

#+name: block_data
| 1 | 1 |  2 |
| 1 | 2 |  5 |
| 1 | 3 | 10 |
|   |   |    |
| 2 | 1 |  5 |
| 2 | 2 |  8 |
| 2 | 3 | 13 |
|   |   |    |
| 3 | 1 | 10 |
| 3 | 2 | 12 |
| 3 | 3 | 18 |
|   |   |    |
|   |   |    |
| 1 | 1 | 12 |
| 1 | 2 | 15 |
| 1 | 3 | 20 |
|   |   |    |
| 2 | 1 | 15 |
| 2 | 2 | 18 |
| 2 | 3 | 23 |
|   |   |    |
| 3 | 1 | 20 |
| 3 | 2 | 23 |
| 3 | 3 | 28 |

#+name: gnuplot_1
#+begin_src gnuplot :var d=block_data
  set ticslevel 0 # This removes the non-zero offset in z.
  splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
#+end_src

#+name: block_data_missing
| 1 | 1 |  2 |
| 1 | 2 |  5 |
| 1 | 3 | 10 |
|   |   |    |
| 2 | 1 |  5 |
| 2 | 2 | 10 |
| 2 | 3 | 13 |
|   |   |    |
| 3 | 1 | 10 |
| 3 | 2 |  ? |
| 3 | 3 | 18 |
|   |   |    |
|   |   |    |
| 1 | 1 | 12 |
| 1 | 2 | 15 |
| 1 | 3 | 20 |
|   |   |    |
| 2 | 1 | 15 |
| 2 | 2 | 18 |
| 2 | 3 | 23 |
|   |   |    |
| 3 | 1 |  ? |
| 3 | 2 | 23 |
| 3 | 3 | 28 |

#+name: gnuplot_2
#+begin_src gnuplot :var d=block_data_missing :missing "?"
  set ticslevel 0
  splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
#+end_src

#+name: gnuplot_3
#+begin_src gnuplot :var d=block_data_missing
  set ticslevel 0
  splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
#+end_src

#+name: clean
#+begin_src sh :var data=block_data_missing :results file :file /tmp/data.gnuplot
  echo "$data" > /tmp/data.gnuplot
#+end_src

#+name: gnuplot_4
#+begin_src gnuplot :var d=clean(block_data_missing)
  set ticslevel 0
  set datafile missing '?'
  splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
#+end_src

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

Paul Stansell <paulstansell@gmail.com> writes:

> Hi Eric,
>
> Thanks for your work on this.  I've tried your latest changes and I
> think they're a big improvement.
>
> I've attached another org-mode file for you to try.  It illustrates
> the use of double blank lines in the first table (called block_data)
> to inform gnuplot that there're two separate data sets in the single
> data file.  The first code block (gnuplot_1) plots these two data sets
> as red and green surfaces.  It's really nice that gnuplot can now be
> used in this way from within org-mode.
>
> One slight drawback, however, is that there still doesn't seem to be a
> way to specify missing values in the org table in the way they're
> frequently used by gnuplot.  An illustration of what I mean by this is
> given in the table block_data_missing in the attached org file.
> You'll see that I've inserted two '?' characters for missing z-values.
>  If you execute the second and third code blocks (gnuplot_2 and
> gnuplot_3) you'll see that neither give the desired result.
>

I see, I've change the `org-babel-gnuplot-quote-tsv-field' function so
that it will only wrap a value in double quotes if that value actually
needs these wraps (e.g., contains a quote or a space).

With this change your example 3 now works as expected (nice looking
plots by the way).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnuplot3.org --]
[-- Type: text/x-org, Size: 555 bytes --]

#+name: block_data_missing
| 1 | 1 |  2 |
| 1 | 2 |  5 |
| 1 | 3 | 10 |
|   |   |    |
| 2 | 1 |  5 |
| 2 | 2 | 10 |
| 2 | 3 | 13 |
|   |   |    |
| 3 | 1 | 10 |
| 3 | 2 |  ? |
| 3 | 3 | 18 |
|   |   |    |
|   |   |    |
| 1 | 1 | 12 |
| 1 | 2 | 15 |
| 1 | 3 | 20 |
|   |   |    |
| 2 | 1 | 15 |
| 2 | 2 | 18 |
| 2 | 3 | 23 |
|   |   |    |
| 3 | 1 |  ? |
| 3 | 2 | 23 |
| 3 | 3 | 28 |

#+name: gnuplot_3
#+begin_src gnuplot :var d=block_data_missing
  set ticslevel 0
  splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
#+end_src

[-- Attachment #3: Type: text/plain, Size: 643 bytes --]


Are there any use cases which are still not addressed?

>
> If the block_data_missing table were exported exactly as it is except
> for the removal of the '|' characters and "set datafile missing '?'"
> were specified in the gnuplot script this would enable gnuplot to plot
> the red and green surfaces with the missing values.  To see this try
> executing the last code block (gnuplot_4) which plots the data which
> is first "cleaned" by the shell command you suggested in a previous
> post.  This would also remove the need to have the :missing header
> argument.
>
> Kind regards,
>
> Paul
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Hi Eric,

That's seems perfect now; I can't see any other issues.  Thanks!

Kind regards,

Paul

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Dear Eric,

Recently you made some nice changes to the way gnuplot plots surface
data from org tables.  Your changes are present on
git://orgmode.org/org-mode.git, but I switched to using ELPA recently
and I've just noticed that your changes are not yet present on
http://orgmode.org/elpa/.  Do you know if it's just because the latter
lags the former by 3 months or more?

As far as I can tell, I'm using the following versions:

  From git: org-version: 8.0.7 (release_8.0.7-406-g4b7490)
  From ELPA: Org-mode version 8.0.7 (8.0.7-7-g9e5104-elpa (org-20130826)

Kind regards,

Paul

On 17 May 2013 22:33, Paul Stansell <paulstansell@gmail.com> wrote:
> Hi Eric,
>
> That's seems perfect now; I can't see any other issues.  Thanks!
>
> Kind regards,
>
> Paul

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Achim Gratz]

Paul Stansell writes:
> I've just noticed that your changes are not yet present on
> http://orgmode.org/elpa/.  Do you know if it's just because the latter
> lags the former by 3 months or more?

ELPA doesn't lag anything like three months.  However, it is tracking
maint, not master.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]

Hi Eric,

If I change your example org file a little by adding row and column
names to the table as in the attached file I get the following
behaviour that I don't understand:  if I execute C-c on the shell
function the file /tmp/data.gnuplot has the format that gnuplot can
plot;  however, if I execute C-c on the gnuplot function the form of
/tmp/data.gnuplot is mangled with "" and () so it can no longer be
plotted by the gnuplot command.

Can you shed any light on this?

Thanks,

Paul

[-- Attachment #2: splot.org --]
[-- Type: application/octet-stream, Size: 580 bytes --]

#+name: data
|----+---+---+----|
|    | x | y |  z |
|----+---+---+----|
| r1 | 1 | 1 |  2 |
| r2 | 1 | 2 |  5 |
| r3 | 1 | 3 | 10 |
|    |   |   |    |
| r4 | 2 | 1 |  5 |
| r5 | 2 | 2 |  8 |
| r6 | 2 | 3 | 13 |
|    |   |   |    |
| r7 | 3 | 1 | 10 |
| r8 | 3 | 2 | 13 |
| r9 | 3 | 3 | 18 |
|----+---+---+----|


#+name: clean
#+begin_src sh :var data=data :results file :file /tmp/data.gnuplot
  echo "$data" > /tmp/data.gnuplot
#+end_src

#+RESULTS: clean
[[file:/tmp/data.gnuplot]]

#+begin_src gnuplot :var data=clean(data) :results silent
  splot "$data" u 1:2:3
#+end_src

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

Paul Stansell <paulstansell@gmail.com> writes:

> Hi Eric,
>
> If I change your example org file a little by adding row and column
> names to the table as in the attached file I get the following
> behaviour that I don't understand:  if I execute C-c on the shell
> function the file /tmp/data.gnuplot has the format that gnuplot can
> plot;  however, if I execute C-c on the gnuplot function the form of
> /tmp/data.gnuplot is mangled with "" and () so it can no longer be
> plotted by the gnuplot command.
>
> Can you shed any light on this?
>

Very weird,

This does look like a bug to me.  Can you isolate "where" the value of
data in your example is first assigned the wrong value?

Thanks for reporting.  I will save your example for when I next have
development time.

Best,

>
> Thanks,
>
> Paul
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Hi Eric,

It seems to be the first "hline" that causes the problem.  For
example, if the table data in splot.org is replaced by

#+name: data
|    | x |
|----+---|
| r1 | 1 |

it gets exported correctly as

	x
r1	1

by both the shell and gnuplot commands.

However,  if the data is replace by

#+name: data
|----+---|
|    | x |
| r1 | 1 |

it still gets exported correctly by the shell command but the gnuplot
command exports it as

hline
("" "x")
("r1" 1)


Thanks for your help with this,

Paul

>
> Very weird,
>
> This does look like a bug to me.  Can you isolate "where" the value of
> data in your example is first assigned the wrong value?
>
> Thanks for reporting.  I will save your example for when I next have
> development time.
>
> Best,

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Achim Gratz]

[-- Attachment #1: Type: text/plain, Size: 799 bytes --]

Eric Schulte writes:
> This does look like a bug to me.  Can you isolate "where" the value of
> data in your example is first assigned the wrong value?

The error lies in how ob-sh tries to determine if it needs to process a
table or a vector and then forgets to check for 'hline, thus ending up
in the vector branch with the table data.

> Thanks for reporting.  I will save your example for when I next have
> development time.

Here's a suggestion for a quick fix.  I've also implemented :hlines
processing for ob-sh (it was ignoring that header arg unconditionally)
and added an experimental header arg :hline-string to control how hlines
get actually exported.  BTW, trailing blank lines in the output get
swallowed when re-importing to Org, I'm not sure if that is supposed to
happen or not.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-sh-detect-tables-correctly-add-hlines-processing-.patch --]
[-- Type: text/x-patch, Size: 2589 bytes --]

From 06056792c96c4c5996db5dd13e857658e6d4a573 Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Sun, 12 May 2013 20:36:51 +0200
Subject: [PATCH] ob-sh: detect tables correctly, add :hlines processing and
 :hline-string header arg

* lisp/ob-sh.el (org-babel-variable-assignments:sh): Check for
  ":hlines yes" and use header arg :hlines-string if
  defined (default to "hline") and add this to the call of
  `org-babel-sh-var-to-sh'.
  (org-babel-sh-var-to-sh, org-babel-sh-var-to-string): Add additional
  optional string argument `hline' and use it for the :hline parameter
  in the call to `orgtbl-to-generic'.

Thanks to Paul Stansell for pointing out the error.
---
 lisp/ob-sh.el | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/lisp/ob-sh.el b/lisp/ob-sh.el
index f11b799..72f3c19 100644
--- a/lisp/ob-sh.el
+++ b/lisp/ob-sh.el
@@ -88,26 +88,31 @@ (defun org-babel-load-session:sh (session body params)
 
 (defun org-babel-variable-assignments:sh (params)
   "Return list of shell statements assigning the block's variables."
-  (let ((sep (cdr (assoc :separator params))))
+  (let ((sep (cdr (assoc :separator params)))
+	(hline (when (string= "yes" (cdr (assoc :hlines params)))
+		 (or (cdr (assoc :hline-string params))
+		     "hline"))))
     (mapcar
      (lambda (pair)
        (format "%s=%s"
 	       (car pair)
-	       (org-babel-sh-var-to-sh (cdr pair) sep)))
+	       (org-babel-sh-var-to-sh (cdr pair) sep hline)))
      (mapcar #'cdr (org-babel-get-header params :var)))))
 
-(defun org-babel-sh-var-to-sh (var &optional sep)
+(defun org-babel-sh-var-to-sh (var &optional sep hlines)
   "Convert an elisp value to a shell variable.
 Convert an elisp var into a string of shell commands specifying a
 var of the same value."
-  (format org-babel-sh-var-quote-fmt (org-babel-sh-var-to-string var sep)))
+  (format org-babel-sh-var-quote-fmt (org-babel-sh-var-to-string var sep hline)))
 
-(defun org-babel-sh-var-to-string (var &optional sep)
+(defun org-babel-sh-var-to-string (var &optional sep hline)
   "Convert an elisp value to a string."
   (let ((echo-var (lambda (v) (if (stringp v) v (format "%S" v)))))
     (cond
-     ((and (listp var) (listp (car var)))
-      (orgtbl-to-generic var  (list :sep (or sep "\t") :fmt echo-var)))
+     ((and (listp var) (or (listp (car var)) 'hline))
+      (orgtbl-to-generic var  (list :sep (or sep "\t")
+				    :fmt echo-var
+				    :hline hline)))
      ((listp var)
       (mapconcat echo-var var "\n"))
      (t (funcall echo-var var)))))
-- 
1.8.2.2


[-- Attachment #3: Type: text/plain, Size: 375 bytes --]


Giving ob-gnuplot (and possible ob-awk, which also uses
orgtbl-to-generic) a similar treatment would allow to do this directly
without going through ob.sh of course.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte writes:
>> This does look like a bug to me.  Can you isolate "where" the value of
>> data in your example is first assigned the wrong value?
>
> The error lies in how ob-sh tries to determine if it needs to process a
> table or a vector and then forgets to check for 'hline, thus ending up
> in the vector branch with the table data.
>
>> Thanks for reporting.  I will save your example for when I next have
>> development time.
>
> Here's a suggestion for a quick fix.  I've also implemented :hlines
> processing for ob-sh (it was ignoring that header arg unconditionally)
> and added an experimental header arg :hline-string to control how hlines
> get actually exported.  BTW, trailing blank lines in the output get
> swallowed when re-importing to Org, I'm not sure if that is supposed to
> happen or not.
>

This looks great.  Please go ahead and apply this patch to ob-sh.

>
> Giving ob-gnuplot (and possible ob-awk, which also uses
> orgtbl-to-generic) a similar treatment would allow to do this directly
> without going through ob.sh of course.
>

That sounds like a good solution to me.  Generally whenever a solution
may be made in a generic support function instead of a language specific
file I would consider that a win.  If you have a good implementation
here I'd be very happy if you applied it.

Cheers,

>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Achim Gratz]

Eric Schulte writes:
> This looks great.  Please go ahead and apply this patch to ob-sh.

I've applied the bugfix part to maint and master.

I'm holding back on the :hline-string for now since I want to avoid
introducing yet another header arg for that.  Instead I'll investigate
whether this can be rolled into the :hline arg by allowing other values
than just "yes" and "no".

>> Giving ob-gnuplot (and possible ob-awk, which also uses
>> orgtbl-to-generic) a similar treatment would allow to do this directly
>> without going through ob.sh of course.
>
> That sounds like a good solution to me.  Generally whenever a solution
> may be made in a generic support function instead of a language specific
> file I would consider that a win.  If you have a good implementation
> here I'd be very happy if you applied it.

I'll look into it, but I'll first need to check again if using
orgtbl-to-generic is the right thing to use here.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Achim Gratz]

Eric Schulte writes:
> This looks great.  Please go ahead and apply this patch to ob-sh.

I just see that this was still sitting in my local branch… pushed to
master.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Achim Gratz]

Eric Schulte writes:
> While I can't claim to fully follow your gnuplot examples, i would
> recommend using an intervening shell code block to parse the Org-mode
> table data into something that gnuplot will ingest.

Implementing something just for gnuplot seems to miss the mark.  I can
see utility for "records" in more Babel languages than just gnuplot.
This could be implemented using the already existing facilities by
starting a new record for each row group.

#+name: data
| 1 | 1 |  2 |
| 1 | 2 |  5 |
| 1 | 3 | 10 |
|---+---+----|
| 2 | 1 |  5 |
| 2 | 2 |  8 |
| 2 | 3 | 13 |
|---+---+----|
| 3 | 1 | 10 |
| 3 | 2 | 13 |
| 3 | 3 | 18 |

For gnuplot the record separator happens to be a blank line, for other
languages we'd have different conventions (like adding a third dimension
to the variable that the table is transported in).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Hi Eric,

There appears to be a slight bug in the way gnuplot handles blocked
data with missing entries.

An example is provided in the org script at then end of this note.

In the code block gnuplot_1 the use of ':missing "?"' seems to cause
the exported data file to lack the blanks lines that are essential
for the plot.  The plot from this code block is incorrect.

In gnuplot_2 ':missing' is not used, but instead 'set datafile missing
"?"' is set explicitly in the gnuplot code block.  The plot from this
code block is correct.

Surely gnuplot_1 should produce exactly the same plot as gnuplot_2.

Kind regards,

Paul


Exported data file does not include blank lines.
#+name: gnuplot_1
#+begin_src gnuplot :var d=block_data_missing :missing "?" :results silent
  set ticslevel 0
  splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
#+end_src

Exported data file includes blank lines.
#+name: gnuplot_2
#+begin_src gnuplot :var d=block_data_missing :results silent
  set datafile missing "?"
  set ticslevel 0
  splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
#+end_src

#+name: block_data_missing
| 1 | 1 |  2 |
| 1 | 2 |  5 |
| 1 | 3 | 10 |
|   |   |    |
| 2 | 1 |  5 |
| 2 | 2 | 10 |
| 2 | 3 | 13 |
|   |   |    |
| 3 | 1 | 10 |
| 3 | 2 |  ? |
| 3 | 3 | 18 |
|   |   |    |
|   |   |    |
| 1 | 1 | 12 |
| 1 | 2 | 15 |
| 1 | 3 | 20 |
|   |   |    |
| 2 | 1 | 15 |
| 2 | 2 | 18 |
| 2 | 3 | 23 |
|   |   |    |
| 3 | 1 |  ? |
| 3 | 2 | 23 |
| 3 | 3 | 28 |

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

Paul Stansell <paulstansell@gmail.com> writes:

> Hi Eric,
>
> There appears to be a slight bug in the way gnuplot handles blocked
> data with missing entries.
>
> An example is provided in the org script at then end of this note.
>
> In the code block gnuplot_1 the use of ':missing "?"' seems to cause
> the exported data file to lack the blanks lines that are essential
> for the plot.  The plot from this code block is incorrect.
>
> In gnuplot_2 ':missing' is not used, but instead 'set datafile missing
> "?"' is set explicitly in the gnuplot code block.  The plot from this
> code block is correct.
>
> Surely gnuplot_1 should produce exactly the same plot as gnuplot_2.
>

Hi Paul,

Currently the :missing header argument for gnuplot code blocks does two
things.  It adds the "set datafile missing..." line to the code block
body *and* it binds `*org-babel-gnuplot-missing*' to the value of the
:missing header argument when the datafile is written, so empty table
cells are replaced with the value of the :missing header argument.

The reason you're not getting a blank line is because it is replaced
with three "missing" values, in this case "? ? ?".

Do you think this behavior should be changed or split between two header
arguments?

Thanks,

>
> Kind regards,
>
> Paul
>
>
> Exported data file does not include blank lines.
> #+name: gnuplot_1
> #+begin_src gnuplot :var d=block_data_missing :missing "?" :results silent
>   set ticslevel 0
>   splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
> #+end_src
>
> Exported data file includes blank lines.
> #+name: gnuplot_2
> #+begin_src gnuplot :var d=block_data_missing :results silent
>   set datafile missing "?"
>   set ticslevel 0
>   splot d index 0 u 1:2:3 w lp pt 6 ps 5, d index 1 u 1:2:3 w lp pt 6 ps 5
> #+end_src
>
> #+name: block_data_missing
> | 1 | 1 |  2 |
> | 1 | 2 |  5 |
> | 1 | 3 | 10 |
> |   |   |    |
> | 2 | 1 |  5 |
> | 2 | 2 | 10 |
> | 2 | 3 | 13 |
> |   |   |    |
> | 3 | 1 | 10 |
> | 3 | 2 |  ? |
> | 3 | 3 | 18 |
> |   |   |    |
> |   |   |    |
> | 1 | 1 | 12 |
> | 1 | 2 | 15 |
> | 1 | 3 | 20 |
> |   |   |    |
> | 2 | 1 | 15 |
> | 2 | 2 | 18 |
> | 2 | 3 | 23 |
> |   |   |    |
> | 3 | 1 |  ? |
> | 3 | 2 | 23 |
> | 3 | 3 | 28 |
>
>
>
>
>

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

Hi Eric,

> Currently the :missing header argument for gnuplot code blocks does two
> things.  It adds the "set datafile missing..." line to the code block
> body *and* it binds `*org-babel-gnuplot-missing*' to the value of the
> :missing header argument when the datafile is written, so empty table
> cells are replaced with the value of the :missing header argument.

Thanks for the explanation.

> The reason you're not getting a blank line is because it is replaced
> with three "missing" values, in this case "? ? ?".
>
> Do you think this behavior should be changed or split between two header
> arguments?

I think the :missing header should not replace empty fields in an org
table by the missing character as blank lines have a special meaning
in gnuplot.  To fully use the versatility of gnuplot one needs blank
lines and missing values.  The "set datafile missing" command sets a
specific character string to be treated as a missing value.  The
character string can even be a number, eg. set datafile missing "-1".

It says under gnuplot's "help missing" that: "There is no default
character for `missing`, but in many cases any non-parsible string of
characters found where a numerical value is expected will be treated
as missing data."

My view is that for the purposes of exporting org tables for plotting
with gnuplot all blank entries in the org table (fields and whole
lines) should be exported as blanks, and all characters (letters or
numerical digits etc.) should be exported exactly as they appear in
the table (without any additional surrounding quotes being added).

Maybe I'm missing the point of the :missing header, but I find it
easier and clearer to include the datafile missing command in the
gnuplot code block.  I don't really see a need for the :missing
header.  Maybe the need stems from wanting to be able to use different
plotting packages to plot the same org table data, but it's not clear
to me.

Kind regards,

Paul

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Eric Schulte]

>
> Maybe I'm missing the point of the :missing header, but I find it
> easier and clearer to include the datafile missing command in the
> gnuplot code block.  I don't really see a need for the :missing
> header.  Maybe the need stems from wanting to be able to use different
> plotting packages to plot the same org table data, but it's not clear
> to me.
>

I agree that it is easier and more clear to include "set datafile
missing..." in the body of a gnuplot code block.  I *do* think that the
:missing header argument has value specifically as a way to replace
missing values in the original table data.

I've changed the behavior of the :missing header argument so that it
*only* replaces missing values in table data and does not add a "set
datafile missing..." line to the code block.

Cheers,

-- 
Eric Schulte
https://cs.unm.edu/~eschulte
PGP: 0x614CA05D

Re: using gnuplot's "splot" and "every" commands on org-mode table data

[Paul Stansell]

> I agree that it is easier and more clear to include "set datafile
> missing..." in the body of a gnuplot code block.  I *do* think that the
> :missing header argument has value specifically as a way to replace
> missing values in the original table data.
>
> I've changed the behavior of the :missing header argument so that it
> *only* replaces missing values in table data and does not add a "set
> datafile missing..." line to the code block.

If I understand you correctly I think that sounds like a good
solution.  As I understand it, your recent changes mean that if the
:missing header is not used the table data will be exported "as is",
ie, with no changes, replacements or additions to any of the
characters in the table cells; however, if the ':missing "?"' header
is used any empty table cells will be exported as ? (with no quotes)
and one needs to explicitly add

  set datafile missing "?"

to the code block.  In fact, it's not actually necessary to add this
line to the code block as gnuplot will treat the non-parsible "?"
string as missing data without the line.

Regards,

Paul