BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

[Robert Klein <roklein@roklein.de>]

[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]

Hi everyone,
 a bug report:


Emacs  : GNU Emacs 27.2 (build 1, x86_64-unknown-openbsd, GTK+ Version 2.24.33)
 of 2021-04-20
Package: Org mode version 9.4.6 (release_9.4.6-4-g093c94 @ /home/roklein/.emacs.d/org-mode/lisp/)


When trying to run a table with missing values with ob-gnuplot.el
there's a “:missing value” option.  ob-gnuplot is supposed to put the
“value” into the place of empty cell values before calling on gnuplot.
It doesn't do this currently.  Bi-secting suggests,
“9209aa3c9d456bcfdd9d8cd223367f7cfb6b3231 is the first bad commit”:

--  commit 9209aa3c9d456bcfdd9d8cd223367f7cfb6b3231
--  Author: Nicolas Goaziou <mail@nicolasgoaziou.fr>
--  --  Sun Aug 24 01:31:56 2014 +0200
--
--  org-table: Use "ox.el" internally for radio tables


By my reading:
- in ob-gnuplot.el:org-babel-gnuplot-process-vars :missing variable is
  read, used in
- ob-gnuplot.el:org-babel-gnuplot-quote-tsv-field, called from
- ob-gnuplot.el:org-babel-gnuplot-table-to-data
- used in org-table.el:orgtbl-to-generic (as (part of the) :fmt param)
- which calls org-table--to-generic-cell
- which calle org-table.el:org-table--generic-apply
- which I don't understand

my dotemacs and a sample org file are attached.

When running the code (e.g. C-c C-c on the #+Call: line), temporary
files are created in /tmp/babel-<random>/gnuplot<other-random>.

These files should have "?" (as choosen :missing).  In the current
version there is nothing, so gnuplot uses up the values in the following
columns and the resulting graph is wrong.

Also, I couldn't find this in the manual, but there's a thread “using
gnuplot's "splot" and "every" comma nds on org-mode table” on the
mailing list (2013/09/23 - 3013/09/25) about the :missing feature.

Thanks for any help.

Best regards
Robert


[-- Attachment #2: gp.org --]
[-- Type: application/octet-stream, Size: 8311 bytes --]

# -*- fill-column: 72; coding: utf-8-unix;  -*-

#+call: create_health_monitor_plot2()

#+RESULTS:
: nil

* blood pressure 2021

#+Name: bdq2_21
| Datum | sys. | dia. | pul. | sys.2 | dia.2 | pul.2 |    5ds |   5dd |   5ds2 |  5dd2 |
|-------+------+------+------+-------+-------+-------+--------+-------+--------+-------|
| 23.3. |      |      |      |   121 |    79 |    80 | 120.50 | 79.50 | 120.00 | 80.00 |
| 24.3. |  120 |   80 |   60 |   120 |    80 |    80 | 121.00 | 79.00 | 119.50 | 80.50 |
| 25.3. |  121 |   79 |   60 |   119 |    81 |    80 | 121.50 | 78.50 | 119.00 | 81.00 |
| 26.3. |  122 |   78 |   60 |   118 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.00 |
| 27.3. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.40 | 82.60 |
| 28.3. |  124 |   76 |   60 |   116 |    84 |    80 | 122.80 | 77.20 | 117.20 | 82.80 |
| 29.3. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.40 | 82.60 |
| 30.3. |  122 |   78 |   60 |   118 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.00 |
| 31.3. |  121 |   79 |   60 |   119 |    81 |    80 | 121.40 | 78.60 | 118.60 | 81.40 |
|  1.4. |  120 |   80 |   60 |   120 |    80 |    80 | 121.20 | 78.80 | 118.80 | 81.20 |
|  2.4. |  121 |   79 |   60 |   119 |    81 |    80 | 121.40 | 78.60 | 118.60 | 81.40 |
|  3.4. |  122 |   78 |   60 |   118 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.00 |
|  4.4. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.40 | 82.60 |
|  5.4. |  124 |   76 |   60 |   116 |    84 |    80 | 122.80 | 77.20 | 117.20 | 82.80 |
|  6.4. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.40 | 82.60 |
|  7.4. |  122 |   78 |   60 |   118 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.00 |
|  8.4. |  121 |   79 |   60 |   119 |    81 |    80 | 121.40 | 78.60 | 118.60 | 81.40 |
|  9.4. |  120 |   80 |   60 |   120 |    80 |    80 | 121.20 | 78.80 | 119.00 | 81.00 |
| 10.4. |  121 |   79 |   60 |   119 |    81 |    80 | 121.40 | 78.60 | 118.75 | 81.25 |
| 11.4. |  122 |   78 |   60 |       |       |       | 122.00 | 78.00 | 118.00 | 82.00 |
| 12.4. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.25 | 82.75 |
| 13.4. |  124 |   76 |   60 |   116 |    84 |    80 | 122.80 | 77.20 | 117.25 | 83.00 |
| 14.4. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.25 | 83.00 |
| 15.4. |  122 |   78 |   60 |   119 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.25 |
| 16.4. |  121 |   79 |   60 |       |       |       | 121.50 | 78.50 | 118.67 | 81.67 |
| 17.4. |  120 |   80 |   60 |   120 |    80 |    80 | 121.00 | 79.00 | 119.50 | 81.00 |
#+TBLFM: @>$9=if("@-2$2..@>$3" == "[]", string(""), vmean(@-2$3..@>$3));%.2f::@>>$9=if("@-2$3..@+1$3" == "[]", string(""), vmean(@-2$3..@+1$3));%.2f::@>$8=if("@-2$2..@>$2" == "[]", string(""), vmean(@-2$2..@>$2));%.2f::@>>$8=if("@-2$2..@+1$2" == "[]", string(""), vmean(@-2$2..@+1$2));%.2f::@>$10=if("@-2$5..@>$5" == "[]", string(""), vmean(@-2$5..@>$5));%.2f::@>>$10=if("@-2$5..@+1$5" == "[]", string(""), vmean(@-2$5..@+1$5));%.2f::@>$11=if("@-2$6..@>$6" == "[]", string(""), vmean(@-2$6..@>$6));%.2f::@>>$11=if("@-2$6..@+1$6" == "[]", string(""), vmean(@-2$6..@+1$6));%.2f::@2$8=if("@2$2..@+2$2" == "[]", string(""), vmean(@2$2..@+2$2));%.2f::@2$9=if("@2$3..@+2$3" == "[]", string(""), vmean(@2$3..@+2$3));%.2f::@2$10=if("@2$5..@+2$5" == "[]", string(""), vmean(@2$5..@+2$5));%.2f::@2$11=if("@2$6..@+2$6" == "[]", string(""), vmean(@2$6..@+2$6));%.2f::@3$8=if("@-1$2..@+2$2" == "[]", string(""), vmean(@-1$2..@+2$2));%.2f::@3$9=if("@-1$3..@+2$3" == "[]", string(""), vmean(@-1$3..@+2$3));%.2f::@3$10=if("@-1$5..@+2$5" == "[]", string(""), vmean(@-1$5..@+2$5));%.2f::@3$11=if("@-1$6..@+2$6" == "[]", string(""), vmean(@-1$6..@+2$6));%.2f::@4$8..@>>>$8=if("@-2$2..@+2$2" == "[]", string(""), vmean(@-2$2..@+2$2));%.2f::@4$9..@>>>$9=if("@-2$3..@+2$3" == "[]", string(""), vmean(@-2$3..@+2$3));%.2f::@4$10..@>>>$10=if("@-2$5..@+2$5" == "[]", string(""), vmean(@-2$5..@+2$5));%.2f::@4$11..@>>>$11=if("@-2$6..@+2$6" == "[]", string(""), vmean(@-2$6..@+2$6));%.2f




#+name: create_health_monitor_plot2
#+BEGIN_SRC gnuplot "," :missing "?" :var data=bdq2_21 :results silent
  set terminal png

  # get stats data
  stats data
  set terminal png enhanced giant size 1280,768

  set output 'q2-1.png'

  ## Define X Axis
  # globally set the time format for input data
  set timefmt "%d.%m."
  # X coordinates represent time values (here: date)
  set xdata time
  # set format of X axis labels to "%d."
  set xtics timedate format "%d.%m."
  # no minor tics on X Axis
  #unset mxtics
  set mxtics 14
  # don't extend x-axis beyond x values in data
  set autoscale noextend

  ## Define Y axis
  # Tics and microtics, so each integer y value gets a (micro)tic
  set ytics 10 out nomirror
  set mytics 10
  set yrange [60:200]
  set ylabel "blood pressure (mmHg)"
  # labeled tics also on the  right
  set y2tics 10 out
  set mytics 10

  ## Tics on both Axis outward
  set grid xtics ytics front

  # missing data
  set datafile missing "?"

  # set style fill  transparent solid 0.50 noborder
  # plot "~/mm.txt" u 1:3 axes x1y1 title "mg/dL"

  ## colors
  # darkred: 8b0000
  # firebrick: b22222
  # crimson: dc143c
  # coral: ff7f50
  # peachpuff: ffdab9
  # palegoldenrod: eee8aa
  # palegreen: 98fb98

  plot \
       400 axes x1y1 with filledcurves y2=180 fc "#b22222" notitle, \
       180 axes x1y1 with filledcurves y2=160 fc "#dc143c" notitle, \
       160 axes x1y1 with filledcurves y2=140 fc "#ff7f50" notitle, \
       140 axes x1y1 with filledcurves y2=130 fc "#ffdab9" notitle, \
       130 axes x1y1 with filledcurves y2=120 fc "#eee8aa" notitle, \
       120 axes x1y1 with filledcurves y2=105 fc "#98fb98" notitle, \
       105 axes x1y1 with filledcurves y2=100 fc "#dc143c" notitle, \
       \
       100 axes x1y1 with filledcurves y2=90  fc "#ff7f50" notitle, \
       90  axes x1y1 with filledcurves y2=85  fc "#ffdab9" notitle, \
       85  axes x1y1 with filledcurves y2=80  fc "#eee8aa" notitle, \
       80  axes x1y1 with filledcurves y2=65  fc "#98fb98" notitle, \
       60  axes x1y1 with filledcurves y2=65  fc "#dc143c" notitle, \
       data u 1:2 axes x1y1 with lines lc "dark-green" lw 2 title "syst. blood pressure (am)", \
       data every ::0::1                             u 1:8 axes x1y1 \
            with lines lc "gray" lw 1 notitle, \
       data every ::1::2                             u 1:8 axes x1y1  \
            with lines lc "gray60" lw 1 notitle, \
       data every ::2::STATS_records-3               u 1:8 axes x1y1 \
            with lines lc "gray30" lw 1 title "5-day moving syst.", \
       data every ::STATS_records-3::STATS_records-2 u 1:8 axes x1y1 \
            with lines lc "gray60" lw 1 notitle, \
       data every ::STATS_records-2::STATS_records-1 u 1:8 axes x1y1 \
            with lines lc "gray" lw 1 notitle, \
     \
       data u 1:3 axes x1y1 with lines lc "dark-blue" lw 2 title "diast. blood pressure (am)", \
       data every ::0::1                             u 1:9 axes x1y1 \
            with lines lc "gray" lw 1 notitle, \
       data every ::1::2                             u 1:9 axes x1y1  \
            with lines lc "gray60" lw 1 notitle, \
       data every ::2::STATS_records-3               u 1:9 axes x1y1 \
            with lines lc "gray30" lw 1 title "5-day moving diast.", \
       data every ::STATS_records-3::STATS_records-2 u 1:9 axes x1y1 \
            with lines lc "gray60" lw 1 notitle, \
       data every ::STATS_records-2::STATS_records-1 u 1:9 axes x1y1 \
            with lines lc "gray" lw 1 notitle, \
     \
       data u 1:($5) axes x1y1 with lines lc "green" lw 2 title "syst. blood pressure (pm)", \
     \
       data u 1:($6) axes x1y1 with lines lc "blue" lw 2 title "diast. blood pressure (pm)", \
     \

  #     data every ::0::1                             u 1:2 axes x1y1 \
  #          with lines lc "gray" lw 1 notitle, \
  #     data every ::1::2                             u 1:2 axes x1y1  \
  #          with lines lc "gray60" lw 1 notitle, \
  #     data every ::2::STATS_records-3               u 1:2 axes x1y1 \
  #          with lines lc "gray30" lw 1 title "5-day moving syst.", \

  quit
#+END_SRC

#+Results: create_health_monitor_plot2

[-- Attachment #3: dotemacs --]
[-- Type: application/octet-stream, Size: 1180 bytes --]

;; -*- lisp-mode:t; -*-
;; get rid of C-z
;;(unbind-key "C-z")
(global-unset-key (kbd "C-z"))
;; no startup message
(setq inhibit-startup-message t)

;; no toolbar
(if (not (equal system-configuration "aarch64-unknown-linux-android"))
    (tool-bar-mode -1))

;; I'm feeling religious about this:
(setq-default indent-tabs-mode nil)
;; changed my mind back.
;; (setq-default indent-tabs-mode t)

;;  ask y or n instead of yes or no
(fset 'yes-or-no-p 'y-or-n-p)

;; my ears hurt. Especially when wearing headphones
(setq visible-bell t)

;; color-theme
(setq-default custom-enabled-themes (quote (adwaita)))
(load-theme 'adwaita)

(setq load-path (cons "~/gp/org-mode/lisp" load-path))
(add-to-list 'auto-mode-alist '("\\.\\(org\\|org_archive\\|txt\\)$" . org-mode))
(add-hook 'org-mode-hook (lambda () (electric-indent-local-mode -1)))

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (gnuplot . t)))

(add-to-list 'load-path
             (expand-file-name "~/gp/gnuplot-0.8.0"))
  ;; autoload gnuplot-mode
  ;(autoload 'gnuplot-mode "gnuplot" "gnuplot major mode" t)
  ;(autoload 'gnuplot-make-buffer "gnuplot" "open a buffer in gnuplot-mode" t)