Re: searching for csv utilities

Re: searching for csv utilities

[Jude DaShiell]

| Date             |               Sys | Dia | Pul | Sugar |
|------------------+-------------------+-----+-----+-------|
| [2014-04-27 Sun] |               125 |  88 |  78 |    92 |
| [2014-04-28 Mon] |               102 |  88 |  86 |    92 |
| Averages:        | =$2=vmean(@<..@>) |     |     |       |
#+TBLFM: $2=$2=vmean(@<..@>)

This is a cut down version of my full record set.  Sometimes when I key 
formulas in I get ?ERROR back for a result after keying in c-c+c-c once 
I've completed the formula and hit tab.  If I do c-u+c-c+c-c that 
sometimes generated ?ERROR.  Other times I key in a formula and the cursor 
gets locked and I have to hit c-g to exit #+TBLFM: mode; I don't know 
what's actually happening when that situation arises since other than 
suddenly finding the cursor locked I can neither tell what state I'm in or 
if a few more keystrokes are needed or if I've generated an error 
situation.

  --

Re: searching for csv utilities

[Jonathan Leech-Pepin]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

Hello,

On 2 June 2015 at 07:44, Jude DaShiell <jdashiel@panix.com> wrote:

> | Date             |               Sys | Dia | Pul | Sugar |
> |------------------+-------------------+-----+-----+-------|
> | [2014-04-27 Sun] |               125 |  88 |  78 |    92 |
> | [2014-04-28 Mon] |               102 |  88 |  86 |    92 |
> | Averages:        | =$2=vmean(@<..@>) |     |     |       |
> #+TBLFM: $2=$2=vmean(@<..@>)
>

The formula in question is the culprit in this case (at least as stated
there).

: $2=$2=vmean(@<..@>)

Second column is equal to the second column which is equal to the mean of
all the values in the second column (including the header "Sys").

If you change the table as follows:

| Date             |   Sys | Dia | Pul | Sugar |
|------------------+-------+-----+-----+-------|
| [2014-04-27 Sun] |   125 |  88 |  78 |    92 |
| [2014-04-28 Mon] |   102 |  88 |  86 |    92 |
|------------------+-------+-----+-----+-------|
| Averages:        | 113.5 |  88 |  82 |    92 |
#+TBLFM: @>$2..@>$>=vmean(@I..@II)

All the values will properly compute.  If you want to avoid the second
HLINE above Averages: then change =@II= to =@>>= (penultimate row)

Regards,
Jon


> This is a cut down version of my full record set.  Sometimes when I key
> formulas in I get ?ERROR back for a result after keying in c-c+c-c once
> I've completed the formula and hit tab.  If I do c-u+c-c+c-c that sometimes
> generated ?ERROR.  Other times I key in a formula and the cursor gets
> locked and I have to hit c-g to exit #+TBLFM: mode; I don't know what's
> actually happening when that situation arises since other than suddenly
> finding the cursor locked I can neither tell what state I'm in or if a few
> more keystrokes are needed or if I've generated an error situation.
>
>  --
>
>
>

[-- Attachment #2: Type: text/html, Size: 2589 bytes --]

Re: searching for csv utilities

[Jude DaShiell]

Jonathan,

Your modified ecm in this case works.
Having applied the ecm to a larger file with seven additional 
calculation rows in it the averages are now displaying on the sums line.
I tried @>-7 at the start of the #+TBLFM: line and that generated about 
as many errors as someone's first efforts at a Cobol program.
I also put a printf format string into that TBLFM: line that rounds 
averages to two decimal places and also prints out right parens at the 
end of each result.  Is that an emacs-orgmode convention?  I think the 
2>$2 piece of the #+TBLFM: line tells emacs-orgmode to put that 
calculation's results on the last line of the file and in column 2.

Are these calculations elisp format?  If so, I didn't read the elisp 
section of the spreadsheet info for emacs-orgmode yet.

On Tue, 2 Jun 2015, Jonathan Leech-Pepin wrote:

> Date: Tue, 2 Jun 2015 08:04:20
> From: Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com>
> To: Jude DaShiell <jdashiel@panix.com>
> Cc: Org Mode Mailing List <emacs-orgmode@gnu.org>
> Subject: Re: [O] searching for csv utilities
> 
> Hello,
>
> On 2 June 2015 at 07:44, Jude DaShiell <jdashiel@panix.com> wrote:
>
>> | Date             |               Sys | Dia | Pul | Sugar |
>> |------------------+-------------------+-----+-----+-------|
>> | [2014-04-27 Sun] |               125 |  88 |  78 |    92 |
>> | [2014-04-28 Mon] |               102 |  88 |  86 |    92 |
>> | Averages:        | =$2=vmean(@<..@>) |     |     |       |
>> #+TBLFM: $2=$2=vmean(@<..@>)
>>
>
> The formula in question is the culprit in this case (at least as stated
> there).
>
> : $2=$2=vmean(@<..@>)
>
> Second column is equal to the second column which is equal to the mean of
> all the values in the second column (including the header "Sys").
>
> If you change the table as follows:
>
> | Date             |   Sys | Dia | Pul | Sugar |
> |------------------+-------+-----+-----+-------|
> | [2014-04-27 Sun] |   125 |  88 |  78 |    92 |
> | [2014-04-28 Mon] |   102 |  88 |  86 |    92 |
> |------------------+-------+-----+-----+-------|
> | Averages:        | 113.5 |  88 |  82 |    92 |
> #+TBLFM: @>$2..@>$>=vmean(@I..@II)
>
> All the values will properly compute.  If you want to avoid the second
> HLINE above Averages: then change =@II= to =@>>= (penultimate row)
>
> Regards,
> Jon
>
>
>> This is a cut down version of my full record set.  Sometimes when I key
>> formulas in I get ?ERROR back for a result after keying in c-c+c-c once
>> I've completed the formula and hit tab.  If I do c-u+c-c+c-c that sometimes
>> generated ?ERROR.  Other times I key in a formula and the cursor gets
>> locked and I have to hit c-g to exit #+TBLFM: mode; I don't know what's
>> actually happening when that situation arises since other than suddenly
>> finding the cursor locked I can neither tell what state I'm in or if a few
>> more keystrokes are needed or if I've generated an error situation.
>>
>>  --
>>
>>
>>
>

-- 

Re: searching for csv utilities

[Jude DaShiell]

This is a piece of a modified ecm that may show what's going on.

cut here.
|----------------------+---------+--------+--------+--------|
| Averages:            |         |        |        |        |
| Counts:              |         |        |        |        |
| Maximums:            |         |        |        |        |
| Medians:             |         |        |        |        |
| Minimums:            |         |        |        |        |
| Modes:               |         |        |        |        |
| Standard Deviations: |         |        |        |        |
| Sums:                | 108.69) | 70.45) | 66.62) | 92.93) |
|----------------------+---------+--------+--------+--------|
#+TBLFM: @>$2..@>$>=vmean(@I..@>>;%.2f)

I haven't even attempted the rest of the math since I have no way to 
predict where any of the results will land.
On Tue, 2 
Jun 2015, Jonathan Leech-Pepin wrote:

> Date: Tue, 2 Jun 2015 08:04:20
> From: Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com>
> To: Jude DaShiell <jdashiel@panix.com>
> Cc: Org Mode Mailing List <emacs-orgmode@gnu.org>
> Subject: Re: [O] searching for csv utilities
> 
> Hello,
>
> On 2 June 2015 at 07:44, Jude DaShiell <jdashiel@panix.com> wrote:
>
>> | Date             |               Sys | Dia | Pul | Sugar |
>> |------------------+-------------------+-----+-----+-------|
>> | [2014-04-27 Sun] |               125 |  88 |  78 |    92 |
>> | [2014-04-28 Mon] |               102 |  88 |  86 |    92 |
>> | Averages:        | =$2=vmean(@<..@>) |     |     |       |
>> #+TBLFM: $2=$2=vmean(@<..@>)
>>
>
> The formula in question is the culprit in this case (at least as stated
> there).
>
> : $2=$2=vmean(@<..@>)
>
> Second column is equal to the second column which is equal to the mean of
> all the values in the second column (including the header "Sys").
>
> If you change the table as follows:
>
> | Date             |   Sys | Dia | Pul | Sugar |
> |------------------+-------+-----+-----+-------|
> | [2014-04-27 Sun] |   125 |  88 |  78 |    92 |
> | [2014-04-28 Mon] |   102 |  88 |  86 |    92 |
> |------------------+-------+-----+-----+-------|
> | Averages:        | 113.5 |  88 |  82 |    92 |
> #+TBLFM: @>$2..@>$>=vmean(@I..@II)
>
> All the values will properly compute.  If you want to avoid the second
> HLINE above Averages: then change =@II= to =@>>= (penultimate row)
>
> Regards,
> Jon
>
>
>> This is a cut down version of my full record set.  Sometimes when I key
>> formulas in I get ?ERROR back for a result after keying in c-c+c-c once
>> I've completed the formula and hit tab.  If I do c-u+c-c+c-c that sometimes
>> generated ?ERROR.  Other times I key in a formula and the cursor gets
>> locked and I have to hit c-g to exit #+TBLFM: mode; I don't know what's
>> actually happening when that situation arises since other than suddenly
>> finding the cursor locked I can neither tell what state I'm in or if a few
>> more keystrokes are needed or if I've generated an error situation.
>>
>>  --
>>
>>
>>
>

-- 

Re: searching for csv utilities

[Jonathan Leech-Pepin]

[-- Attachment #1: Type: text/plain, Size: 5246 bytes --]

On 3 June 2015 at 12:07, Jude DaShiell <jdashiel@panix.com> wrote:

> This is a piece of a modified ecm that may show what's going on.
>
> cut here.
> |----------------------+---------+--------+--------+--------|
> | Averages:            |         |        |        |        |
> | Counts:              |         |        |        |        |
> | Maximums:            |         |        |        |        |
> | Medians:             |         |        |        |        |
> | Minimums:            |         |        |        |        |
> | Modes:               |         |        |        |        |
> | Standard Deviations: |         |        |        |        |
> | Sums:                | 108.69) | 70.45) | 66.62) | 92.93) |
> |----------------------+---------+--------+--------+--------|
> #+TBLFM: @>$2..@>$>=vmean(@I..@>>;%.2f)
>
> I haven't even attempted the rest of the math since I have no way to
> predict where any of the results will land.
>

@> means last line, @>> is second to last, @>>> third to last and so on.
So for 7th from the bottom it would be @>>>>>>>.

Re: PrintF specification
Everything after the =;= is considered part of the specification, so the
=)= used to close the vmean is actually part of the specification.
Changing that to =vmean();%.2f= will correct it.



For your sample ECM (plus original data and one sample line to actually
confirm median works) you would work with the following (Apologies for the
very long TBLFM line):

I was unable to find a built-in to determine the mode.  I've found sample
functions on Stack Overflow that would calculate it based on a list, but
I'm not familiar enough with Org-Table format on how to go from cell
references =@I..@II= to a list of values for the sake of manipulating them.

: | Date                 |    Sys |    Dia |    Pul |  Sugar |
: |----------------------+--------+--------+--------+--------|
: | [2014-04-27 Sun]     |    125 |     88 |     78 |     92 |
: | [2014-04-28 Mon]     |    102 |     88 |     86 |     92 |
: | [2014-04-29 Tue]     |    115 |     88 |     85 |     95 |
: |----------------------+--------+--------+--------+--------|
: | Averages:            | 114.00 |  88.00 |  83.00 |  93.00 |
: | Counts:              |   3.00 |   3.00 |   3.00 |   3.00 |
: | Maximums:            | 125.00 |  88.00 |  86.00 |  95.00 |
: | Medians:             | 115.00 |  88.00 |  85.00 |  92.00 |
: | Minimums:            | 102.00 |  88.00 |  78.00 |  92.00 |
: | Modes:               |        |        |        |        |
: | Standard Deviations: |  11.53 |   0.00 |   4.36 |   1.73 |
: | Sums:                | 342.00 | 264.00 | 249.00 | 279.00 |
: |----------------------+--------+--------+--------+--------|
: #+TBLFM: @>$2..@>$>=vsum(@I..@II);%.2f::@>>$2..@>>$>=vsdev(@I..@II
);%.2f::@>>>>$2..@>>>>$>=vmin(@I..@II);%.2f::@>>>>>$2..@
>>>>>$>=vmedian(@I..@II);%.2f::@>>>>>>$2..@>>>>>>$>=vmax(@I..@II);%.2f::@
>>>>>>>$2..@>>>>>>>$>=vcount(@I..@II);%.2f::@>>>>>>>>$2..@
>>>>>>>>$>=vmean(@I..@II);%.2f

Regards,
Jonathan

On Tue, 2 Jun 2015, Jonathan Leech-Pepin wrote:
>
>  Date: Tue, 2 Jun 2015 08:04:20
>> From: Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com>
>> To: Jude DaShiell <jdashiel@panix.com>
>> Cc: Org Mode Mailing List <emacs-orgmode@gnu.org>
>> Subject: Re: [O] searching for csv utilities
>>
>> Hello,
>>
>> On 2 June 2015 at 07:44, Jude DaShiell <jdashiel@panix.com> wrote:
>>
>>  | Date             |               Sys | Dia | Pul | Sugar |
>>> |------------------+-------------------+-----+-----+-------|
>>> | [2014-04-27 Sun] |               125 |  88 |  78 |    92 |
>>> | [2014-04-28 Mon] |               102 |  88 |  86 |    92 |
>>> | Averages:        | =$2=vmean(@<..@>) |     |     |       |
>>> #+TBLFM: $2=$2=vmean(@<..@>)
>>>
>>>
>> The formula in question is the culprit in this case (at least as stated
>> there).
>>
>> : $2=$2=vmean(@<..@>)
>>
>> Second column is equal to the second column which is equal to the mean of
>> all the values in the second column (including the header "Sys").
>>
>> If you change the table as follows:
>>
>> | Date             |   Sys | Dia | Pul | Sugar |
>> |------------------+-------+-----+-----+-------|
>> | [2014-04-27 Sun] |   125 |  88 |  78 |    92 |
>> | [2014-04-28 Mon] |   102 |  88 |  86 |    92 |
>> |------------------+-------+-----+-----+-------|
>> | Averages:        | 113.5 |  88 |  82 |    92 |
>> #+TBLFM: @>$2..@>$>=vmean(@I..@II)
>>
>> All the values will properly compute.  If you want to avoid the second
>> HLINE above Averages: then change =@II= to =@>>= (penultimate row)
>>
>> Regards,
>> Jon
>>
>>
>>  This is a cut down version of my full record set.  Sometimes when I key
>>> formulas in I get ?ERROR back for a result after keying in c-c+c-c once
>>> I've completed the formula and hit tab.  If I do c-u+c-c+c-c that
>>> sometimes
>>> generated ?ERROR.  Other times I key in a formula and the cursor gets
>>> locked and I have to hit c-g to exit #+TBLFM: mode; I don't know what's
>>> actually happening when that situation arises since other than suddenly
>>> finding the cursor locked I can neither tell what state I'm in or if a
>>> few
>>> more keystrokes are needed or if I've generated an error situation.
>>>
>>>  --
>>>
>>>
>>>
>>>
>>
> --
>
>

[-- Attachment #2: Type: text/html, Size: 7679 bytes --]

Re: searching for csv utilities

[Jude DaShiell]

Hi Jonathan,

calc has vmode if that helps any.
On Wed, 3 Jun 2015, Jonathan Leech-Pepin 
wrote:

> Date: Wed, 3 Jun 2015 12:38:37
> From: Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com>
> To: Jude DaShiell <jdashiel@panix.com>
> Cc: Org Mode Mailing List <emacs-orgmode@gnu.org>
> Subject: Re: [O] searching for csv utilities
> 
> On 3 June 2015 at 12:07, Jude DaShiell <jdashiel@panix.com> wrote:
>
>> This is a piece of a modified ecm that may show what's going on.
>>
>> cut here.
>> |----------------------+---------+--------+--------+--------|
>> | Averages:            |         |        |        |        |
>> | Counts:              |         |        |        |        |
>> | Maximums:            |         |        |        |        |
>> | Medians:             |         |        |        |        |
>> | Minimums:            |         |        |        |        |
>> | Modes:               |         |        |        |        |
>> | Standard Deviations: |         |        |        |        |
>> | Sums:                | 108.69) | 70.45) | 66.62) | 92.93) |
>> |----------------------+---------+--------+--------+--------|
>> #+TBLFM: @>$2..@>$>=vmean(@I..@>>;%.2f)
>>
>> I haven't even attempted the rest of the math since I have no way to
>> predict where any of the results will land.
>>
>
> @> means last line, @>> is second to last, @>>> third to last and so on.
> So for 7th from the bottom it would be @>>>>>>>.
>
> Re: PrintF specification
> Everything after the =;= is considered part of the specification, so the
> =)= used to close the vmean is actually part of the specification.
> Changing that to =vmean();%.2f= will correct it.
>
>
>
> For your sample ECM (plus original data and one sample line to actually
> confirm median works) you would work with the following (Apologies for the
> very long TBLFM line):
>
> I was unable to find a built-in to determine the mode.  I've found sample
> functions on Stack Overflow that would calculate it based on a list, but
> I'm not familiar enough with Org-Table format on how to go from cell
> references =@I..@II= to a list of values for the sake of manipulating them.
>
> : | Date                 |    Sys |    Dia |    Pul |  Sugar |
> : |----------------------+--------+--------+--------+--------|
> : | [2014-04-27 Sun]     |    125 |     88 |     78 |     92 |
> : | [2014-04-28 Mon]     |    102 |     88 |     86 |     92 |
> : | [2014-04-29 Tue]     |    115 |     88 |     85 |     95 |
> : |----------------------+--------+--------+--------+--------|
> : | Averages:            | 114.00 |  88.00 |  83.00 |  93.00 |
> : | Counts:              |   3.00 |   3.00 |   3.00 |   3.00 |
> : | Maximums:            | 125.00 |  88.00 |  86.00 |  95.00 |
> : | Medians:             | 115.00 |  88.00 |  85.00 |  92.00 |
> : | Minimums:            | 102.00 |  88.00 |  78.00 |  92.00 |
> : | Modes:               |        |        |        |        |
> : | Standard Deviations: |  11.53 |   0.00 |   4.36 |   1.73 |
> : | Sums:                | 342.00 | 264.00 | 249.00 | 279.00 |
> : |----------------------+--------+--------+--------+--------|
> : #+TBLFM: @>$2..@>$>=vsum(@I..@II);%.2f::@>>$2..@>>$>=vsdev(@I..@II
> );%.2f::@>>>>$2..@>>>>$>=vmin(@I..@II);%.2f::@>>>>>$2..@
>>>>>> $>=vmedian(@I..@II);%.2f::@>>>>>>$2..@>>>>>>$>=vmax(@I..@II);%.2f::@
>>>>>>>> $2..@>>>>>>>$>=vcount(@I..@II);%.2f::@>>>>>>>>$2..@
>>>>>>>>> $>=vmean(@I..@II);%.2f
>
> Regards,
> Jonathan
>
> On Tue, 2 Jun 2015, Jonathan Leech-Pepin wrote:
>>
>>  Date: Tue, 2 Jun 2015 08:04:20
>>> From: Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com>
>>> To: Jude DaShiell <jdashiel@panix.com>
>>> Cc: Org Mode Mailing List <emacs-orgmode@gnu.org>
>>> Subject: Re: [O] searching for csv utilities
>>>
>>> Hello,
>>>
>>> On 2 June 2015 at 07:44, Jude DaShiell <jdashiel@panix.com> wrote:
>>>
>>>  | Date             |               Sys | Dia | Pul | Sugar |
>>>> |------------------+-------------------+-----+-----+-------|
>>>> | [2014-04-27 Sun] |               125 |  88 |  78 |    92 |
>>>> | [2014-04-28 Mon] |               102 |  88 |  86 |    92 |
>>>> | Averages:        | =$2=vmean(@<..@>) |     |     |       |
>>>> #+TBLFM: $2=$2=vmean(@<..@>)
>>>>
>>>>
>>> The formula in question is the culprit in this case (at least as stated
>>> there).
>>>
>>> : $2=$2=vmean(@<..@>)
>>>
>>> Second column is equal to the second column which is equal to the mean of
>>> all the values in the second column (including the header "Sys").
>>>
>>> If you change the table as follows:
>>>
>>> | Date             |   Sys | Dia | Pul | Sugar |
>>> |------------------+-------+-----+-----+-------|
>>> | [2014-04-27 Sun] |   125 |  88 |  78 |    92 |
>>> | [2014-04-28 Mon] |   102 |  88 |  86 |    92 |
>>> |------------------+-------+-----+-----+-------|
>>> | Averages:        | 113.5 |  88 |  82 |    92 |
>>> #+TBLFM: @>$2..@>$>=vmean(@I..@II)
>>>
>>> All the values will properly compute.  If you want to avoid the second
>>> HLINE above Averages: then change =@II= to =@>>= (penultimate row)
>>>
>>> Regards,
>>> Jon
>>>
>>>
>>>  This is a cut down version of my full record set.  Sometimes when I key
>>>> formulas in I get ?ERROR back for a result after keying in c-c+c-c once
>>>> I've completed the formula and hit tab.  If I do c-u+c-c+c-c that
>>>> sometimes
>>>> generated ?ERROR.  Other times I key in a formula and the cursor gets
>>>> locked and I have to hit c-g to exit #+TBLFM: mode; I don't know what's
>>>> actually happening when that situation arises since other than suddenly
>>>> finding the cursor locked I can neither tell what state I'm in or if a
>>>> few
>>>> more keystrokes are needed or if I've generated an error situation.
>>>>
>>>>  --
>>>>
>>>>
>>>>
>>>>
>>>
>> --
>>
>>
>

-- 

Re: searching for csv utilities

[Nick Dokos]

Jude DaShiell <jdashiel@panix.com> writes:

> Hi Jonathan,
>
> calc has vmode if that helps any.

I think Jonathan is right that calc does not have a vmode. I thought it
did at first, but looking further I could not find any reference to it
in the calc manual or in emacs help. Why do you think it exists?

Nick

Re: searching for csv utilities

[Nick Dokos]

Jude DaShiell <jdashiel@panix.com> writes:

> This is a piece of a modified ecm that may show what's going on.
>
> cut here.
> |----------------------+---------+--------+--------+--------|
> | Averages:            |         |        |        |        |
> | Counts:              |         |        |        |        |
> | Maximums:            |         |        |        |        |
> | Medians:             |         |        |        |        |
> | Minimums:            |         |        |        |        |
> | Modes:               |         |        |        |        |
> | Standard Deviations: |         |        |        |        |
> | Sums:                | 108.69) | 70.45) | 66.62) | 92.93) |
> |----------------------+---------+--------+--------+--------|
> #+TBLFM: @>$2..@>$>=vmean(@I..@>>;%.2f)
>

The syntax is wrong - the format has to go *outside* the parens
of the function. Also the row spec is almost certainly wrong -
something like the following is needed (but see below for a more
exact specification of what the table looks like and what the
actual TBLFM line should be):

#+TBLFM: @>$2..@>$>=vmean(@I..@II);%.2f

The @I..@II restricts the range of rows to be considered to be between
the first and the second hline. IOW, the assumption is that you have
three areas in your table: a header line, followed by an hline, a number
of rows with data, followed by an hline and a number of rows containing
calculated statistics, followed by an (optional) hline - something like
this:

| Date                 |    Sys |   Dia |   Pul | Sugar |
|----------------------+--------+-------+-------+-------|
| [2014-04-27 Sun]     |    125 |    88 |    78 |    92 |
| [2014-04-28 Mon]     |    102 |    88 |    86 |    92 |
|----------------------+--------+-------+-------+-------|
| Averages:            | 113.50 | 88.00 | 82.00 | 92.00 |
| Counts:              |      2 |     2 |     2 |     2 |
| Maximums:            | 125.00 | 88.00 | 86.00 | 92.00 |
| Medians:             |        |       |       |       |
| Minimums:            |        |       |       |       |
| Modes:               |        |       |       |       |
| Standard Deviations: |        |       |       |       |
| Sums:                |        |       |       |       |
|----------------------+--------+-------+-------+-------|
#+TBLFM: @>>>>>>>>$2..@>>>>>>>>$> = vmean(@I..@II);%.2f :: @>>>>>>>$2..@>>>>>>>$> = vcount(@I..@II) :: @>>>>>>$2..@>>>>>>$> = vmax(@I..@II);%.2f

The only challenges left are to refer to the correct row and to find the
appropriate calc function for the statistic at hand.

Unfortunately, org mode is not particularly flexible in referring to
cells (although it is a *lot* more flexible than it used to be). In
particular, referring to the "Averages:" row can not be done simply
(afaik): it's the first row after the second hline, so you'd think that
you can refer to it as @II+1 but hline-relative references are not
allowed on the LHS, so the best that I could do is refer to it as the
eightth column from the bottom: @>>>>>>>> and similarly for the others -
looks like chicken scratchings (or what amounts to the same thing, Perl
:-) ).

With that caveat, the above TBLFM line will calculate the first three of
the eight statistics you want - the rest can be done similarly with the
calc functions vmedian, vmin, vmode, vstdev and vsum; but you can see
it's getting unwieldy, particularly since the whole thing has to be on
one line (or has that changed? I'm not keeping track any longer...)

I believe that you are (partially or completely?) blind, which makes
this all the more difficult: counting how many "greater than" signs are
in each of those row specs is a pain for anybody, but I can't even
imagine how a blind person would deal with it.

> I haven't even attempted the rest of the math since I have no way to 
> predict where any of the results will land.
> On Tue, 2 
> Jun 2015, Jonathan Leech-Pepin wrote:
>
>> Date: Tue, 2 Jun 2015 08:04:20
>> From: Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com>
>> To: Jude DaShiell <jdashiel@panix.com>
>> Cc: Org Mode Mailing List <emacs-orgmode@gnu.org>
>> Subject: Re: [O] searching for csv utilities
>> 
>> Hello,
>>
>> On 2 June 2015 at 07:44, Jude DaShiell <jdashiel@panix.com> wrote:
>>
>>> | Date             |               Sys | Dia | Pul | Sugar |
>>> |------------------+-------------------+-----+-----+-------|
>>> | [2014-04-27 Sun] |               125 |  88 |  78 |    92 |
>>> | [2014-04-28 Mon] |               102 |  88 |  86 |    92 |
>>> | Averages:        | =$2=vmean(@<..@>) |     |     |       |
>>> #+TBLFM: $2=$2=vmean(@<..@>)
>>>
>>
>> The formula in question is the culprit in this case (at least as stated
>> there).
>>
>> : $2=$2=vmean(@<..@>)
>>
>> Second column is equal to the second column which is equal to the mean of
>> all the values in the second column (including the header "Sys").
>>
>> If you change the table as follows:
>>
>> | Date             |   Sys | Dia | Pul | Sugar |
>> |------------------+-------+-----+-----+-------|
>> | [2014-04-27 Sun] |   125 |  88 |  78 |    92 |
>> | [2014-04-28 Mon] |   102 |  88 |  86 |    92 |
>> |------------------+-------+-----+-----+-------|
>> | Averages:        | 113.5 |  88 |  82 |    92 |
>> #+TBLFM: @>$2..@>$>=vmean(@I..@II)
>>
>> All the values will properly compute.  If you want to avoid the second
>> HLINE above Averages: then change =@II= to =@>>= (penultimate row)
>>
>> Regards,
>> Jon
>>
>>
>>> This is a cut down version of my full record set.  Sometimes when I key
>>> formulas in I get ?ERROR back for a result after keying in c-c+c-c once
>>> I've completed the formula and hit tab.  If I do c-u+c-c+c-c that sometimes
>>> generated ?ERROR.  Other times I key in a formula and the cursor gets
>>> locked and I have to hit c-g to exit #+TBLFM: mode; I don't know what's
>>> actually happening when that situation arises since other than suddenly
>>> finding the cursor locked I can neither tell what state I'm in or if a few
>>> more keystrokes are needed or if I've generated an error situation.
>>>
>>>  --
>>>
>>>
>>>
>>

HTH,
Nick

Re: searching for csv utilities

[Nick Dokos]

Nick Dokos <ndokos@gmail.com> writes:

> With that caveat, the above TBLFM line will calculate the first three of
> the eight statistics you want - the rest can be done similarly with the
> calc functions vmedian, vmin, vmode, vstdev and vsum

As Jonathan Leech-Pepin points out, there is no vmode - my mistake.

Nick

Re: searching for csv utilities

[Michael Brand]

On Wed, Jun 3, 2015 at 7:21 PM, Nick Dokos <ndokos@gmail.com> wrote:
> | Date                 |    Sys |   Dia |   Pul | Sugar |
> |----------------------+--------+-------+-------+-------|
> | [2014-04-27 Sun]     |    125 |    88 |    78 |    92 |
> | [2014-04-28 Mon]     |    102 |    88 |    86 |    92 |
> |----------------------+--------+-------+-------+-------|
> | Averages:            | 113.50 | 88.00 | 82.00 | 92.00 |
> | Counts:              |      2 |     2 |     2 |     2 |
> | Maximums:            | 125.00 | 88.00 | 86.00 | 92.00 |
> | Medians:             |        |       |       |       |
> | Minimums:            |        |       |       |       |
> | Modes:               |        |       |       |       |
> | Standard Deviations: |        |       |       |       |
> | Sums:                |        |       |       |       |
> |----------------------+--------+-------+-------+-------|
> #+TBLFM: @>>>>>>>>$2..@>>>>>>>>$> = vmean(@I..@II);%.2f :: @>>>>>>>$2..@>>>>>>>$> = vcount(@I..@II) :: @>>>>>>$2..@>>>>>>$> = vmax(@I..@II);%.2f

I just found the Calc function apply() which leads to this
simplification:

| Date             |    Sys |    Dia |    Pul |  Sugar |
|------------------+--------+--------+--------+--------|
| [2014-04-27 Sun] |    125 |     88 |     78 |     92 |
| [2014-04-28 Mon] |    102 |     88 |     86 |     92 |
|------------------+--------+--------+--------+--------|
| vmean            | 113.50 |  88.00 |  82.00 |  92.00 |
| vcount           |   2.00 |   2.00 |   2.00 |   2.00 |
| vmax             | 125.00 |  88.00 |  86.00 |  92.00 |
| vmedian          | 113.50 |  88.00 |  82.00 |  92.00 |
| vmin             | 102.00 |  88.00 |  78.00 |  92.00 |
| vstdev           |   0.00 |   0.00 |   0.00 |   0.00 |
| vsum             | 227.00 | 176.00 | 164.00 | 184.00 |
|------------------+--------+--------+--------+--------|
#+TBLFM: @>>>>>>>$2..@>$> = apply($1, @I..@II); %.2f

Michael

Re: searching for csv utilities

[Nick Dokos]

Michael Brand <michael.ch.brand@gmail.com> writes:

> On Wed, Jun 3, 2015 at 7:21 PM, Nick Dokos <ndokos@gmail.com> wrote:
>> | Date                 |    Sys |   Dia |   Pul | Sugar |
>> |----------------------+--------+-------+-------+-------|
>> | [2014-04-27 Sun]     |    125 |    88 |    78 |    92 |
>> | [2014-04-28 Mon]     |    102 |    88 |    86 |    92 |
>> |----------------------+--------+-------+-------+-------|
>> | Averages:            | 113.50 | 88.00 | 82.00 | 92.00 |
>> | Counts:              |      2 |     2 |     2 |     2 |
>> | Maximums:            | 125.00 | 88.00 | 86.00 | 92.00 |
>> | Medians:             |        |       |       |       |
>> | Minimums:            |        |       |       |       |
>> | Modes:               |        |       |       |       |
>> | Standard Deviations: |        |       |       |       |
>> | Sums:                |        |       |       |       |
>> |----------------------+--------+-------+-------+-------|
>> #+TBLFM: @>>>>>>>>$2..@>>>>>>>>$> = vmean(@I..@II);%.2f ::
>> @>>>>>>>$2..@>>>>>>>$> = vcount(@I..@II) :: @>>>>>>$2..@>>>>>>$> =
>> vmax(@I..@II);%.2f
>
> I just found the Calc function apply() which leads to this
> simplification:
>
> | Date             |    Sys |    Dia |    Pul |  Sugar |
> |------------------+--------+--------+--------+--------|
> | [2014-04-27 Sun] |    125 |     88 |     78 |     92 |
> | [2014-04-28 Mon] |    102 |     88 |     86 |     92 |
> |------------------+--------+--------+--------+--------|
> | vmean            | 113.50 |  88.00 |  82.00 |  92.00 |
> | vcount           |   2.00 |   2.00 |   2.00 |   2.00 |
> | vmax             | 125.00 |  88.00 |  86.00 |  92.00 |
> | vmedian          | 113.50 |  88.00 |  82.00 |  92.00 |
> | vmin             | 102.00 |  88.00 |  78.00 |  92.00 |
> | vstdev           |   0.00 |   0.00 |   0.00 |   0.00 |
> | vsum             | 227.00 | 176.00 | 164.00 | 184.00 |
> |------------------+--------+--------+--------+--------|
> #+TBLFM: @>>>>>>>$2..@>$> = apply($1, @I..@II); %.2f
>

Delicious!

Thanks,
Nick

Re: searching for csv utilities

[e.fraga]

On Wednesday,  3 Jun 2015 at 22:05, Michael Brand wrote:

[...]

> I just found the Calc function apply() which leads to this
> simplification:
>
> | Date             |    Sys |    Dia |    Pul |  Sugar |
> |------------------+--------+--------+--------+--------|
> | [2014-04-27 Sun] |    125 |     88 |     78 |     92 |
> | [2014-04-28 Mon] |    102 |     88 |     86 |     92 |
> |------------------+--------+--------+--------+--------|
> | vmean            | 113.50 |  88.00 |  82.00 |  92.00 |
> | vcount           |   2.00 |   2.00 |   2.00 |   2.00 |
> | vmax             | 125.00 |  88.00 |  86.00 |  92.00 |
> | vmedian          | 113.50 |  88.00 |  82.00 |  92.00 |
> | vmin             | 102.00 |  88.00 |  78.00 |  92.00 |
> | vstdev           |   0.00 |   0.00 |   0.00 |   0.00 |
> | vsum             | 227.00 | 176.00 | 164.00 | 184.00 |
> |------------------+--------+--------+--------+--------|
>
> #+TBLFM: @>>>>>>>$2..@>$> = apply($1, @I..@II); %.2f
>
> Michael

very nice!  one to remember.

however, vstdev does not appear to work?

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.4.1, Org release_8.3beta-790-gb719c1.dirty

Re: searching for csv utilities

[Michael Brand]

Hi Eric

On Thu, Jun 4, 2015 at 9:21 AM,  <e.fraga@ucl.ac.uk> wrote:
> however, vstdev does not appear to work?

My mistake, it should be vsdev or vpsdev.

Michael

Re: searching for csv utilities

[Nick Dokos]

Michael Brand <michael.ch.brand@gmail.com> writes:

> Hi Eric
>
> On Thu, Jun 4, 2015 at 9:21 AM,  <e.fraga@ucl.ac.uk> wrote:
>> however, vstdev does not appear to work?
>
> My mistake, it should be vsdev or vpsdev.
>

I mispelled it the same way: I can never remember the right name.

Also, if one uses a format on an undefined function (e.g. using
vmode(@I..@II); %.2f), I get 0.0 as a result and the error is hidden -
if the format is omitted, the literal vmode([...]) appears in the table,
so in that case at least one gets a hint that something is wrong. That
probably qualifies as a bug, but I don't know how easy it would be to fix.

Nick

Re: searching for csv utilities

[Nick Dokos]

Nick Dokos <ndokos@gmail.com> writes:

>
> I mispelled it the same way: I can never remember the right name.
>

... and I misspelled "misspelled" :-)

Re: searching for csv utilities

[Michael Brand]

Hi Nick

On Thu, Jun 4, 2015 at 5:22 PM, Nick Dokos <ndokos@gmail.com> wrote:
> Also, if one uses a format on an undefined function (e.g. using
> vmode(@I..@II); %.2f), I get 0.0 as a result and the error is hidden -
> if the format is omitted, the literal vmode([...]) appears in the table,
> so in that case at least one gets a hint that something is wrong. That
> probably qualifies as a bug, but I don't know how easy it would be to fix.

Yes. Anyway, Calc has its own formatter which can be a solution:

| Date             |              Sys |            Dia |
Pul |          Sugar |
|------------------+------------------+----------------+----------------+----------------|
| [2014-04-27 Sun] |              125 |             88 |
78 |             92 |
| [2014-04-28 Mon] |              102 |             88 |
86 |             92 |
|------------------+------------------+----------------+----------------+----------------|
| vmean            |           113.50 |          88.00 |
82.00 |          92.00 |
| vcount           |             2.00 |           2.00 |
2.00 |           2.00 |
| vmax             |           125.00 |          88.00 |
86.00 |          92.00 |
| vmedian          |           113.50 |          88.00 |
82.00 |          92.00 |
| vmin             |           102.00 |          88.00 |
78.00 |          92.00 |
| vstdev           | vstdev(125, 102) | vstdev(88, 88) | vstdev(78,
86) | vstdev(92, 92) |
| vsum             |           227.00 |         176.00 |
164.00 |         184.00 |
|------------------+------------------+----------------+----------------+----------------|
#+TBLFM: @>>>>>>>$2..@>$> = apply($1, @I..@II) +.0; f-2

See also the Worg FAQ
http://orgmode.org/worg/org-faq.html#table-float-fraction

Michael

Re: searching for csv utilities

[Jude DaShiell]

First, I tried usinb multiple #+TBLFM: lines to do all of the math and 
that only did the first line of calculations which was the sums which 
appear last.  After replacing the double colons between calculation 
lines in the single line form of the #+TBLFM: line seven of the 8 lines 
worked correctly.  The vmode() function only returned zeros since the 
columns weren't sorted.  I did find a calc-sort function in the calc 
info and tried vmode(sort(@I..@II))%.2f (nested functions) and that also 
returned zeros on the mode line.  Must be some other way to use that 
function I'll have to search out.  Thanks much for the help on this 
project, it probably will be stolen from and used in other projects 
here.

>>  -- >> >> >> >

-- 

Re: searching for csv utilities

[Thierry Banel]

You may want to consider a slightly different approach. It may (or
maynot) be relevant depending on your context.

First, denormalize your table into a "fact table" (which is handy when
dealingwith cubes):

#+name: data
| date             | type  |   q |
|------------------+-------+-----|
| [2014-04-27 Sun] | Sys   | 125 |
| [2014-04-28 Mon] | Sys   | 102 |
| [2014-04-29 Tue] | Sys   | 115 |
| [2014-04-27 Sun] | Dia   |  88 |
| [2014-04-28 Mon] | Dia   |  88 |
| [2014-04-29 Tue] | Dia   |  88 |
| [2014-04-27 Sun] | Pul   |  78 |
| [2014-04-28 Mon] | Pul   |  86 |
| [2014-04-29 Tue] | Pul   |  85 |
| [2014-04-27 Sun] | Sugar |  92 |
| [2014-04-28 Mon] | Sugar |  92 |
| [2014-04-29 Tue] | Sugar |  95 |

Then using a package named "org-aggregate" (on Melpa) you can derivean
aggregated table:

#+name: aggr
#+BEGIN: aggregate :table data :cols "type vcount() vmean(q) vmax(q)
vsdev(q);f3"
| type  | vcount() | vmean(q) | vmax(q) | vsdev(q);f3 |
|-------+----------+----------+---------+-------------|
| Sys   |        3 |      114 |     125 |      11.533 |
| Dia   |        3 |       88 |      88 |           0 |
| Pul   |        3 |       83 |      86 |       4.359 |
| Sugar |        3 |       93 |      95 |       1.732 |
#+END:

If you want to rotate this table, apply a transposition on it(provided
by the same package):

#+BEGIN: transpose :table "aggr"
| type        |   |    Sys | Dia |   Pul | Sugar |
| vcount()    |   |      3 |   3 |     3 |     3 |
| vmean(q)    |   |    114 |  88 |    83 |    93 |
| vmax(q)     |   |    125 |  88 |    86 |    95 |
| vsdev(q);f3 |   | 11.533 |   0 | 4.359 | 1.732 |
#+END:

You can sort the "data" table on dates without changing theaggregation.

Thierry

Le 02/06/2015 13:44, Jude DaShiell a écrit :
> | Date             |               Sys | Dia | Pul | Sugar |
> |------------------+-------------------+-----+-----+-------|
> | [2014-04-27 Sun] |               125 |  88 |  78 |    92 |
> | [2014-04-28 Mon] |               102 |  88 |  86 |    92 |
> | Averages:        | =$2=vmean(@<..@>) |     |     |       |
> #+TBLFM: $2=$2=vmean(@<..@>)
>
> This is a cut down version of my full record set.  Sometimes when I
> key formulas in I get ?ERROR back for a result after keying in c-c+c-c
> once I've completed the formula and hit tab.  If I do c-u+c-c+c-c that
> sometimes generated ?ERROR.  Other times I key in a formula and the
> cursor gets locked and I have to hit c-g to exit #+TBLFM: mode; I
> don't know what's actually happening when that situation arises since
> other than suddenly finding the cursor locked I can neither tell what
> state I'm in or if a few more keystrokes are needed or if I've
> generated an error situation.
>
>  --
>
>
>

Re: searching for csv utilities

[Eric S Fraga]

On Thursday,  4 Jun 2015 at 23:24, Thierry Banel wrote:
> You may want to consider a slightly different approach. It may (or
> maynot) be relevant depending on your context.

[...]

> Then using a package named "org-aggregate" (on Melpa) you can derivean
> aggregated table:

Thanks for reminding me (us?) about this package.  Very useful!
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.4.1, Org release_8.3beta-1195-g1a7364

Re: searching for csv utilities

[Jude DaShiell]

I spent time looking for vmode in info org using a regular expression 
search and couldn't find it either.  That capability is probably in R 
though it may be under another name.  I have emacs-ess package and R 
installed on this system though haven't used them yet.  I also found sbcl 
so if it gets too difficult to get stuff done with R I'll have an option 
open to start studying common lisp and write a mode function I can use 
with common lisp in conjunction with orgmode.

Interestingly, I did find pop_stdev() in postgresql but it turned out to 
be one of those situations like you were better off never having found 
that function since I managed to break a query that was working up until I 
added that extra code into it.  I was using aliases and pop_stdev() needs 
explicit casting to work at all for some reason.  That's what happens on 
learning curves every so often.



--