Re: Precision setting for spreadsheet calculation

[Carsten Dominik <carsten.dominik@gmail.com>]

On Feb 26, 2010, at 10:58 PM, Michael Brand wrote:

> Carsten Dominik wrote:
>> these are the internals of calc, about which I cannot do much.
>
> But
>
> | 0.16666666666666666667 |
> #+TBLFM: $1 = 1 / 2 * 3; p20 f20
>
> shows that the internals of calc are ok, isn't it?

Hi Michael,

you are completely right, I was wrong.

The other issue that is playing into this is that when
using a printf format specifier, the number is first
converted into a Lisp number, which is a double, with
16 significant digits...  At this step, the additional
digits are lost.

Maybe you find this interesting enough so that I can
persuade you to write a FAQ entry about these issues? :-)

- Carsten


>
>> Michael Brand wrote:
>>> But instead of
>>>
>>> | 0.166666666666666660 |
>>> #+TBLFM: $1 = 1 / 2 * 3; %.18f p20 f20
>>>
>>> which I get I would expect
>>>
>>> | 0.166666666666666667 |
>>> #+TBLFM: $1 = 1 / 2 * 3; %.18f p20 f20

- Carsten