Re: Errors in formula calculation

From: Ron Rosenfeld (ronrosenfeld_at_nospam.org)
Date: 02/08/05


Date: Tue, 08 Feb 2005 14:37:08 -0500

On Tue, 08 Feb 2005 13:58:07 GMT, "john.bedford3" <john.bedford3@ntlworld.com>
wrote:

>Is this simply because it is converting to binary and back to decimal?
>
>If I attempt to use precision as displayed a warning comes up that data will
>permanently lose accuracy. Yet it is inaccuracy of the calculation I am
>trying to get rid of.
>
>I have now tried using ROUND and this appears to solve the problem although
>I did not think of that before as simple addition does not indicate that
>there should be more than one decimal place in the answer.
>
>Hopefully this has solved my problem.
>
>Thank you for your help

I'm glad you have solved your problem. And Lewis has given a much clearer
explanation of the issues.

--ron



Relevant Pages

  • Re: demonic numbers !
    ... > corresponding calculation of error intervals. ... Then they're either not doing enough math for rounding errors to ... accumulate or they're not using a wide enough range of magnitudes. ... They don't need more than 53 bits of precision to store those values. ...
    (comp.lang.lisp)
  • =?Utf-8?Q?Re:_I_don=C2=B4t_view_the_value_right?= =?Utf-8?Q?_after_six_decimal_place_in_Exce
    ... You might consider setting the option Tools> Options> Calculation> ... Precision As Displayed. ... final result, not intermediated calculations (subexpressions). ... the workbook first because some changes cannot be undone by simply ...
    (microsoft.public.excel.misc)
  • Re: ROUNDING RESULT OF CALCULATION UP OR DOWN
    ... In one of the calculation rules we had to round to a whole year. ... I keep repeating we need decimal arithmetic in Excel, ... | ??>> the limits of precision? ...
    (microsoft.public.excel.misc)
  • Re: For Andrew Breen, plasma
    ... physicists use slide rules and everything. ... you can get almost all the precision you want! ... calculation with it, but have forgotten the usage of the log-log scales ... I had a couple of K&E 20 inch rules. ...
    (sci.military.naval)
  • Floating Point , Wide zero etc
    ... The default precision of 4 is used. ... require 6 places for GST calculation. ... The simple rule is never store unconverted data for accounting. ... the binary approximation converts back to ...
    (comp.databases.pick)