Re: Precision problem in VBasic 6

From: Rick Rothstein (rickNOSPAMnews_at_NOSPAMcomcast.net)
Date: 02/04/05 

Date: Fri, 4 Feb 2005 12:09:47 -0500

> BUT, that what I expect is, no matter how this is done, the result
should be
> accurate in this level of precision.
> Forgive me, but I really believe that this is ridiculous.
> ( -0.7-0.1= -0.8000001)

Given that

     Print (1 / 3) ==> 0.333333333333333

it is no more so ridiculous than this result

     Print (1 / 3) + (1 / 3) ==> 0.666666666666667

When a number cannot be represented with the limits of precision of the
math being used, some kind of approximation must be used somewhere.

> (by the way if you do the same calculations on a simple calculator you
will
> get the correct answer)

Calculators are fooling you. They use something called "guard digits" to
protect the integrity of the display. What this means is that a
calculator calculates to more significant digits than it displays. That
way, any errors that accumulate will be flushed out when the answer is
rounded to the number of digits that calculator actually displays. Think
of this process as using a Double data type, but always rounding your
answers to, say, 12 significant digits (note I said digits, not decimal
place).

Rick - MVP

Relevant Pages

  • Re: float bug? perl 5.8, DBI and oracle 10.2.0
    ... precision numbers in oracle, you've got 38 decimal digits to play ... and with minimal coaxing perl will handle them as ... digits from a 32 bit floating point number - I'll go out on a limb ... and hazard that one can expect 12 or so digits from a 64 bit floating ...
    (perl.dbi.users)
  • Re: Calculating Wishes (was hpcatalog.com)
    ... But compared to modern computers, ... Sun-Earth distance useful even if it's know to a precision clearly ... digits of precision. ... > If a calculator offerred say 19 or more digits of precision, ...
    (comp.sys.hp48)
  • Re: how to add two no. of 100 digits or more?
    ... I wonder how many of those digits are significant? ... Probably about 10 digits of actual precision. ... the precision of that calculator doesn't impress me. ... triple-dubya dott tustinfreezone dott org ...
    (comp.lang.c)
  • RE: float bug? perl 5.8, DBI and oracle 10.2.0
    ... I would not characterise 32-bit signed integers as giving 10 digits ... truncate and tell people you get 9 digits of precision. ... perl 5.8, DBI and oracle 10.2.0 ... Floating point values are typically stored in 64 bits or sometimes 96 ...
    (perl.dbi.users)
  • Re: float bug? perl 5.8, DBI and oracle 10.2.0
    ... I would not characterise 32-bit signed integers as giving 10 digits ... truncate and tell people you get 9 digits of precision. ... perl 5.8, DBI and oracle 10.2.0 ... Floating point values are typically stored in 64 bits or sometimes 96 ...
    (perl.dbi.users)