Re: Problem in Float Arithmetic

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

On Mon, 23 Jun 2008 15:53:31 +0200, "Christoph Conrad"
<spamcruncher@xxxxxxxxxxxxxxxxxxxx> wrote:

Hi Doug,

* Doug Harrison [MVP] <dsh@xxxxxxxx> wrote:

So you should use "double".

That will give him a more accurate result, but it still won't be
exact.

That is correct, and the meaning of "precision".

No, I meant "accurate". It won't merely have more digits (precision), the
result will be closer to the mathematical result (accuracy).

--
Doug Harrison
Visual C++ MVP
.

Relevant Pages

  • Re: float bug? perl 5.8, DBI and oracle 10.2.0
    ... precision numbers in oracle, you've got 38 decimal digits to play ... and with minimal coaxing perl will handle them as ... digits from a 32 bit floating point number - I'll go out on a limb ... and hazard that one can expect 12 or so digits from a 64 bit floating ...
    (perl.dbi.users)
  • RE: float bug? perl 5.8, DBI and oracle 10.2.0
    ... I would not characterise 32-bit signed integers as giving 10 digits ... truncate and tell people you get 9 digits of precision. ... perl 5.8, DBI and oracle 10.2.0 ... Floating point values are typically stored in 64 bits or sometimes 96 ...
    (perl.dbi.users)
  • Re: float bug? perl 5.8, DBI and oracle 10.2.0
    ... I would not characterise 32-bit signed integers as giving 10 digits ... truncate and tell people you get 9 digits of precision. ... perl 5.8, DBI and oracle 10.2.0 ... Floating point values are typically stored in 64 bits or sometimes 96 ...
    (perl.dbi.users)
  • Re: Fun With Numbers
    ... If you want to do exact addition, subtraction, multiplication, ... has reports that generate nine digits worth in some reports. ... referred to the ECMAScript standard (ECMA-262 5.1 Edition of June ... that it is good for 15.95 digits of precision. ...
    (comp.lang.javascript)
  • Re: 15 Significant Digits Limitation a Mistake for Spatial Informa
    ... DP does not restrict to 15 decimal digits. ... Input and output precision are more tightly linked in Excel ... Decimal data type or roll your own extended precision data types. ...
    (microsoft.public.excel)