Re: This calculation is just wrong / computer can't count!

"AliR (VC++ MVP)" <AliR@xxxxxxxxxxxxx> wrote in message
news:En7Ni.30834$eY.28056@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Maybe this will clear some of it up:
http://www.google.com/search?q=Significant+digits+double&hl=en

That info on significant digits was all very interesting, but how do I get
my code to work properly?!? I want to force the computer to calculate and
store as many digits as it can handle and no extra spurious digits, so that
my calculations get nice accurate results. At the moment part of my
calculation ends up with -0.00000000000007 (haven't counted the zeros here,
but you get the idea!), where it should be zero.


.

Relevant Pages

  • Re: An uncountable countable set
    ... >> That the calculation of the digits can be done in parallel? ... but it may depend on the ordering of the calculations. ... By Cantor, Dedekind, Weierstrass and a host of others (using ... binary representations but were not). ...
    (sci.math)
  • Re: FAQ 4.2 Why is int() broken?
    ... To express 1/10 in binary you need an infinite ... number of digits, just like you need an infinite number of digits ... Even among different types of computers floating point calculations are ...
    (comp.lang.perl.misc)
  • Re: FAQ 4.2 Why is int() broken?
    ... To express 1/10 in binary you need an infinite ... number of digits, just like you need an infinite number of digits ... Even among different types of computers floating point calculations are ... is essentially tiny computer with a 6 MHz cpu and yet it can do many ...
    (comp.lang.perl.misc)
  • Re: Problem with printf formats
    ... Count of digits in a number indicates the accuracy of the number. ... to add zeros to -2158.5 because the accurate amount was -2158.470. ... The significance of zeros in indicating accuracy can be only perceived ... significant digits, i.e. if -2000 is accurate to first, second, third ...
    (comp.lang.c)
  • Re: Excel Calculation Errors
    ... have to do all of your calculations in VB. ... >>certain limits and found that Excel frequently gave misleading results ... >Excel's specifications include a maximum of 15 digits. ... >precision, I believe. ...
    (microsoft.public.excel.misc)