Re: This calculation is just wrong / computer can't count!

Tech-Archive recommends: Fix windows errors by optimizing your registry

I'm sorry to disappoint you, but the computer doesn't do rounding in decimal
digits. It does rounding in binary digits. Then whatever gets printed is
just a decimal representation of the rounded binary fractional number. I'm
sorry, but in computational math you don't deal with computational error in
terms of decimal digits, they are only used in simplified rules that an
elementary school student could understand.
You use relative error, which may not map into a particular digit. If you
want nice looking result, you just need to discard any decimal positions
that can be polluted by a calculation error. To choose number of significant
decimal digits is up to you. If you feel that lat '7' is wrong in this
particular calculation, just print one less digit.

"GT" <ContactGT_remove_@xxxxxxxxxxx> wrote in message
news:017b6d06$0$20663$c3e8da3@xxxxxxxxxxxxxxxxxxxx

One problem here is everyone seems to be banding around the words precise
and accurate - they have different meanings. Precision is the number of
detail in something - the number of decimal palces. Accuracy is how
*correct* something is. Example; 0.83 is accurate (mathematically
correct), but not very precise. 0.83333333333337 is very precise, but not
accurate or in other words mathematically 'wrong'. I don't need my
intermediary calculations stored to a high level of precision (only
require calculations to work to 6 or more digits), but I need these
calculations to be accurate (mathematically correct). If you base a series
of calculations on a fundamentally flawed number, then once it has been
squared a few times and multiplied / divided by some other numbers that
are incorrect beyond the 16th digit, then the calculations starts to
display problems. The algorithm works perfectly on paper and when run
through a calculator, but not when programmed in C++.



.

Relevant Pages

  • Re: XQ and ->Qpi bug on large X
    ... you shouldn't be rounding to only two digits ... I am trying to make a similar point here about rounding; ... and round it, ... prior to using the input values in calculations. ...
    (comp.sys.hp48)
  • Re: Rounding errors
    ... >>number of random digits following the point of rounding, ... > an infinite number of digits right of decimal. ... multiplication nor what the number of digits in the calculations was. ...
    (comp.lang.cobol)
  • Re: An uncountable countable set
    ... >> That the calculation of the digits can be done in parallel? ... but it may depend on the ordering of the calculations. ... By Cantor, Dedekind, Weierstrass and a host of others (using ... binary representations but were not). ...
    (sci.math)
  • Re: FAQ 4.2 Why is int() broken?
    ... To express 1/10 in binary you need an infinite ... number of digits, just like you need an infinite number of digits ... Even among different types of computers floating point calculations are ...
    (comp.lang.perl.misc)
  • Re: FAQ 4.2 Why is int() broken?
    ... To express 1/10 in binary you need an infinite ... number of digits, just like you need an infinite number of digits ... Even among different types of computers floating point calculations are ... is essentially tiny computer with a 6 MHz cpu and yet it can do many ...
    (comp.lang.perl.misc)