Re: This calculation is just wrong / computer can't count!

On 5 Okt., 11:38, "GT" <ContactGT_remo...@xxxxxxxxxxx> wrote:
I would like to know how to use the basic C++ datatypes and where the last
digit is 'wrong' or 'unreliable', just don't use it. Everyone keeps saying
'just use the digits you can rely on', but how? I have a double
effortChangeProportion which has 15 or 16 decimal places. HOW do I tell it
to just use the first 10 decimal places, thereby dropping the final digit
which is knocking out my calculations? I can't find any methods on a double
which tell it to truncate digits.

You cant tell yout computer "drop the last x digits in decimal system" when your number is stored in BINARY.
if you're interested in the last 5 digits of 0.444444443094876 _as a
number_ then you should do a

int iResult = (0.444444443094876 * 10000);
then do the rest of your calculation and divide by 10000 at the end
(according to your function you of course may do more than this).

if you're interested in the last 5 digits of 0.444444443094876 _as a
string_ then you should simply do a

CString str;
str.Format("%.5f", 0.444444443094876 );

rhavin;)

.

Relevant Pages

  • Re: An uncountable countable set
    ... >> That the calculation of the digits can be done in parallel? ... but it may depend on the ordering of the calculations. ... By Cantor, Dedekind, Weierstrass and a host of others (using ... binary representations but were not). ...
    (sci.math)
  • Re: FAQ 4.2 Why is int() broken?
    ... To express 1/10 in binary you need an infinite ... number of digits, just like you need an infinite number of digits ... Even among different types of computers floating point calculations are ...
    (comp.lang.perl.misc)
  • Re: FAQ 4.2 Why is int() broken?
    ... To express 1/10 in binary you need an infinite ... number of digits, just like you need an infinite number of digits ... Even among different types of computers floating point calculations are ... is essentially tiny computer with a 6 MHz cpu and yet it can do many ...
    (comp.lang.perl.misc)
  • Re: Excel Calculation Errors
    ... have to do all of your calculations in VB. ... >>certain limits and found that Excel frequently gave misleading results ... >Excel's specifications include a maximum of 15 digits. ... >precision, I believe. ...
    (microsoft.public.excel.misc)
  • Re: How to increase Working Precision?
    ... string of digits or iterates a function many times or ... arbitrary precision routines, evidently baffling the... ... and you get an answer whose number of digits is ... When it comes to the calculations, ...
    (sci.math.symbolic)
Loading