Re: This calculation is just wrong / computer can't count!
- From: "AliR \(VC++ MVP\)" <AliR@xxxxxxxxxxxxx>
- Date: Thu, 04 Oct 2007 15:10:28 GMT
Maybe this will clear some of it up:
http://www.google.com/search?q=Significant+digits+double&hl=en
AliR.
"GT" <ContactGT_removeme_@xxxxxxxxxxx> wrote in message
news:00237e30$0$5087$c3e8da3@xxxxxxxxxxxxxxxxxxxx
I have been debugging something for ages now. I have a method that does
some complex maths, but right at the beginning it works out a proportion
and a few ratios and the maths is simply wrong. In my code I (obviously)
use variables and the values vary each time the method is called, but there
seems to a problem with the maths. I have narrowed the problem down to the
following. Can someone else please try this simply calculation and see what
their computer gets.
Code Line 1:
double effortChangeProportion = (55.0 - 30.0) / 30.0;
This first line does 55-30 and divides the result by 30. In other words
25/30, which is 0.8333 (recurring 3s).
The computer manages to give the answer 0.83333333333333337 !!
Code Line 2:
effortChangeProportion++;
or
effortChangeProportion = effortChangeProportion + 1.0;
The second line of code (both alternatives give the same result) builds on
the first by simply adding 1 (so I can then multiply other numbers by this
proportion).
In this case 0.8333 becomes 1.8333 , but again the computer gets this
wrong. It tries to add 1 to 0.83333333333333337 and gets
1.8333333333333335.
Obviously this can easily be done in 1 line of code, but it is broken down
to demonstrate the maths going wrong twice!
Can anyone shed some light on this for me please?
GT
.
- Follow-Ups:
- References:
- Prev by Date: Re: This calculation is just wrong / computer can't count!
- Next by Date: Re: This calculation is just wrong / computer can't count!
- Previous by thread: Re: This calculation is just wrong / computer can't count!
- Next by thread: Re: This calculation is just wrong / computer can't count!
- Index(es):
Relevant Pages
|
