Re: This calculation is just wrong / computer can't count!
- From: xrxst32 <martin.dangelmeyr@xxxxxxx>
- Date: Thu, 04 Oct 2007 22:54:33 -0700
On Oct 4, 4:24 pm, "GT" <ContactGT_remove...@xxxxxxxxxxx> wrote:
I have been debugging something for ages now. I have a method that does some
complex maths, but right at the beginning it works out a proportion and a
few ratios and the maths is simply wrong. In my code I (obviously) use
variables and the values vary each time the method is called, but there
seems to a problem with the maths. I have narrowed the problem down to the
following. Can someone else please try this simply calculation and see what
their computer gets.
Code Line 1:
double effortChangeProportion = (55.0 - 30.0) / 30.0;
This first line does 55-30 and divides the result by 30. In other words
25/30, which is 0.8333 (recurring 3s).
The computer manages to give the answer 0.83333333333333337 !!
Code Line 2:
effortChangeProportion++;
or
effortChangeProportion = effortChangeProportion + 1.0;
The second line of code (both alternatives give the same result) builds on
the first by simply adding 1 (so I can then multiply other numbers by this
proportion).
In this case 0.8333 becomes 1.8333 , but again the computer gets this wrong.
It tries to add 1 to 0.83333333333333337 and gets 1.8333333333333335.
Obviously this can easily be done in 1 line of code, but it is broken down
to demonstrate the maths going wrong twice!
Can anyone shed some light on this for me please?
GT
Try the DECIMAL datatype of OLE which uses decimal arithmetics
(instead of binary of the IEEE datatypes float/double).
The speed is much slower because there's no hardware support but the
accuracy is much better.
There some nice wrapper on CodeProject like
http://www.codeproject.com/com/decimalwrap.asp
.
- References:
- Prev by Date: Re: How to detect page refresh in vc++
- Next by Date: Downloadbegin fires without page reload and page navigation also
- Previous by thread: Re: This calculation is just wrong / computer can't count!
- Next by thread: Re: This calculation is just wrong / computer can't count!
- Index(es):
Relevant Pages
|
Loading
