This calculation is just wrong / computer can't count!
- From: "GT" <ContactGT_removeme_@xxxxxxxxxxx>
- Date: Thu, 4 Oct 2007 15:24:51 +0100
I have been debugging something for ages now. I have a method that does some
complex maths, but right at the beginning it works out a proportion and a
few ratios and the maths is simply wrong. In my code I (obviously) use
variables and the values vary each time the method is called, but there
seems to a problem with the maths. I have narrowed the problem down to the
following. Can someone else please try this simply calculation and see what
their computer gets.
Code Line 1:
double effortChangeProportion = (55.0 - 30.0) / 30.0;
This first line does 55-30 and divides the result by 30. In other words
25/30, which is 0.8333 (recurring 3s).
The computer manages to give the answer 0.83333333333333337 !!
Code Line 2:
effortChangeProportion++;
or
effortChangeProportion = effortChangeProportion + 1.0;
The second line of code (both alternatives give the same result) builds on
the first by simply adding 1 (so I can then multiply other numbers by this
proportion).
In this case 0.8333 becomes 1.8333 , but again the computer gets this wrong.
It tries to add 1 to 0.83333333333333337 and gets 1.8333333333333335.
Obviously this can easily be done in 1 line of code, but it is broken down
to demonstrate the maths going wrong twice!
Can anyone shed some light on this for me please?
GT
.
- Follow-Ups:
- Re: This calculation is just wrong / computer can't count!
- From: Joseph M . Newcomer
- Re: This calculation is just wrong / computer can't count!
- From: Eric Hill
- Re: This calculation is just wrong / computer can't count!
- From: Geoff
- Re: This calculation is just wrong / computer can't count!
- From: Giovanni Dicanio
- Re: This calculation is just wrong / computer can't count!
- From: Steve Kelley
- Re: This calculation is just wrong / computer can't count!
- From: xrxst32
- Re: This calculation is just wrong / computer can't count!
- From: Les
- Re: This calculation is just wrong / computer can't count!
- From: AliR \(VC++ MVP\)
- Re: This calculation is just wrong / computer can't count!
- From: GT
- Re: This calculation is just wrong / computer can't count!
- From: .rhavin grobert
- Re: This calculation is just wrong / computer can't count!
- Prev by Date: Re: Error-- Argument not optional
- Next by Date: Re: Recreate a Window ... how to stop asserts...?
- Previous by thread: Owner draw menu does not call DrawItem
- Next by thread: Re: This calculation is just wrong / computer can't count!
- Index(es):
Relevant Pages
|