Re: addition of 0.7 and 0.1 is not 0.8 CDbl / CStr vbs problem


<robin@xxxxxxxxx> wrote in message
news:1182502459.529450.56690@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
On 21 Jun., 17:01, "Anthony Jones" <A...@xxxxxxxxxxxxxxxx> wrote:
<r...@xxxxxxxxx> wrote in message

news:1182436865.902509.220780@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx



Hello Group,

i have some vbs code here and would be happy if one of you could
describe whats wrong with it.
i just would like to make addition of 0.1 and 0.7 but the result just
surprises me everytime again.
btw. i work on a system with german number format.

IF ((CDbl("0,7")+CDbl("0,1"))=(CDbl("0,8"))) THEN
MSGBOX "0,7 + 0,1 = 0,8"
ELSE
MSGBOX "0,7 + 0,1 IS NOT 0,8"
END IF

the solution for this problem is:
IF (CDbl(CStr(CDbl("0,7")+CDbl("0,1")))=(CDbl("0,8"))) THEN
MSGBOX "0,7 + 0,1 = 0,8"
ELSE
MSGBOX "0,7 + 0,1 IS NOT 0,8"
END IF

but what to hell is going on ???

kindest regards,
Robin

btw. the same prob occures with 10,7 + 0,1

This is a common question. The fact is that due to the way floating
point
values are stored there will be very small rounding errors. The actual
value stored for 0.7 + 0.1 is very slightly different from 0.8. So
small is
this variation that when it comes to converting to a string the value is
displayed as 0.8 either way it is derived. However = will compare them
exactly and they differ.

Comparing floating point values should be done using a tolerance :-

If Abs(CDbl("0.7") + CDbl("0.1") - CDbl(0.8)) < 0.000001 Then
'Near enough
Else
'Not near enough
End If

Most newbies find this situation shocking yet it's pratically as old as
computer programming itself.

Thanks for answering. Something is not really clear to me with your
explaination. if there are little rounding errors, why do they only
occure with 0.7 and 10.7. using my example with 0.3+0.1=0.4 the
problem does not happen.

It won't happen with all numbers. Some numbers can be exactly represented in
the binary floating point format.
In other cases the resulting 'near enough' value stored after a calculation
may happen to be the same 'near enough' value used when the number is
assigned directly.

and if i use CDbl(CStr()) with the addition,
the problem also does not happen. isn't it a better way dealing with
the situation using CDbl(CStr()) than to compare with "almost near
enough"?

'Better' is subjective. The more convoluted the calculation the more the
rounding errors build up. At some point fairly soon the string generated by
CStr may no longer be what you want. Using CStr will also be slower.





Regards,
Robin




.

Relevant Pages

  • Re: demonic problem descriptions
    ... It's not even USUALLY a disaster in common practice (which is to say, ... The most insidious thing about floating ... you have a better understanding of the representation than the ... calculation you make whose error estimate (which you won't compute, ...
    (comp.lang.lisp)
  • Re: Testable Predictions by HdB
    ... > Jesse F. Hughes wrote: ... > floating point numbers often do not. ... This makes the Testable Predictions ... there are situations in which the calculation will give wrong ...
    (sci.math)
  • Re: Custom fields and internal values
    ... calculation. ... that reads [Duration], I can see a 4 day duration task is shown as ... How did the person know to compare Baseline finish to 10000? ... How is MS Project storing this? ...
    (microsoft.public.project)
  • Re: Comparing floating point values in Java
    ... when comparing floating point values in computers. ... In Java, will unchanged values compare strictly true if equal? ... In strictfp mode, the exact, bit-by-bit, result is predictable from the ...
    (comp.lang.java.programmer)
  • Re: Comparing floating point values in Java
    ... when comparing floating point values in computers. ... In Java, will unchanged values compare strictly true if equal? ... In strictfp mode, the exact, bit-by-bit, result is predictable from the ...
    (comp.lang.java.programmer)