Re: addition of 0.7 and 0.1 is not 0.8 CDbl / CStr vbs problem

On 21 Jun., 17:01, "Anthony Jones" <A...@xxxxxxxxxxxxxxxx> wrote:
<r...@xxxxxxxxx> wrote in message

news:1182436865.902509.220780@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx



Hello Group,

i have some vbs code here and would be happy if one of you could
describe whats wrong with it.
i just would like to make addition of 0.1 and 0.7 but the result just
surprises me everytime again.
btw. i work on a system with german number format.

IF ((CDbl("0,7")+CDbl("0,1"))=(CDbl("0,8"))) THEN
MSGBOX "0,7 + 0,1 = 0,8"
ELSE
MSGBOX "0,7 + 0,1 IS NOT 0,8"
END IF

the solution for this problem is:
IF (CDbl(CStr(CDbl("0,7")+CDbl("0,1")))=(CDbl("0,8"))) THEN
MSGBOX "0,7 + 0,1 = 0,8"
ELSE
MSGBOX "0,7 + 0,1 IS NOT 0,8"
END IF

but what to hell is going on ???

kindest regards,
Robin

btw. the same prob occures with 10,7 + 0,1

This is a common question. The fact is that due to the way floating point
values are stored there will be very small rounding errors. The actual
value stored for 0.7 + 0.1 is very slightly different from 0.8. So small is
this variation that when it comes to converting to a string the value is
displayed as 0.8 either way it is derived. However = will compare them
exactly and they differ.

Comparing floating point values should be done using a tolerance :-

If Abs(CDbl("0.7") + CDbl("0.1") - CDbl(0.8)) < 0.000001 Then
'Near enough
Else
'Not near enough
End If

Most newbies find this situation shocking yet it's pratically as old as
computer programming itself.

Thanks for answering. Something is not really clear to me with your
explaination. if there are little rounding errors, why do they only
occure with 0.7 and 10.7. using my example with 0.3+0.1=0.4 the
problem does not happen. and if i use CDbl(CStr()) with the addition,
the problem also does not happen. isn't it a better way dealing with
the situation using CDbl(CStr()) than to compare with "almost near
enough"?

Regards,
Robin


.

Relevant Pages

  • Re: Weirdness: (-0.666667 + 0.333333)
    ... > becuase of rounding errors. ... > I'm writing a program that draws graphs on a cartesian plane, ... The only thing you can do is to compare your value of x to zero ...
    (comp.lang.cpp)
  • Re: Weirdness: (-0.666667 + 0.333333)
    ... I'm not really trying to compare 2 double which could be in accurate ... becuase of rounding errors. ... > I'm experiencing some weirdness in a program, ...
    (comp.lang.cpp)
  • Comparing floating point values in Java
    ... when comparing floating point values in computers. ... will unchanged values compare strictly true if equal? ... float a = 1; ...
    (comp.lang.java.programmer)
  • Re: Weirdness: (-0.666667 + 0.333333)
    ... In message, Corne' Cornelius ... >I'm not really trying to compare 2 double which could be in accurate ... You can often improve matters by using an integer loop counter: ... It doesn't get rid of the rounding errors, but it does ensure that they ...
    (comp.lang.cpp)