Re: Subtraction Bug in VB

Tech-Archive recommends: Fix windows errors by optimizing your registry

"Sarav" <Sarav@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:7291609B-89C9-4F2C-A452-977112654FE1@xxxxxxxxxxxxxxxx
> Hi Experts,
>
> I am having a peculiar issue in VB while subtracting two number in
> the range 0.1,0.2 & 0.3.
>
> Let us have an example. Goto Immediate window give the command like
> this.
> ?10 - 9.9 obviously we expect the result as 0.1. But what VB returns is
> like
> this : 9.99999999999996E-02. For all other number values it returns good.
> I
> know upon using the Round() function I can get 0.1. But I want to know Why
> VB
> behaves like this especially for the results like 0.1 and 0.2.
>
> If anybody does know the answer, please post it here.
>
> -Sarav

The answer is... it's not a bug. It directly relates to the way PCs store
floating point numbers.

See "What Every Computer Scientist Should Know About Floating-Point
Arithmetic"
http://cch.loria.fr/documentation/IEEE754/ACM/goldberg.pdf

INFO: Visual Basic and Arithmetic Precision
http://support.microsoft.com/default.aspx?scid=kb;en-us;279755

--
Ken Halter - MS-MVP-VB - http://www.vbsight.com
DLL Hell problems? Try ComGuard - http://www.vbsight.com/ComGuard.htm
Please keep all discussions in the groups..


.

Relevant Pages

  • Re: Subtraction Bug in VB
    ... Goto Immediate window give the command like ... > floating point numbers. ... Whether you want to call it a bug or a fault, it is a deficiency and is ...
    (microsoft.public.vb.bugs)
  • Re: Problem with reading a Data File
    ... The last command display the data in rectangular boxes and different ... Symbols but doesnot display floating point numbers. ...
    (comp.soft-sys.matlab)
  • Re: Subtraction Bug in VB
    ... must be right up there in the top ten FAQ;-) ... It directly relates to the way PCs store ... > floating point numbers. ... Visual Basic and Arithmetic Precision ...
    (microsoft.public.vb.bugs)
  • Re: Need help guys
    ... floating above the model. ... What command do I use to connect the surface to the model? ... Considering it is a curved surface so I cannot extrude it conventionally ...
    (comp.cad.solidworks)
  • Problem with reading a Data File
    ... is in Binary format because it doesnot work with Load command. ... The last command display the data in rectangular boxes and different ... Symbols but doesnot display floating point numbers. ...
    (comp.soft-sys.matlab)