Re: Simple Math Question

Tech Tip: Click here to run a free scan for Windows Errors and optimize PC performance

From: One Handed Man \( OHM - Terry Burns \) ("One)
Date: 09/20/04 

Date: Mon, 20 Sep 2004 21:46:41 +0100

Thanks for r eplying to me ( tho u didnt ) 

-- 
OHM ( Terry Burns )
 . . .  One-Handed-Man . . .
If U Need My Email ,Ask Me
Time flies when you don't know what you're doing
"Doug" <dallison@newsgroup.nospam> wrote in message
news:27E878AA-1F6C-4C4E-B2BD-0F01A04FBF8C@microsoft.com...
> Thanks. That makes sense. I am dealing with currency values. I will switch
to
> a decimal data type instead. Thanks again.
>
> "Tom Shelton" wrote:
>
> > In article <BE529D99-C9DF-42AF-ADCB-FD6AB9E7D3EC@microsoft.com>, Doug
wrote:
> > > Can someone help me understand this? Given the following code:
> > >
> > > Dim y As Single
> > > y = 1.44
> > > For x As Single = 0 To 9
> > >      Console.WriteLine(x + y)
> > > Next
> > >
> > > Why the strange values when x = 7 and 8? This seems to only happen
when y =
> > > 1.44 or 2.44 though more testing might uncover more. Obviously the
simple fix
> > > is to do a round to 2 places, but I discovered that a little after the
fact.
> > > Is this a bug? Thanks!
> > >
> > > Doug
> >
> > Your dealing with floating point values...  It is not possible to
> > exactly represent all numbers as a floating point value.  The smaller
> > the precision number, the more numbers that can't be represented.  If
> > you need a more exact representation, like dealing with currency values,
> > then you should use a decimal data type.  It is able to handle these
> > precisely, because a decimal is really a scaled 64-bit integer value...
> >
> > -- 
> > Tom Shelton [MVP]
> >

Relevant Pages

  • Re: about the key of map
    ... > Was it FedEx who had the advert for "when it absolutely, positively, has to ... I still don't recommend using a floating point key. ... Doug Harrison ...
    (microsoft.public.vc.stl)
  • Re: 1 <> 1???
    ... It's floating point arithmetic which byit's nature canot be exact. ... Doug has already explained one strategy to get around your problem. ...
    (microsoft.public.access.formscoding)
  • Re: Bahai Faith and Bibliolatry
    ... I am really getting tired of being misunderstood and dealing with this subject. ... Doug does not really believe that. ... understand the significance and meaning of a particular passage. ...
    (soc.religion.bahai)
  • Re: links to royal barge rehearsal
    ... Nice!....Thanks for that Doug, very interesting! ... I like the floating ... grandstand, I hope it's properly ballasted on the night :-)) ... Huay Khwaang ...
    (soc.culture.thai)
  • Re: Are Urban 4x4s cars for posers?
    ... they were saying: ... Didn't bloody offer it me, ... You're dealing with Doug ... ...
    (uk.transport)