Re: Strange BUG in teh Framework?
From: Jay B. Harlow [MVP - Outlook] (Jay_Harlow_MVP_at_msn.com)
Date: 10/11/04
- Next message: csmba: "double/Single BUG?"
- Previous message: Joerg Jooss: "Re: Using Httpwebrequest to Submit multipart/form-data"
- In reply to: David: "Strange BUG in teh Framework?"
- Next in thread: Jon Skeet [C# MVP]: "Re: Strange BUG in teh Framework?"
- Messages sorted by: [ date ] [ thread ]
Date: Mon, 11 Oct 2004 15:16:37 -0500
David,
I don't believe its a bug per se, its due to the inexact nature of how
floating point numbers are stored.
If you change xSng to a Double or Decimal, you get the correct result.
For a complete discussion (and explaination) of how floating point numbers
work in .NET see:
http://www.yoda.arachsys.com/csharp/floatingpoint.html
http://www.yoda.arachsys.com/csharp/decimal.html
Hope this helps
Jay
"David" <David.Konig@nowhere.com> wrote in message
news:epx1At8rEHA.3964@TK2MSFTNGP10.phx.gbl...
> hello.
> when doing the simple following computation, the value put into the
> variable
> numMinusOne is NOT the same as what the computation is showed to be in the
> Watch window!!
> here is the code:
> Dim xSng As Single = 6547.972
> Dim yInt As Integer = 8000
> Dim num As Integer = CInt(Math.Floor(xSng * yInt))
> Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1)
>
> After running these 4 lines, numMinusOne will be:52383776
> while the Watch window will claim that the value of :
> CInt(Math.Floor(xSng * yInt) - 1) = 52383775
>
> BTW, the Watch window is correct.
>
> how come the numMinusOne does not get the correct value?
>
>
- Next message: csmba: "double/Single BUG?"
- Previous message: Joerg Jooss: "Re: Using Httpwebrequest to Submit multipart/form-data"
- In reply to: David: "Strange BUG in teh Framework?"
- Next in thread: Jon Skeet [C# MVP]: "Re: Strange BUG in teh Framework?"
- Messages sorted by: [ date ] [ thread ]
Relevant Pages
|
