RE: When double precision isn't very precise
- From: Jeff Dillingham <Jeff Dillingham@xxxxxxxxxxxxxxxxxxxxxxxxx>
- Date: Fri, 29 May 2009 18:00:00 -0700
Peter:
While the response from other writter is correct in advising that you should
not be using floating point calculations for financial analysis, you are also
correct that there is a bug in the floating point arithmetic in VB.Net. The
symptom is that when you add two double precision variables together and
assign the result to another double precision variable, the result may only
be accurate to single precision. The remaining digits are random. This
behavior does not occur all of the time however. It seems to depend on what
has preceeded the calculation. I have seen this in my own code under
repeatable conditions, and it has been verified by a co-worker who
encountered the same problem independently in a different program.
The code below is a simple subroutine that will test whether the error is
occurring. Just call this from anywhere in your program. I find that
sometimes this test routine passes, and sometimes it fails and traps an error
with the message box. I have not determined any way to determine if/when the
error will occur. This is truely a serious problem that makes VB inadequate
for certain types of engineering calculations where double precision is
important. I have tried converting some my code to use the Decimal data
type, but that actually made the problem worse. It is not practical to
convert my entire program to Decimal, so it is an unresolved problem at this
point.
Oddly, the problem seems related to the assignment of the value to a
variable. If you set a break point in this routine and inspect the
expression in the watch window it will be correct, but the value of the
variable that the expression is assigned to is not correct.
Public Shared Sub TestAddition()
Dim dummy1 As Double, dummy2 As Double, dummy3 As Double
Dim dummy4 As Double
Dim dummy5 As Double
dummy1 = 1.0 / 3.0
dummy2 = 1.0
dummy3 = dummy1 + dummy2 'should be 1.3333333333333333, but is
actually 1.3333333730697632
'Debug.Print(dummy3)
dummy4 = 1.0 + 1.0 / 3.0 'this produces the correct answer:
1.3333333333333333; therefore the problem seems to be with adding
double-precision variables, not literals
'Debug.Print(dummy4)
dummy5 = dummy3 - dummy4 'the difference between the correct and
incorrect answers
'Debug.Print(dummy5)
If dummy5 > 0.000000001 Then
MsgBox("round off error in vb")
End If
End Sub
"Peter B. Steiger" wrote:
I wrote a quick-and-dirty accounting routine that added up charges and
payments to show a running balance on someone's ledger. Our QA team
reported that it is producing the wrong balances, and sure enough it was.
The answers were coming back with umpteen decimal places written in
scientific notation, e.g. 1.23456E-9.
I thought I'd play it smart and wrote a function to do all the work in
integers - it receives the running balance and new amount, multiplies
them by 100 into integer variables, adds the two integers, and divides
the result by 100. It's stupid that I have to do this because our state-
of-the-art computer can't add two-place numbers together and come up with
the right answer, but at least it was an easy enough kludge.
Now it turns out it's also converting to integers wrong. For example,
with a balance of $595.17 added to $80.00, the new balance is $675.16
because it multiplied 595.17 * 100 and came up with... 59516.
The balance and amount are both double precision, so I did some testing:
Module FloatTest
Sub Main()
Dim nFloat1 As Single
Dim nFloat2 As Double
Dim nInt1 As Integer
Dim nInt2 As Integer
nFloat1 = 595.17
nInt1 = Int(nFloat1 * 100)
Console.WriteLine("Int of single " & nFloat1 & " * 100 is " & nInt1)
nFloat2 = 595.17
nInt2 = Int(nFloat2 * 100)
Console.WriteLine("Int of double " & nFloat2 & " * 100 is " & nInt2)
End Sub
End Module
.
The results? Multiply a single-precision number with two decimal places
by 100, and you get the right answer (in this case, 59517). Multiply a
double precision number by 100 and you get the wrong answer (59516).
I know FP math produced weird results on old CPUs (8086 through 80486),
but I would have thought for sure some processor genius would have fixed
the problem by now. Is this a known bug, or am I using the wrong data
types for my purposes, or what?
--
Peter B. Steiger
Cheyenne, WY
If you must reply by email, you can reach me by placing zeroes
where you see stars: wypbs.**1 at gmail.com.
- Follow-Ups:
- Re: When double precision isn't very precise
- From: Armin Zingler
- Re: When double precision isn't very precise
- Prev by Date: MouseWheel Event: Focusing problem
- Next by Date: Re: When double precision isn't very precise
- Previous by thread: MouseWheel Event: Focusing problem
- Next by thread: Re: When double precision isn't very precise
- Index(es):
Relevant Pages
|
