Re: How to overcome very slight calculation error.
- From: "Albert D. Kallal" <PleaseNOOOsPAMmkallal@xxxxxxx>
- Date: Mon, 16 Oct 2006 01:13:06 -0600
Welcome to the world of computers.
It turns out that one of the FIRST things a programmer will be taught is
that when a computer stores a "real" number (as opposed to a integer
number), the computer can only store a approximation of that number.
so, when you store a value such as 3 / 10, the resulting binary
representation of .3 is only a approximation of that result.
The following code if run in excel, word, or ms-access will produce the same
rounding errors:
Public Sub TestAdd()
Dim MyNumber As Single
Dim i As Integer
For i = 1 To 10
MyNumber = MyNumber + 1.01
Debug.Print MyNumber
Next i
End Sub
Here is the actual outpput of the above:
1.01
2.02
3.03
4.04
5.05
6.06
7.070001
8.080001
9.090001
10.1
You can see that after just 7 addtions..already rounding is occuring
and if we add the follwing line of code to the end of the above:
if MyNumber = 10.1 = True then
msgbox "the number is 10.1"
else
msgbox "the number is somthing else"
endif
The above will actuall produce:
the number is something else
So, what this means is that if you plan to use any financial calculations,
you can't store, or nor use "real" numbers, as they are a approximates. The
solution in this case is to use decimal number, or a data type of currency.
(these are stored as scaled integers).
So, if you are working with financial numbers, and don't want rounding
errors, then use currency.
further, you don't say what data type "c" is.
I just tried:
Dim c As Single
c = 0.175
Debug.Print c * 1325
the output was:
231.875
In fact, in the debug window, I just typed in:
? 0.175 * 1325
The reuslt is
231.875
However, if I type in:
? (0.175 * 1325) = (231.875 )
output is:
false
Howeer:
? ccur(0.175 * 1325.0) = ccur(231.875)
true
So, just use currency for these finical calculations. (they have a max of 4
digits after the decimal place, and are actually scaled integers).
If that data is from a table, you should convert that column to currency, or
perhaps decimal type.
I would also ensure that you always use a option explicit in your code....
--
Albert D. Kallal (Access MVP)
Edmonton, Alberta Canada
pleaseNOOSpamKallal@xxxxxxx
.
- Prev by Date: Re: Check Box
- Next by Date: Re: How to overcome very slight calculation error.
- Previous by thread: Re: Check Box
- Next by thread: Re: How to overcome very slight calculation error.
- Index(es):
Relevant Pages
|
