Re: Rounding errors

Some of this is rounding error, but what version of Access did you try?

Access 2007 gives these results:

a) In the Immediate Window:
? 1/.165/203650
2.97599119106607E-05

b) In the Immediate Window:
? CDec(1) / CDec(0.165) / CDec(203650)
0.0000297599119106607444441964

c) In this query:
SELECT 1/0.165/203650 AS MyResult;
0.00002975991191066074444

d) IN this query:
SELECT CDbl(1)/CDbl(0.165)/CDbl(203650) AS MyResult;
2.97599119106607E-05

(a) and (d) are clearly working with the data type Double - same as your field type. And Excel 2007 gives the same result.

(b) and (c) are treating the data as Decimal, and so gives greater precision.

I was actually surprised to see JET using Decimal for (c), but it reports the query's field as type dbDecimal. It doesn't always do that. If you try:
SELECT 1/2/3 AS MyResult;
it creates a field of type Double.

If you need further description of the rounding accuracy issue inherent in floating point numbers, see:
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

--
Allen Browne - Microsoft MVP. Perth, Western Australia
Tips for Access users - http://allenbrowne.com/tips.html
Reply to group, rather than allenbrowne at mvps dot org.

"JimBlob" <JimBlob@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:6F636C1C-5347-47E7-8DD8-69BF307B615E@xxxxxxxxxxxxxxxx
I have been trying to create a query in Access, that divides three numbers
For Example: 1/.165/203650. and have been unsuccessful in matching the
output to match what I get in either Excel, or in windows Calc beyond the
12th decimal place. Is this rounding error?

2.975991191066111E-05 (Access)
2.975991191066070000E-05 (Excel)
2.9759911910660744444196445178522e-5(calc)

I have the field set as double, 15 decimal places.

Thank you in advance!
Jim



.

Loading