Re: Rounding errors

Im using 2002. My original query was: SELECT
[Frequency]/[bindiff]/[SumOfFrequency] AS Expr1
FROM tblData INNER JOIN qryFrequencySum ON tblData.MeasurementID =
qryFrequencySum.MeasurementID;

Using Cdbl worked.

Thank You!


"Allen Browne" wrote:

Some of this is rounding error, but what version of Access did you try?

Access 2007 gives these results:

a) In the Immediate Window:
? 1/.165/203650
2.97599119106607E-05

b) In the Immediate Window:
? CDec(1) / CDec(0.165) / CDec(203650)
0.0000297599119106607444441964

c) In this query:
SELECT 1/0.165/203650 AS MyResult;
0.00002975991191066074444

d) IN this query:
SELECT CDbl(1)/CDbl(0.165)/CDbl(203650) AS MyResult;
2.97599119106607E-05

(a) and (d) are clearly working with the data type Double - same as your
field type. And Excel 2007 gives the same result.

(b) and (c) are treating the data as Decimal, and so gives greater
precision.

I was actually surprised to see JET using Decimal for (c), but it reports
the query's field as type dbDecimal. It doesn't always do that. If you try:
SELECT 1/2/3 AS MyResult;
it creates a field of type Double.

If you need further description of the rounding accuracy issue inherent in
floating point numbers, see:
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

--
Allen Browne - Microsoft MVP. Perth, Western Australia
Tips for Access users - http://allenbrowne.com/tips.html
Reply to group, rather than allenbrowne at mvps dot org.

"JimBlob" <JimBlob@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:6F636C1C-5347-47E7-8DD8-69BF307B615E@xxxxxxxxxxxxxxxx
I have been trying to create a query in Access, that divides three numbers
For Example: 1/.165/203650. and have been unsuccessful in matching the
output to match what I get in either Excel, or in windows Calc beyond the
12th decimal place. Is this rounding error?

2.975991191066111E-05 (Access)
2.975991191066070000E-05 (Excel)
2.9759911910660744444196445178522e-5(calc)

I have the field set as double, 15 decimal places.

Thank you in advance!
Jim




.

Relevant Pages

  • Re: Problem with fConcatChild()
    ... If I change the query to a Make Table query and make a table exactly like ... Allen Browne - Microsoft MVP. ... That is what I got before when I captured ths sql string with a MsgBox. ... open the Immediate Window. ...
    (microsoft.public.access.formscoding)
  • Re: need help creating criteria for time
    ... Allen Browne - Microsoft MVP. ... I then removed the criteria and left everything else and ran it ... Now let's verify that these date/time fields have *only* date or time, ... If you switch your query to SQL View (View menu, ...
    (microsoft.public.access.queries)
  • Re: Problem with Send Object
    ... Inserted into criteria of report query: ... How are you passing that filter string to the report, ... Allen Browne - Microsoft MVP. ...
    (microsoft.public.access.formscoding)
  • Re: >> Current Items only
    ... I didn't get the bit about needing a UNION query. ... Allen Browne - Microsoft MVP. ... Assuming a RowSource like this: ... To do this I have the combobox recordsource listing all items. ...
    (microsoft.public.access.formscoding)
  • Re: Please help!
    ... From the 2nd row down to the last is the same duplicate ... Allen Browne - Microsoft MVP. ... In a query, depress the Total button on the Toolbar. ... Access adds a Total row to the grid. ...
    (microsoft.public.access.forms)