Re: Rounding error while saving a recordset as XML
- From: "Stephen Howe" <stephenPOINThoweATtns-globalPOINTcom>
- Date: Fri, 22 Jun 2007 14:28:48 +0100
Hi All,
I am facing one rounding error while saving a recordset as XML. Is there
a way to get rid of this error without altering the datatype of QTY field
to string/variant.
No. Because you not understanding how floating-point works.
You know that the decimal fraction 2/3 is 0.66666666666666... in base 10,
forever. It is said to recur.
And no matter where the fraction is chopped, it will never represent 2/3
The only fractions that are exact in base 10 are multiples of powers of 1/5
and 1/2 combined because 5 and 2 are divisors of 10.
So 3/40, 7/25, 9/16 are exact but 1/7, 1/13 are not.
Well floating-point uses base 2 internally,
And in base 2, only fractions that are powers of 1/2 are exact, all others
recur.
In particular 0.1 and 0.01 will never have an exact representation, they are
not multiples of powers of 1/2
So that means there is no exact 3.01 representation, when you convert from
text to double, it will choose the nearest binary fp number and when you
convert back, this what you see.
Programmers usually go through shock then rage then depression then
realisation on how to work with fp.
You should leave the XML text as it is. When it is read back and converted
to a double, you can output to 2 decimal places, rounded and at that point,
you will see 3.01
When manipulating fp numbers, adding, subtracting, mutiplying - all effort
should bent on minimising scaling error.
Stephen Howe
.
- References:
- Prev by Date: Rounding error while saving a recordset as XML
- Next by Date: Re: appendChunk method resulting in incorrect data
- Previous by thread: Rounding error while saving a recordset as XML
- Index(es):
Relevant Pages
|
|
