RE: REAL/DOUBLE PRECISION

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

From: Mike Epprecht (SQL MVP) (mike_at_epprecht.net)
Date: 12/30/04 

Date: Thu, 30 Dec 2004 02:53:02 -0800

Hi

Real and float are approximate data types (not all vaules can be represented
correctly)

For true accuracy, use decimal/numeric (10 ^ 38 -1 ) and when working with
money, use the money data type (2 ^ 63 -1) if it is big enough (should be,
our GDP does not exceed the money data type yet)

Regards
Mike

"Justus" wrote:

> Hi
>
> We had some problems with the precision of large incomes in a swiss social
> security application
> using SQL Server 2000.
>
>
>
> So I did some tests:
>
> SQL SERVER 2000 Float Precision
>
> MS-Help says:
>
> REAL: FLOAT(1) .. FLOAT(24)
> 4 - Byte Float with 7 - digit (decimal?) mantissa
>
> DOUBLE PRECISIOIN: FLOAT(25) .. FLOAT(53)
> 8 - Byte Float with 15 - digit (decimal?) mantissa
>
> ================================================================================
>
> UPDATE TABLE1
> SET ATTRIBUTE = ...
>
> SELECT TABLE1
> FROM ATTRIBUTE
>
> --------------------------------------------------------------------------------
>
> UPDATE QUERYANALYZER ENTERPRISEMANAGER
>
> 99.99 {04} -> {04} 99.989998 99.99
> 99999.99 {07} -> {07} 99999.992 99999.99
> 100000.01 {08} -> {07} 100000.01 100000
> 100000.85 {08} -> {07} 100000.85 100000.9
> 100000.99 {08} -> {07} 100000.99 100001
> 999999.99 {08} -> {07} 1000000.0 1000000
> 3141592.65 {09} -> {07} 3141592.8 3141593
> 31415926.53 {10} -> {07} 3.1415926E+7 3.141593E+07
> (31415926) (31415930)
> 3141592653.59 {12} -> {07} 3.1415926E+9 3.141593E+09
> (3141592600) (3141593000)
> 3141592 {07} -> {07} 3141592.0 3141592
> 31415926 {08} -> {07} 3.1415926E+7 3.141593E+07
> 314159265359 {12} -> {07} 3.1415927E+11 3.141593E+11
>
> -------------------------------------------------------------------------------
>
> 99.234564 (8) -> fault ?! 99.234566 99.23457
>
> ================================================================================
>
>
> I did some reading on IEEE Float, too.
> It seems, that behind SQL-Server Float is an IEEE Float representation. Is it?
> If yes, why is the 'precision boundary' on a decimal value like 99999.99 ->
> 100000.01?
> Is this (99.234564 -> 99.23457) a bug or do I misunderstand something?
>
>
> Kind Regards
>
> Justus Gadient
>
>

Relevant Pages

  • Convert from unsigned char array to float!
    ... I need to convert from a unsigned char array to a float. ... Linuxon a Intel pentium machine. ... Regards, ...
    (comp.os.linux.development.apps)
  • Re: sagulator for glass?
    ... but for shelving purposes, almost all float is interchangable. ...
    (rec.woodworking)
  • Re: float or doubles twos complement representation.
    ... byte format and reconstruct the same to float. ... store it into a byte array). ... you MUST make decoding exactly reverse ...
    (comp.lang.java.programmer)
  • Re: Float to string and vice versa
    ... There are an infinite number of values between .9999 and .99999 and with ... float or double, there are only a finite number of choices to represent them ... > CString strComfortValue; ...
    (microsoft.public.vc.mfc)
  • Re: Convert float to double - weird failure
    ... bill robertson wrote: ... Float 1.2 is uninterpreted hex 0x3f99999a ... It's converting the float representation of 1.2 which is 0x3f99999a to a double representation which is not even 0x3f99999a00000000. ... unsigned int mantissa:23; ...
    (comp.programming)