Re: Float value changes unexpectedly when performing math on field
- From: "Russell Fields" <russellfields@xxxxxxxxxx>
- Date: Fri, 15 May 2009 21:22:00 -0400
AkAlan,
Float fields, by definition, have some built-in imprecision. This is a side effect being able to store very large numbers in smaller spaces. Even though you might expect that imprecision on very large numbers, it happens on small numbers as well. (It is all part of the representation of numbers.)
So, ideally, if that is not what you want, you should NUMERIC or DECIMAL, but not float. (Although, I am getting 100.9 from a float when I enter 100.9.)
Having said that though, you can get the display that you want by:
SELECT FloatColumn AS A, ROUND(FloatColumn) AS B
FROM table
Should give you:
A B
-------------- ---------------
100.8999999 100.9
Hope that it helps.
RLF
"AkAlan" <AkAlan@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message news:B4B97F70-F19F-4C1B-9B3D-60ABEC65B949@xxxxxxxxxxxxxxxx
I have a Float field that stores a meter reading. The actual meter can
rollover to zero so I have a rollover count and rollover point so I can
display the reading as the user sees it, but still store the total number of
hours the equipment has operated. The problem I'm having on one meter reading
is, the user inputs 100.9, the table stores 100.9 but the sp displays
100.8999999. Here is the select statement I have in the sp:
CASE WHEN [RolloverCount] > 0 THEN [MeterReading] - ([RolloverPoint] *
[RolloverCount]) ELSE [MeterReading] END
What can I do differently so the correct value is returned?
.
- References:
- Prev by Date: Re: Passing a parameter to a TRIGGER
- Next by Date: Re: Passing a parameter to a TRIGGER
- Previous by thread: Float value changes unexpectedly when performing math on field
- Next by thread: Passing a parameter to a TRIGGER
- Index(es):
Relevant Pages
|
