Re: Strange rounding issue in formatted number field

On 3/21/08 8:39 AM, in article ee95931.-1@xxxxxxxxxxxxxxxxxxxxxxx,
"pattera5@xxxxxxxxxxxxxxxx" <pattera5@xxxxxxxxxxxxxxxx> wrote:

Version: 2008
Operating System: Mac OS X 10.5 (Leopard)
Processor: intel

I have a cell formatted as a number with 13 decimal places, I have a formula
in there which returns a bizarre result.

It can be easily replicated by putting the following in the cell

The result I get is 30.7999999999993. This is simple subtraction, where is
this rounding issue coming from? any ideas?
This is an artifact of the conversion of binary to decimal and vice versa in
accordance with industry accepted standards. This is just the way Excel (and
almost every other program) works. Here is a tutorial to give you much more
information than you ever wanted to know:

Bob Greenblatt [MVP], Macintosh


Relevant Pages