Re: Strange rounding issue in formatted number field



On 3/21/08 8:39 AM, in article ee95931.-1@xxxxxxxxxxxxxxxxxxxxxxx,
"pattera5@xxxxxxxxxxxxxxxx" <pattera5@xxxxxxxxxxxxxxxx> wrote:

Version: 2008
Operating System: Mac OS X 10.5 (Leopard)
Processor: intel

I have a cell formatted as a number with 13 decimal places, I have a formula
in there which returns a bizarre result.

It can be easily replicated by putting the following in the cell
=4935.48-4904.68

The result I get is 30.7999999999993. This is simple subtraction, where is
this rounding issue coming from? any ideas?
This is an artifact of the conversion of binary to decimal and vice versa in
accordance with industry accepted standards. This is just the way Excel (and
almost every other program) works. Here is a tutorial to give you much more
information than you ever wanted to know:
http://support.microsoft.com/kb/42980/en-us

--
Bob Greenblatt [MVP], Macintosh
bobgreenblattATmsnDOTcom

.



Relevant Pages

  • Re: If then Statements
    ... Operating System: Mac OS X 10.5 (Leopard) ... Processor: intel ...
    (microsoft.public.mac.office.excel)
  • GRAPHICS
    ... Operating System: Mac OS X 10.5 (Leopard) ... Processor: intel ...
    (microsoft.public.mac.office.word)
  • converting word list to excel data merge document
    ... Operating System: Mac OS X 10.5 (Leopard) ... Processor: intel ...
    (microsoft.public.mac.office.excel)
  • Re: Files cant be read
    ... Operating System: ... After installing the 11.4.1 ... Mac Desktop (2 x 3 GHz Quad-Core Intel; ...
    (microsoft.public.mac.office.excel)
  • My Day Problems
    ... Operating System: Mac OS X 10.5 (Leopard) ... Processor: intel ...
    (microsoft.public.mac.office.entourage)