Re: scwewy answers that don't belong

Tech-Archive recommends: Fix windows errors by optimizing your registry

"SteveDB1" wrote:

Hopefully MS will actually resolve this one day.

Actually, this indicates that you have not understood what you have read.
For example, consider =12.22-12.2-0.02 which mathematically is zero, but
numerically returns 1.35E-15 in Excel and almost all other software. There
is nothing wrong with the math that is performed here, but none of the three
numbers involved have exact binary representations, and hence must be
approximated. The decimal representations of the best 53-bit binary
approximations (dictated by the IEEE 754 standard that is followed by almost
all software and hardware) to these three numbers are
12.2200000000000006394884621840901672840118408203125
-12.199999999999999289457264239899814128875732421875
-0.0200000000000000011102230246251565404236316680908203125
If you do the math, you will see that the answer of 1.35E-15 is actually
correct, given the unavoidable initial approximations, and not "skewed" in
any way. It is because the math is correct that rounding is the appropriate
solution, but the appropriate level of rounding depends on the particular
calculation, and hence is not something that MS can do for you.

I also tried to test one of the issues that JW Lewis quandried about in the
same article.
I.e., 1-1/3-2/3 and 1-2/3-1/3. I got an absolute zero on both of mine. I

As you should. If you read that post more carefully, you will see that my
example applied to a hypothetical decimal computer, not to Excel. I was
illustrating the point that even if computers switched from binary to decimal
they would still not be able to eliminate this issue. It is not
mathematically possible to completely eliminate such issues as because they
are a consequence of finite precision, not merely a particular number base
system.

Jerry
.

Relevant Pages

  • Re: prePEP: Decimal data type
    ... I think it's a good thing that you're forced to round ... >is about rounding rules and it's quite easy to avoid the "fail to meet the ... ISTM these things are 'way beyond the concern of a math package per se, ... for the meaning of "context" in this PEP context, ...
    (comp.lang.python)
  • Re: [ANNOUNCE/RFC] Really Fair Scheduler
    ... surprised why you focus on rounding so much (and why you attack CFS's ... math implementation so vehemently and IMO so unfairly) - and we had this ... discussion before in the "CFS review" thread that you started. ... trigger these problems due to the rounding problems. ...
    (Linux-Kernel)
  • Re: [ANNOUNCE/RFC] Really Fair Scheduler
    ... surprised why you focus on rounding so much (and why you attack CFS's ... The main reason i'm interested in changing the fairness math under CFS ... Now your patch does not actually do it that way in a clearly discernible ...
    (Linux-Kernel)
  • Re: Problem withe ABS Function
    ... I'll use rounding to resolve my original "problem". ... That was some pretty fancy math you used, Jerry, but it illustrated your point very well. ... Second, other cells in my worksheet that perform the identical calculation, albeit on different ranges of cells, produce the correct result. ...
    (microsoft.public.excel.worksheet.functions)
  • Re: demonic problem descriptions
    ... >> The last company I worked at doing financials was a long distance ... >> direct and third party billing, taxes, commission accounts, ... > rounding to the nearest 1000 anyway. ... write a set of fixed point math functions- add, subtract, multiply, ...
    (comp.lang.lisp)