Re: This calculation is just wrong / computer can't count!

On Mon, 8 Oct 2007 10:37:22 +0100, "GT"
<ContactGT_remove_@xxxxxxxxxxx> wrote:

What is 25/30?

25/30 = 5/6 and that is as accurate an answer as you can get. It is
EXACT. The decimal number you get from a calculator is only an
approximation. The math in calculators is carried out to some number
of digits but it displays far fewer digits than it calculates, this
makes life simple for people like you.

Use CString::Format("%15f", result); to display your number.

Relevant Pages

  • Re: .99999... still=/= 1
    ... > A scientific calculator does approximate convergence with a certain ... > number significant digits. ... i.e. any partial sum, or do you mean the whole series, which is defined ... The expression you wrote with it stopping after the mth term ...
  • Re: Thatll Teach Em - Long Division
    ... if only so you are not totally stumped if your calculator fails. ... I remember a teacher when I was at school in the 1970s whose standard ... punishment for any misdemeanour was "cubes". ... boys to call out digits (the number of digits depended on the severity of ...
  • HP30S numerical anomaly
    ... I posted about a peculiar HP30S behavior over on MoHP, ... Recently while looking at the calculator forensics web site, ... functions can give substantially less than 24 digits; ...
  • Re: Math errors in python
    ... They don't even share three digits beyond the decimal point. ... > yet not all fractions can be represented by decimals. ... Hardware designers have settled on a binary representation of floating ... And BTW, your calculator is not, in general, more accurate than the ...
  • Re: FAQ 4.2 Why is int() broken?
    ... is much of an issue - 8 or 12 digits are enough. ... For example on my HP-48 calculator. ... can display 0.999999999999 exactly, ...