Re: This calculation is just wrong / computer can't count!

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance


GT schrieb:

[snip]
0.83333333333333337

is accurate and CORRECT as a representation of 25/30.

No it isn't, it is an approximation and the approximation (conversion from bin to dec) has introduced a level of error, rendering the number incorrect.

So you finally realise that this error comes from the conversion from bin to dec. Since this conversion does *not* take place during the calculations why do you think it matters at all? The binary value the computer uses is corect, so what do you care about a decimal conversion?
This conversion only takes place when you print out the final result to the screen, and that is totally under *your* control.

NOrbert

.

Relevant Pages

  • Re: error in simple loop !!
    ... This is closely related to your DO loop problem and understanding it is ... approximation available in the particular binary representation being ... That conversion ...
    (comp.lang.fortran)
  • Re: Float BIN to String Conversion in PL/I
    ... You stated in another post that you are using bin float. ... converted to charusing the bfl2chr subroutine given below. ... That's the best you can hope for; it is impossible to get 18 decimal digits of accuracy out of a system of representation that is only accurate to 15.955 decimal digits. ... Even this is really just a coincidence and also depends on the fact that PL/I's input conversion routine rounds the intermediate result in converting 1.212345678900000e0 to bin float. ...
    (comp.lang.pl1)
  • .ELF to .BIN conversion is working for one set of source code , but it is not working with another s
    ... ..ELF to .BIN conversion is working for one set of source code, ...
    (codewarrior.embedded)
  • bitwise XOR calc
    ... I need to calculate the XOR of three values in excel. ... The conversion to bin = ...
    (microsoft.public.excel.worksheet.functions)
  • Re: Palin?
    ... Unless by that point the Rapture has happened and Bin Laden's done ... conversion. ... YOUR taste at work... ...
    (rec.arts.movies.current-films)