Re: Error in precision with number operations

From: Chris Barber (chris_at_blue-canoe.co.uk.NOSPAM)
Date: 03/05/04 

Date: Fri, 5 Mar 2004 22:30:12 -0000

See:
http://www.merlyn.demon.co.uk/js-round.htm

for really in-depth info on floating point arithmetic and rounding issues.

Chris.

"Jon" <spatxun@hotmail.com> wrote in message
news:u1RNYhsAEHA.1604@TK2MSFTNGP11.phx.gbl...
Hi, all

I´ve a problem with some number operations in VBScript::
10.11 - 10.10 = 9,99999999999979E-03
and the real result is 0.01
żAny idea?
żAny solution, patch, service pack, etc?
I am running the script in Intenet Explorer 6.0.

Thanks

Relevant Pages

  • Re: Why not FP for Money?
    ... I had started with the assumption that the IEEE 754 rounding methods ... accounting has rules about rounding, ... Floating point arithmetic is useful for continuous quantities, ... applies here if you are dealing with actual pennies. ...
    (comp.lang.python)
  • Re: converting float to double
    ... Although I can not see a guarantee from your floating ... rounding, it does not generalise to, e.g., ...
    (comp.lang.c)
  • Re: Bug in Delphi SimpleRoundTo function?
    ... calculations also use bankers rounding. ... rounding rule that most programmers can agree on. ... the trap of using floating *binary* point numbers, ... to represent decimal fraction ...
    (borland.public.delphi.non-technical)
  • Cfv: Floating point truncation
    ... RfD: Floating point truncation ... which, respectively, provide rounding to the nearest integer, and rounding to the nearest integer on the side of the number line towards negative infinity. ... ANS Forth does not provide a word for rounding towards zero, and returning a floating point representation of the resulting integer. ... Include a standard word, named "FTRUNC", to perform "truncation", i.e. rounding of a floating point number towards zero, returning a floating point representation of the resulting integer. ...
    (comp.lang.forth)
  • Re: converting float to double
    ... > the floating point value. ... The "rounding" required to ... What algorithm should be implemented? ... If accounting rules so specifiy, ...
    (comp.lang.c)
Loading