Re: Pixel Mania

"Rick Rothstein (MVP - VB)" <rickNOSPAMnews@xxxxxxxxxxxxxxxxx> wrote in
message news:OUVJ3ItiHHA.4516@xxxxxxxxxxxxxxxxxxxxxxx
Remember, Banker's Rounding (round-to-even) only comes into play when the
number ends exactly in 5 (only zeroes after it is fine) and your are
rounding up to the position in front of the 5... that will never be the
case with Pi as it is an irrational, transcendental number which has an
infinite number of decimal digits following the decimal point.

That is true.

I'm not so sure Banker's Rounding comes into play here either. The 16 bit
truncated number that VB rounds to 15 displayed digits is an integer power
of two (I think it is only considered as a decimal if the power of ten is
negative), so I don't think it can end in exactly 5.

It can. The number 5 ends in a 5. So does 0.5.

But even if it did, I don't think rounding in the 15th digit would screw
up anyone's calculations. The problem is when VB applies the Banker's
Rounding in situations one doesn't expect. One example of this is 'integer
division'... most people would think A\B means Int(A\B)... but it doesn't.
VB interprets it as Int(Round(A)/Round(B)).

Most people would expect it to round first IMO. It's integer division. In
all my time I've never even considered using it with non integers.

Michael


.

Relevant Pages

  • Re: Deviation of double
    ... int iDigits); ... I believe you're confused about rounding. ... round to 90.62 if only two decimal digits are required. ... Why are you expecting 90.625? ...
    (comp.lang.cpp)
  • Re: This calculation is just wrong / computer cant count!
    ... BCD also has the advantage of providing more significant decimal ... arithemtic, AS IMPLEMENTED ON THE PENTIUM FPU, as demonstrated by my Floating Point ... If you can't understand that using binary to represent decimal digits ... properly rounding. ...
    (microsoft.public.vc.mfc)
  • Re: Rounding error when doing float-to-integer
    ... Agree with Glenn here. ... significant figures" step. ... The rounding is to 24 binary digits. ... The "7 decimal digits" bit is only a rough approximation that allows ...
    (comp.lang.fortran)
  • Re: Pixel Mania
    ... truncated number that VB rounds to 15 displayed digits is an integer power ... a double precision floating point number. ... He claimed that banker's rounding would apply there as well, ... surprised me, since floating point formats are industry standards, not specific ...
    (microsoft.public.vb.general.discussion)
  • Re: Arithmetic operation bug
    ... Manoj Chourasia wrote: ... decimal digits are shown, taking rounding into account. ... doesn't round it), ...
    (comp.lang.ruby)
Quantcast