Re: Precision problem

From: Carl Daniel [VC++ MVP] (cpdaniel_remove_this_and_nospam_at_mvps.org.nospam)
Date: 05/11/04 

Date: Tue, 11 May 2004 15:44:09 -0700

Raghuram wrote:
> Hi Ng,
> I am developing engineering software which is using floating point
> arithmetic widely. I am showing some of the calculated results in
> some spread ***. The spread *** only accepts the string entries
> so that i am converting it to string using gcvt() and sprintf() etc.
> After this some precision loss is happening. In VC debugger it shows
> the exact values but my converted string shows different though it is
> minute but undesirable. How the VC debugger shows it correctly? How
> can I get the exact value as VC++ debugger?
> I would appreciate any kind of help

More likely, the VC debugger is rounding the result to a value that happens
to correspond to your expectations, while sprintf is converting the value
with higher precision, thus yielding a slightly different value that doesn't
meet your expectations.

See http://docs.sun.com/source/806-3568/ncg_goldberg.html for lots of
details. The bottom line is that you probably need to determine the
precision that you need and ensure that your results are computed to that
level of precision and rounded appropriately when converted to a string
representation.

If you can show some code, or determine the exact binary representation of a
floating point value that's turning out different from your expectations,
someone can probably make a more concrete recommendation.

-cd