Re: Precision problem

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

From: Carl Daniel [VC++ MVP] (cpdaniel_remove_this_and_nospam_at_mvps.org.nospam)
Date: 05/11/04 

Date: Tue, 11 May 2004 15:44:09 -0700

Raghuram wrote:
> Hi Ng,
> I am developing engineering software which is using floating point
> arithmetic widely. I am showing some of the calculated results in
> some spread sheet. The spread sheet only accepts the string entries
> so that i am converting it to string using gcvt() and sprintf() etc.
> After this some precision loss is happening. In VC debugger it shows
> the exact values but my converted string shows different though it is
> minute but undesirable. How the VC debugger shows it correctly? How
> can I get the exact value as VC++ debugger?
> I would appreciate any kind of help

More likely, the VC debugger is rounding the result to a value that happens
to correspond to your expectations, while sprintf is converting the value
with higher precision, thus yielding a slightly different value that doesn't
meet your expectations.

See http://docs.sun.com/source/806-3568/ncg_goldberg.html for lots of
details. The bottom line is that you probably need to determine the
precision that you need and ensure that your results are computed to that
level of precision and rounded appropriately when converted to a string
representation.

If you can show some code, or determine the exact binary representation of a
floating point value that's turning out different from your expectations,
someone can probably make a more concrete recommendation.

-cd

Relevant Pages

  • Re: Precision problem
    ... > so that i am converting it to string using gcvtand sprintfetc. ... > After this some precision loss is happening. ... floating point value that's turning out different from your expectations, ...
    (microsoft.public.vc.mfc)
  • Re: Calculate the precision of a floating point number (ie: the number of decimal places)
    ... I did not think you wanted the precision (since with internal floating ... binary floating point). ... string as if you were going to print it out. ...
    (comp.programming)
  • Integers have docstring for int()
    ... docstrings that describe the functions to create them. ... A floating point\nargument will be truncated towards zero (this does not i ... string, use\nthe optional base. ... It is an error to supply a base when converting ...
    (comp.lang.python)
  • Re: Floating point arithmetic support in DCL
    ... >> However, why all this stuff about holding the floating point value as a string, ... > converted to a string for display or concatenation to a string etc. ... Converting from latitude to ...
    (comp.os.vms)
  • Re: How to calculate days:Hours:Minutes:
    ... This syntax makes it SINGLE precision. ... > Dim s1 As String ... >Dana DeLouis ... It stops converting when it hits a character (like the D in ...
    (microsoft.public.excel.worksheet.functions)