floating point precision

Tech Tip: Click here to run a free scan for Windows Errors and optimize PC performance

zak_at_secretlevel.com
Date: 12/15/04 

Date: Tue, 14 Dec 2004 21:31:03 -0800

hello,

i have recently discovered that d3d silently switches the floating point
precision to 24bits. this was giving me undue headaches!! i have my own
temporary solution of setting the floating point precision back to the
default 53bits and then restoring it after using _controlfp(...) function. i
am curious as to why msft does this and if it must be this way. either way i
think it is very important that you document this somewhere public (like in
the sdk documentation!!! ).

thanks,
zak taylor
zak@secretlevel.com

Relevant Pages

  • Re: Precision
    ... Whether one does the subsequent calculations in internal ie integer format ... I have always used external format with Precision 4 since my ... floating point calcs which are done in binary have around 3% error albeit at ... positions it is IMPOSIBLE to have a result of 4 decimals with the 2 ...
    (comp.databases.pick)
  • Re: Linear Algebra Challenge
    ... Since I'm using floating point, so I'll never be able to calculate one ... floating point math set to 99 digits. ... As close as I'm willing to wait if I use arbitrary precision. ... This mode is fast; when you select arbitry ...
    (comp.sys.hp48)
  • Re: Floating point environment
    ... In all functions like fetestexcept, for instance, it is assumed that there is only one floating point environment ... one for double precision ... the 64 bit format and another for the 80 bit format. ... All versions of SSE use a status/control word separate from the main FPU, however, so they can indeed be considered a separate environment. ...
    (comp.std.c)
  • Re: Precision
    ... It's the same if we use Precision, floating point, or ... positions it is IMPOSIBLE to have a result of 4 decimals with the 2 ... May be something wrong in the floating ... remember what the associated actuarial calculations were. ...
    (comp.databases.pick)
  • Re: Precision
    ... The something wrong is that floating point binary was used. ... It's the same if we use Precision, floating point, or ... positions it is IMPOSIBLE to have a result of 4 decimals with the 2 ... We do not use Galleons or Triganic Pus so all accounting uses decimal ...
    (comp.databases.pick)