Re: Error in CF Decimal-type calculations?

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

From: Darren Shaffer (darrenshaffer_at_discussions.microsoft.com)
Date: 12/14/04 

Date: Tue, 14 Dec 2004 08:16:26 -0700

Dave,

I was able to recreate the issue and the only workaround I found was to
either do the whole
problem using floating point types or to cast to double before using the
128-bit number as the
denominator of the inversion. Just out of curiosity, what sort of
application are you building that
has to have 128-bit precision? Sounds like it might be a really interesting
use of mobile technology.

-Darren

"DAve" <dave@my.home.uk> wrote in message
news:emQtp1b4EHA.1408@TK2MSFTNGP10.phx.gbl...
> Thanks for the idea, but really - if we were happy with the accuracy of
> the
> Double type, then we'd be using that instead of Decimal!
>
> If there were someway to know when the Decimal type is going to go wrong,
> then we could perhaps just use Double then. Do you know how to decide if
> the problem is going to occur in Decimal?
>
> For instance, the problem occurs when trying to calculate
> 1/0.0010416666666666666666666667 but not for 1/0.0010416666666666666667.
>
> Many thanks for any more ideas!
>
> "Darren Shaffer" <darrenshaffer@discussions.microsoft.com> wrote in
> message
> news:euQC0QY4EHA.2452@TK2MSFTNGP14.phx.gbl...
>> Dave,
>>
>> Replace your last line with:
>>
>> double r = 1/(double)d;
>> MessageBox.Show("inverting again yields: " + r.ToString()); // 60
>>
>> -Darren Shaffer
>>
>>
>>
>> "DAve" <dave@my.home.uk> wrote in message
>> news:%23ZKKYCT4EHA.2404@TK2MSFTNGP14.phx.gbl...
>> > Hi -
>> >
>> > Anyone seen this? Using the 'Decimal' type to do calculations in the
>> > Compact Framework, some calculations are going wrong ...
>> >
>> > For instance, running the following three lines should give "60" but
>> > instead
>> > gives "41.43..." !!
>> >
>> > Dim k as Integer = 60
>> > Dim d as Decimal = 1 / New Decimal(k)
>> > MessageBox.Show(1 / d)
>> >
>> > Many other values for 'k' also seem to go wrong (eg 60, 70, 77, 91, 92,
>> > ....).
>> >
>> > Any help greatly appreciated!
>> >
>> >
>> >
>>
>>
>
>

Relevant Pages

  • Floating Point Accuracy Issue
    ... Im a little rusty regarding floating point accuracy, ... by a maximum tolerance of 3.5, rounded up and then cast back to unsigned ... cast to a DWORD cycles the result back to 0 as its 1 number higher than can ... const double dMaxTol = 3.5; ...
    (microsoft.public.vc.language)
  • Re: Known bugs in 3.2.2
    ... There's a workaround described ... I thought this was amusing: ... Step 1: Cast a glyphed Power Word: Shield on a target ...
    (alt.games.warcraft)
  • Re: allocating vectors in a function
    ... Don't cast the result of calloc(or malloc(), ... Any unnecessary cast is bad style, ... Don't use calloc() for floating point values in the first place. ...
    (comp.lang.c)
  • Re: invoke problem
    ... > I did this workaround, ... > no general way to pass pointer vales of the arbitrary type to the invoke? ... First the invoke forses me to cast arguments to the Object* and loose ... I do think it might be a bug, ...
    (microsoft.public.dotnet.languages.vc)
  • Re: POLLING
    ... the bread they have cast on the waters comes floating back to them, ...
    (alt.true-crime)