Re: Strange BUG in teh Framework?

From: csmba (csmba_at_nowhere.com)
Date: 10/12/04 

Date: Tue, 12 Oct 2004 15:44:57 -0700

yes, this 6547.972 is the length in seconds of a wav file.
and because of the calculation I am using (8000 is the sampling rate), all
that is actually true is the .xxx digits. anything that the double adds to
it like the .xxx167... is just wrong. and was never there to begin with (the
number is a perfect factor of 8000).
so since you said that "actually" the number is stored as 6547.972164... if
I can ignore the digits behind the first 3 I get exactly what I want.... (I
can do it of course by multiplying by 1000, then flooring the number to a
int, then dividing by 1000 again)--> but if I end up storing it ina single,
and he again adds those random 167.. it will do no good.

Decimal is out of the question, I am doing hundred of thousands of such
calculation per file.

while you said it is luck that
dim d as double = 6547.972
works, I find it more then strange. it can not be luck since I tested many
different numbers, all work the same: they all work when set like above. but
some return "strange" values when cast from single to double.
it is as if the problem is in the casting.

as for rounding the float: what I wanted is a way to guarantee that
6547.972 * 8000 gives the right answer. it seems like a fair thing to do. I
am surprised that except for the suggestion of using decimal, I did not find
any real answer to this.
again, it actually does work, as long as the 6547.972 value is stored in a
double or in a single. it FAILS only when it is stored in a single and CAST
to a double as in CDbl(sing)*8000 where sing is a single parameter with
6547.972 as a value.

H.

> > well... I get it from a COM object (BLHaaA)
>
> Oh dear :( On the other hand, it means you've probably done the
> floating point conversion already, so any attempts to make it
> "accurate" again (by using Decimal) are likely to be fruitless. Maybe
> I'm wrong though... it depends on how exactly the COM object is doing
> things, I guess.
>
> > it is a c++ object, that loads a wav file to memory and I need to ask it
> > what is the length of the file. it returns a float.
>
> That's the length of the file in seconds?
>
> > Decimal just seems soooo slow.
>
> Well, it's significantly slower than float - but is this actually the
> bottleneck in your app?
>
> > is there a trick to use a single but tell it to keep only 3 points
> > precision? in that case, I will not have a problem when I multiply by
> > 8000.
>
> No, but you could always round the float afterwards. I'm not sure how I
> see why that's a good thing though, to be honest.
>
> --
> Jon Skeet - <skeet@pobox.com>
> http://www.pobox.com/~skeet
> If replying to the group, please do not mail me too

Relevant Pages

  • Re: Strange BUG in teh Framework?
    ... and because of the calculation I am using, ... some return "strange" values when cast from single to double. ... it returns a float. ...
    (microsoft.public.dotnet.framework)
  • Re: Strange BUG in teh Framework?
    ... and because of the calculation I am using, ... some return "strange" values when cast from single to double. ... it returns a float. ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Richard heathfields casting operation
    ... in this book there is a section named "How and why to cast" ... if you say 3.0 it is a type disambiguation,and the compiler ... If you need a float with the value 3.0f, just say 3.0f - there is no need ... I manually did the calculation for one value by using calculator ...
    (comp.lang.c)
  • Re: Float based math class. Is it exist?
    ... > because the processor was doing a double calculation internally ... > Mr.Jon Skeet, ... > I can't image a situation that double is calculated faters than float. ... float precision math is 2x faster than double. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: internals of typecasting
    ... > object of type int is applied an cast of float the result ... > would be of type float. ... casting from floating-point numbers to integers. ...
    (comp.lang.c)
Loading