Re: This calculation is just wrong / computer can't count!
- From: ".rhavin grobert" <clqrq@xxxxxxxx>
- Date: Thu, 04 Oct 2007 08:19:48 -0700
On 4 Okt., 17:00, "GT" <ContactGT_remove...@xxxxxxxxxxx> wrote:
".rhavin grobert" <cl...@xxxxxxxx> wrote in message
On 4 Okt., 16:24, "GT" <ContactGT_remove...@xxxxxxxxxxx> wrote:What do you mean by the number might be below double granularity on the
computer? What can I do about this?
your computer stores floatingpoints (float, double, long double) in a
format like
[sign][exponent][fraction]. that means that it can distinguish between
0.0000003 and 0.0000004 but perhaps not between 700000.0000003 and
700000.0000004. on the other hand, it can handle numbers up to
3.4028235*(10^38) for a 32-bit floating point variable.
.
- Follow-Ups:
- References:
- Prev by Date: Re: This calculation is just wrong / computer can't count!
- Next by Date: Re: This calculation is just wrong / computer can't count!
- Previous by thread: Re: This calculation is just wrong / computer can't count!
- Next by thread: Re: This calculation is just wrong / computer can't count!
- Index(es):