Re: This calculation is just wrong / computer can't count!

".rhavin grobert" <clqrq@xxxxxxxx> wrote in message
news:1191512413.771033.139730@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
On 4 Okt., 17:28, "GT" <ContactGT_remove...@xxxxxxxxxxx> wrote:
".rhavin grobert" <cl...@xxxxxxxx> wrote in message

news:1191511188.715360.292900@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

On 4 Okt., 17:00, "GT" <ContactGT_remove...@xxxxxxxxxxx> wrote:
".rhavin grobert" <cl...@xxxxxxxx> wrote in message
On 4 Okt., 16:24, "GT" <ContactGT_remove...@xxxxxxxxxxx> wrote:
What do you mean by the number might be below double granularity on
the
computer? What can I do about this?

your computer stores floatingpoints (float, double, long double) in a
format like

[sign][exponent][fraction]. that means that it can distinguish between
0.0000003 and 0.0000004 but perhaps not between 700000.0000003 and
700000.0000004. on the other hand, it can handle numbers up to
3.4028235*(10^38) for a 32-bit floating point variable.

But if the computer stores numbers in this format, then why does it add
an
extra digit on the end that it can't handle and more importantly - how do
I
stop it from adding the extra digit on the end?

it doesnt "add" it.

If you do...
int a = 1.34;

and then look into the memory at &a you'll find a "1", because the
compiler does a

int a = (int) 1.34; => int a = 1;

because the constant is first cast into the appropriate value (1 for
int, because granuarlity of int is 1) and then moved into memory at
&a. For doubles, thats the same. But doubles have granuarlity
antiproportional to absolute (unsigned) size. You can imagine it as a
"floating point" in decimals: guess you have (for example) 10 digits,
a sign (on= + /off = -) and something that states: put decimal point
here...

if you do a = 0.123456789, then it would be printed as 0.123456789, if
you do a = 0.1234567895 than it would be also printet as 0.123456789
because you're below granuarlity.

if you do a a = 1234567890 ten wit will be printed as 1234567890, if
you substact 0.4 then you're below granuarlity and may get 1234567890
or 1234567889 depending on your system.

I understand, but how do I stop the computer from working outwith its
granularity? I understand that there are a finite number of digits in a
double, plus a sign, plus an exponent, but why does the computer display an
extra digit? I asked the computer to do 25/30 and store the answer as a
double. The answer is 0.8333333. The computer should surely only store as
many of the trailing 3s as it has space for. If it has 8 digits, then it
should store 0.83333333, but instead it shows a 7 as the last digit. This
isn't a rounding error, this is just wrong. How do I stop it from
showing/using the extra digits?


.

Relevant Pages

  • Re: This calculation is just wrong / computer cant count!
    ... What do you mean by the number might be below double granularity on ... extra digit on the end that it can't handle and more importantly - how do ... because granuarlity of int is 1) and then moved into memory at ... The computer should surely only store as ...
    (microsoft.public.vc.mfc)
  • Re: VBA code outputs currency value rounded to 2 decimal points
    ... I did modify the number format to include an ... extra digit, but based on my observation that the value is changed ... prior to it being written to the spreadsheet, ...
    (microsoft.public.excel.programming)
  • Re: Foreign Phone Number Format
    ... I dont believe there is any std format ... London present format is 999 ... with an extra digit, in one of the first two groupings, in any case format ... > Does anyone have a Web resource for standards in formatting foreign phone ...
    (microsoft.public.access.tablesdbdesign)
Loading