Re: Subtraction Giving an incorrect value
- From: "Wayne Morgan" <comprev_gothroughthenewsgroup@xxxxxxxxxxx>
- Date: Thu, 19 Jan 2006 10:44:55 -0600
1.907348635E-07 is 1.907348635 time 10 to the minus 7th power (i.e.
0.0000001907348635). As you can see, this is almost zero. The problem is
that floating point numbers aren't handled perfectly, these tends to be a
round-off error. There are some data types that get handle this problem
better, but if you don't specify, the floating point numbers get treated as
Single or Double, which each have this problem. Currency and Decimal don't
because of the way they handle floating point numbers is different. You can
use the Currency data type even if the value isn't going to be currency, but
it does limit you to four decimal places. There is no Decimal data type for
VBA, so to use Decimal data types, you have to use Variants and the CDec()
function as you assign the value to the variant. The other option is to
round off the result to the number of decimals you need. In this case, if
you rounded to one decimal, the result would then be zero.
--
Wayne Morgan
MS Access MVP
"Ann" <Ann@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:62251596-1C7D-45CC-9640-8E2CE1F55C0B@xxxxxxxxxxxxxxxx
>I have a control called PTO Carryover that returns 5.8. I also have an if
> statement (named Carryover) that returns the same value, 5.8. The
> statement
> is =Sum(IIf([PTO Type] Like "Carryover",[Paid Time Off],"0")). These
> values
> are correct and I can see both of them on my report but when I subtract
> Carryover from PTO Carryover (5.8 - 5.8) I get 1.907348635E-07.
>
> Can anyone tell me what is happening? I've tried so many different ways
> to
> get this to come out to 0 but I always get 1.907348635E-07.
>
> Thanks in advance.
.
- Prev by Date: Re: Auto-expand continuous subform memo fields
- Next by Date: Re: Command button to reset form for fresh entry?
- Previous by thread: Re: Auto-expand continuous subform memo fields
- Next by thread: Re: Command button to reset form for fresh entry?
- Index(es):
Relevant Pages
|
