Re: Whole Nos!

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

> I have created a calculator in VB6. The calculator also has a Inverse
> button clicking which the inverse of the no. will be displayed. Now the
> problem is suppose I want the inverse of 6. The calci correctly
> computes the reciprocal of 6 i.e. 0.166666666666667 but when I again
> click the inverse button, the output shown is 5.99999999999999 & not 6.
> How do I overcome this?

This is down to the way floating point values are stored, they're trying to be too precise for their own good. There's
probably better solutions to this but here's one you may want to try, it simply trims a number to a certain number of
decimal places and performs rounding where necessary:

'***
Private Function TrimDP(ByVal inVal As Double, ByVal inDP As Long) As Double
Dim DPOffset As Variant
Dim OffVal As Variant, IntBit As Variant

If (inDP >= 0) Then
DPOffset = CDec(10 ^ inDP)
OffVal = DPOffset * inVal
IntBit = CDec(Int(OffVal)) ' Perform bankers rounding
If ((OffVal - IntBit) > 0.5) Then IntBit = IntBit + 1
TrimDP = IntBit / DPOffset
End If
End Function
'***

Now when you try this:

'***
Debug.Print TrimDP(1 / (1 / 6), 10)
'***

You get 6 as expected. Of course this is a bit of a cheat since 1/(1/6) actually returns 6 anyway, however with your
conversion to string in the middle it would fail and this does fix that situation:

'***
Dim TempVal As String
Dim ResVal As Double

TempVal = CStr(1 / 6)
ResVal = 1 / CDbl(TempVal)

Debug.Print "Before: " & CStr(ResVal)
Debug.Print "After: " & CStr(TrimDP(ResVal, 10))
'***

If you avoid your conversion to string in the middle and just use a Double value stored internally to keep the current
value (the text box is just for display) then you may simply circumvent these problems entirely. You'll see exactly the
same problem in Windows calculator too, try this:
Perform the calculation 1/6 it will print out "0.16666666666666666666666666666667", copy that (Ctrl+C), clear, and paste
the same value back in (Ctrl+V, thus forcing it to do a string to double conversion internally). Stick that value in
memory (M+), clear, then perform 1/MR it will print out a number very close to 6 but not quite 6.
Hope this helps,

Mike


- Microsoft Visual Basic MVP -
E-Mail: EDais@xxxxxxxx
WWW: Http://EDais.mvps.org/


.

Relevant Pages

  • Re: Output data from multiple records in a table
    ... > Dim rsR As DAO.Recordset ... > Dim strSQL As String ... > such as display it in a textbox on your form. ...
    (microsoft.public.access.modulesdaovba)
  • SendObject Action Message
    ... Dim strMsg As String, strTitle As String ... 'There is no need to check if the form is blank (as with report and preview) ... 'will display the name of the report. ...
    (microsoft.public.access.formscoding)
  • Re: Output data from multiple records in a table
    ... Dim rsR As DAO.Recordset ... Dim strSQL As String ... >the other record into wordpad for display on the screen. ...
    (microsoft.public.access.modulesdaovba)
  • Re: Allen Browne Audit Log
    ... >> also need to ensure that the PK added to the audit table begins with aud ... >> Function AudText(sAudTable As String, sKeyField As String, AudID As ... >> Dim rs As Recordset ... If you wanted to display just the fields that have been ...
    (microsoft.public.access.formscoding)
  • RE: Email
    ... If i use Display instead of send i get the New email' window like expected. ... I did notice that you're not explicitly destroying the MyOutlook object as ... Public Sub SendEMail(strMailTo As String, strFile As String, _ ... Dim MailList As DAO.Recordset ...
    (microsoft.public.access.modulesdaovba)