Reading UDTs out of a byte array (CopyMemory vs Get on a disk file)

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

From: Beowulf (beowulf_is_not_here_at_hotmail.com)
Date: 06/18/04 

Date: Fri, 18 Jun 2004 14:26:17 -0400

Ok, I've been banging my head against the wall on this one for the
better part of a day, now and am hoping someone can point me in the
right direction.

I have an ActiveX DLL that converts data structures stored in a
proprietary binary file into COM objects. It's been working fine for
quite a while, but recently some users have complained about how long
it takes to load a file over the network (50 - 75 seconds), so I'm
looking into buffering the file to speed up access.

The file is a data dictionary for a proprietary database system
developed by my company. Anyway, it consists of a 64 byte header
followed by some number of 128 byte "variable records" laid out
contiguously on disk and then some number of 64 byte "segment records"
also laid out contiguously on disk.

Here is what I am doing to create a byte array in memory of the
contents of the data file:

Public Sub LoadBuffer(sFile As String)

  Dim hFile As Integer
  Dim lNumBytes As Long
  
  ' Open file
  hFile = FreeFile
  Open sFile For Binary Access Read As #hFile
  
  ' Dimension array
  lNumBytes = LOF(hFile)
  ReDim aryBuffer(lNumBytes - 1) As Byte
  
  ' Fill buffer
  Get #hFile, , aryBuffer
  
  ' Close file
  Close #hFile

End Sub

LoadBuffer() executes in about 1 second over the network, so it holds
promise. Using the current/slow method, I would just do sequential
Get #hfile calls on the file to fill in my UDTs. However, now that I
have the bytes in memory, I'm trying to use the Subs below to get the
data into the UDTs (declaration of UDTs below). I've used a hex
editor to inspect the bytes in the source file, and then gone through
my byte arrays and the bytes are all the same. For some reason,
though, CopyMemory doesn't store the bytes into the UDTs correctly.

This is what the data for a single "variable record" file looks like
using the slow method (which fills the UDT correctly) and what I'm
trying to get working using CopyMemory:

ReadFrom|Method|left_ptr_offset|right_ptr_offset|balancing_byte|unique_var|seg_var_name|offset_value|var_attrib|var_status|format_offset_h|format_offset_s|segment_offset|var_offset|lo_bound|hi_bound|var_length|var_new_length|redefine_begin|default_missing_values|order_number|var_label|Digits_only|response_require|question_number|available_byte|order_number_ptr|new_format_offset_h|new_format_offset_s|delivery_flag|redefine_end|next_var_num|dbsv_filler_1
File on disk|"Get #hfile,,udtVar"|25856|451072|43|U|BASE BASEID
|1|67|65|-1|-1|0|0|0|0|8|0|0| |1|SAMPLE PERSON ID NUMBER PLUS CHECK
DIGIT|32|32| |0|0|-1|-1|32|0|226|32
Byte array in memory|"CopyMemory udtVar, aryBuffer(jump - 1), 128
"|25856|451072|43|U|BASE BASEID |1|67|65|-1|65535|0|0|0|80|0|0|
 |16723|MPLE PERSON ID NUMBER PLUS CHECK DIGIT |32|32|
|0|-1|-1|-503316448|0|8224|0|0|

Any help in figuring out what's going on, would be greatly
appreciated. Thanks.

Private Sub GetDDHeader(udtHeader As data_base_timestamp)
  CopyMemory udtHeader, aryBuffer(0), VAX_TSTAMP_REC_SIZE
End Sub

Private Sub GetDDVarRecord(jump As Long, udtSegVar As
data_base_seg_vars)

  CopyMemory udtSegVar, aryBuffer(jump - 1), VAX_DBSV_REC_SIZE

  ' The commented lines below document the final state of my attempts
  ' at shifting around the bytes in a temporary buffer before copying
  ' them into the UDT at a vain attempt to resolve word alignment
  ' issues. So far, even after various shifts, the data ends up
  ' getting out of alignment when read into the UDT
  '
  ' Read bytes into temporary buffer
  'CopyMemory aryTemp(0), aryBuffer(jump - 1), VAX_DBSV_REC_SIZE
  '
  ' Right-shift the bits to account for word alignment issues
  '
  ' right-shift format_offset_s by 2 bytes
  'CopyMemory aryTemp(35), aryTemp(33), VAX_DBSV_REC_SIZE - 34
  '
  ' right-shift hi_bound by 2 bytes
  'CopyMemory aryTemp(47), aryTemp(45), VAX_DBSV_REC_SIZE + 2 - 46
  '
  ' right-shift var_label by 40 bytes
  'CopyMemory aryTemp(103), aryTemp(63), VAX_DBSV_REC_SIZE + 4 - 64
  '
  ' Read the shifted bits into the structure
  'CopyMemory udtSegVar, aryTemp(0), VAX_DBSV_REC_SIZE
End Sub

Public Type data_base_timestamp
    
    ' Various time stamps
    db_date_created As String * 6 ' Database date created
    db_time_created As String * 6 ' Database time created
    db_date_modified As String * 6 ' Database date modified
    db_time_modified As String * 6 ' Database time modified
    db_date_reorg As String * 6 ' Database date
reorganized
    db_time_reorg As String * 6 ' Database time
reorganized
    
    ' Offset information
    dbsv_head_offset As Long ' Offset value of head
node in segment.variable tree
    dbs_head_offset As Long ' Offset value of start
of segment structure
    
    ' Totals
    total_seg_var_records As Integer ' Total number of
segment.variable records
    total_seg_records As Integer ' Total number of
segment records
    total_fmt_records As Integer ' Total number of
records needed to store ALL format info
    total_formats As Integer ' Total number of
formats defined
    
    ' Miscellaneous vars
    db_study_id As String * 8 ' Heading of dd list
    db_privilege_byte As Byte ' "R" = pending reorg, "
" - no reorg outstanding
    dbt_filler_1 As String * 3 ' A little extra room
for whatever
End Type

' The Data Dictionary segment structure (VAX_DBS_REC_SIZE bytes)
Public Type data_base_segments
    dbs_current_address_ptr As Long ' Pointer to buffer
address of current segment
    dbs_segment_name As String * 8 ' Name of the segment
    dbs_replay_head As Long ' Pointer buffer to hold
random file access reference
    dbs_segment_length As Integer ' Total segment size for
allocation
    dbs_key_length As Integer ' Size of record key for
ISAM access. Set by file open routine.
    dbs_list_head As Long ' Pointer to first
segment record in regular record list
    dbs_record_ctr As Integer ' Count of the number of
records in the list
    dbs_offset_value As Integer ' Equal to occurrence
number
    dbs_file_status As Byte ' 0 = closed, 1 = open
    dbs_fab_addr As Long ' Pointer to File Access
Block (FAB) of the segment
    dbs_prototype_offset As Long ' Prototype offset from
beginning of seg vars (skip timestamp)
    dbs_record_status As Byte ' =1 all records loaded
from disk, also known as dbs_load_status
    dbs_bytes_read As Integer ' Contains the byte
count of the segment record for I/O
    dbs_key_offset As Integer ' Offset of record key
for ISAM access. Set by file open routine.
    dbs_preload_var_offset As Integer ' Offset of segment
preload flag
    dbs_preload_var_length As Integer ' Length of segment
preload flag
    dbs_delete_head As Long ' head of list of
deleted segments (7/31/85)
                                              ' actually, top of stack
of deleted segments, implemented as list
    dbs_list_tail As Long ' -> Last segment record
in regular record list (9/23/85)
    dbs_var_cnt As Integer ' Number of variables on
this segment
    dbs_access As Byte ' "G" =Getseg access
only
    dbs_first_var_num As Integer ' Variable number of var
with lowest order number on this seg
    dbs_filler_1 As String * 2
End Type 

--
What do you get when you multiply 6 by 9?

Relevant Pages

  • RE: Formula in given via input box
    ... Dim segment As String ... Dim formula As String ... End Sub ...
    (microsoft.public.excel.programming)
  • Re: Using SS for non-stack operations?
    ... Jim Leonard wrote: ... > But that *is* my current environment (see earlier messages in the ... A simple SUB SP,buf_size is always safe (given that you have enough ... or should I be setting it to my own segment? ...
    (comp.lang.asm.x86)
  • Re: playing & re-playing 7 .wav files
    ... i will try to use background worker for putting ... I thought I should use this segment of code to get this thing done: ... Private Sub cmdStart_Click(ByVal sender As System.Object, ... Dim i As Integer = 1 ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Segments
    ... Within as form I have a sub that draws segments that go to make a pie chart, ... How can I find out the X & Y of the two points of the segment on the diameter of the circle. ...
    (comp.lang.basic.visual.misc)