Re: Vb6 large text file read / arrays performance issues

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

Firstly, I want to thank you, and others in this thread, for the
interest taken.
I am spending a few days with my family at the moment so i only get
time for looking at the newsgroup.

I am very interested in using more of a disk based solution as it will
be more scaleable.But for now,
ss you raised various questions I thought the least i can do is add
more to the problem description.


I have to receive text messages (approx 20 chars each on average) from
up to 8 socket-based servers and look for matches against multiple
sorted-text-lists, some lists are short (10,000) some are long
(5,500,000). The lists can be treated as static and the application is
to be used with a GUI and started when required and then stopped, ie
this app does not have to run for more than a day.

My problems are, as I see it:

# To index my sorted-text-lists, which have a key field in column1 and
supplimentary data in other columns. [I can do this with fast
searching, but i pay for it in RAM and time to load, both of which i
have available, but my solution is not too scaleable.

# To handle multiple streams of messages from my 1 or 2 clients (I may
run 1 client per 4 message servers)

# To combine my data at some point to a single queue for searching and
GUI display; and data save.


More on my Comms:

I have a winsock control acting as a client receiving strings at rate
of approx 1 per second.

I need to consider scaling this with my client app talking to multiple
servers, say upto a maximum of 4, so max data rate is 4 messages per
second, length of string approx 20 bytes.

I then need to take each message and perform a string search and show
any matches in my GUI and store in database of some sort (I have a SQL
server if I want to use it.), I expect
match rate approx 1 hit in every 1000 messages.

If I only I have 1 client-server connection, therfore only 1 source of
messages, I can use a fnIsMessageMatch call after every message
arrival, my string search function says it takes 0ms using TimeGetTime
api calls, so I think it is pretty quick.

If I then look at multiple client-server connections should I simply
use my string search function again by each of the instances of my
winsock control ?

Or should I put all my messages in some sort of fifo buffer/message
queue ?

I am not sure how the mix of event driven multiple socket connections
behaves in this type of situation.



Hopefully this makes mre sense.

.

Relevant Pages