Re: Multi-Threading

This sounds a lot to me like you're using a hammer on a screw. If I
understand you correctly, you were having performance issues when reading a
100,000+ line log file and decided to use multi-threading as a solution to
this, which is seldom a solution.

Multi-threading on the desktop is useful when you're executing long running
blocking operations and want the GUI to remain responsive, running
background tasks and similar. Multi-threading on the server is useful if
you're servicing multiple requests simultaneously and perhaps want to take
full advantage of multi-core CPUs. It's nowhere near as useful for
improving performance as some people think it is; a program running a task
across 8 threads is NOT 8x faster than the same program running the same
task on just 1 thread.

Multi-threading should be used cautiously and when necessary as it can
create exactly the kinds of headaches you're having with your code. As
you've probably noticed from this thread, everybody has a different take on
how you should sync your resource access across multiple threads.

You need to investigate *why* your app is slowing down as it reads its way
through that log file. Are you allocating a new string each time you read a
line of the log file? Remember that strings are immutable, so once they're
created they won't be destroyed until the GC decides to. This could be
swamping your working set.

Is there a more efficient design pattern you could follow for reading the
file data and writing it into the database?



"Barkingmadscot" <barry@xxxxxxxxxxxx> wrote in message
news:36f3e01a-075f-4772-ac03-4cc59caf666c@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I am having problems with Multi-threading and write to Key Access
Database

I have a log file that and collecting infomation from

getting the info is a problem or is writing it the database, the log
files a big and it takes time to read down as it get further in to log
said, a 100,000 line in

I have changed it to mutli-threading which almost works, the problem
is when writing data to the table. I get dup keys and the whole
stops, this does not happen with the single thread app.

basically i want each thread to wait until thread before finish its
write to the database.

ie.
thread 1 waits for thread 0 to finish writing to table
thread 2 waits for thread 1 to finsih writing to table
thread 4 waits for ..........

during this each for the thread have collected the data they need to
write to the table.

Here is the line code to

Dim intcount As Integer = 1, intcount1 As Integer =
fNoOfLines(fileNm)
Dim AppThreads(4) As Thread
Dim WebData(intcount1) As DoWork

Do Until intcount > intcount1

WebData(Thread) = New DoWork

'################################

Code collecting data is here
################################

'# Setup and start new thread
AppThreads(Thread) = New Thread(AddressOf
WebData(Thread).sWriteWebUsage)
AppThreads(Thread).Start()
System.Threading.Thread.Sleep(15)
intcount = intcount + 12
Loop

I removed all the code that does the data collecting as it works
fine,

I believe my problem is at the Appthreads(thread).start()


.

Relevant Pages

  • Re: multi threading programming
    ... > I am writing an application to process log file. ... like the kind of task which is actually helped much by multi-threading. ...
    (microsoft.public.dotnet.framework)
  • Re: Hard Drive Space
    ... of the distribution, ... security problems it had over the life of the distribution. ... in the space, either the box is 0wn3d, or you have a process that is writing ... If you don't recall removing some log file manually, ...
    (linux.redhat)
  • Re: Logrotate is a pain
    ... >>important feature is that the program, which writes the logfiles does ... > to have missed is that if you rename an open log file, ... > writing to it continues writing to it regardless of the name. ... You can simply put a simple line in it to send SIGHUP to your daemon and the ...
    (comp.os.linux.misc)
  • Re: How to remove a single line from a flat file (Still very off-topic.)
    ... turn off syslog (so it's not writing to the file while you edit it), ... Editting includes rotating the log file, removing old entries, etc. ... part of the problem is the design of these files. ... more time/energy/money is spent on the admin juggling log files ...
    (comp.lang.tcl)
  • Re: Logrotate is a pain
    ... to have missed is that if you rename an open log file, ... writing to it continues writing to it regardless of the name. ... is that the process is writing to an open inode; ...
    (comp.os.linux.misc)