How to make a safe HTTP stream reading?
- From: Morgan Cheng <morgan.chengmo@xxxxxxxxx>
- Date: Wed, 27 Jun 2007 06:09:27 -0700
I am using C# to write a crawler-like program. The program reads
several URI pages sequentially and store HTTP response into a single
byte array. The job should be done in specific time. So, I have both
size limit and time limit.
My current solution is like this. use
HttpWebResponse.GetResponseStream to get the I/O stream. Read from it
with asynchronous BeginRead. In the callback, check whether time and
space is up. If yes, then stop; otherwise, call BeginRead again to
read from stream. The thread kicks off the first BeginRead will wait
on a AutoResetEvent, if time is up, it go ahead and close the
HttpWebResponse.
This solution looks fine, but it actually has a bug. Say, when the
HttpWebResponse instance is closed, perhaps the stream is still being
read. Then a IOException is thrown. That is not good. If add a
monitor, then closing of HttpWebResponse may hang on I/O reading. It
is still not preferable.
So, what's the best way to do it?
.
- Prev by Date: Re: Using ToString to pad an integer with leading zeroes
- Next by Date: Re: recursive GetEnumerator()
- Previous by thread: Modal Dialog and Threads
- Next by thread: MS Shell Dlg 2 in .NET forms?
- Index(es):