How to make a safe HTTP stream reading?

Tech Tip: Click here to run a free scan for Windows Errors and optimize PC performance



I am using C# to write a crawler-like program. The program reads
several URI pages sequentially and store HTTP response into a single
byte array. The job should be done in specific time. So, I have both
size limit and time limit.

My current solution is like this. use
HttpWebResponse.GetResponseStream to get the I/O stream. Read from it
with asynchronous BeginRead. In the callback, check whether time and
space is up. If yes, then stop; otherwise, call BeginRead again to
read from stream. The thread kicks off the first BeginRead will wait
on a AutoResetEvent, if time is up, it go ahead and close the
HttpWebResponse.

This solution looks fine, but it actually has a bug. Say, when the
HttpWebResponse instance is closed, perhaps the stream is still being
read. Then a IOException is thrown. That is not good. If add a
monitor, then closing of HttpWebResponse may hang on I/O reading. It
is still not preferable.

So, what's the best way to do it?

.