Surprising threading issue

In my app, I need to connect to an external web service. I need to
connect to it about 100 times per request and so i use threads to
quicken the processing. Since the external web service accepts only 5
requests per second, I need to check for that too.

I use Jon Skeet's CustomThreadPool class for the Threads (because i
dont want to use the ASP.NET thread pool and I dont have nearly enough
technical expertise to write my own).

Here's my code:

private static DateTime lastHttpCallTime = DateTime.MinValue;
private readonly object weeklyProcessorLocker = new object();
private static readonly object dateTimeLocker = new object();
private int weeksProcessed = 0;


//Code enters here
void MainFunc()
{
//Create thread pool for handling data for each week. Kill any
thread which has been idle for more than 10s
threadPool = new CustomThreadPool();

//Add each web service call item to thread pool
for (int i=0; i<itemsToProcess.Length; i++)
threadPool.AddWorkItem(new WeeklyProcessorDelegate
(ReadDataForWeek), itemsToProcess[i].from, itemsToProcess[i].to);

//Keep waiting till all items are processed
lock (weeklyProcessorLocker)
{
while (weeksProcessed < itemsToProcess.Length)
Monitor.Wait(weeklyProcessorLocker);
}

//Process further
}


//Delegate method for the ThreadPool Work Item
private void ReadDataForWeek(string from, string to)
{
//Create url using from and to
Uri url = new Uri(String.Format(URL_STUB, from, to));
XPathNodeIterator weeklytTrackIt = GetHttpUrlContent(url, XPATH);

//Use weeklytTrackIt for some processing here

//Signal end of processing for this week
lock (weeklyProcessorLocker)
{
weeksProcessed++;
Monitor.Pulse(weeklyProcessorLocker);
}
}

//Code for making http request and getting the xml returned
private XPathNodeIterator GetHttpUrlContent(Uri uri, string xpath)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
Stream responseStream = null;
WebResponse response = null;

try
{
//Wait for 200ms if needed
lock (dateTimeLocker)
{
DateTime now = DateTime.Now;
TimeSpan interval = now - lastHttpCallTime;

if (interval < new TimeSpan(2000000))
Thread.Sleep(new TimeSpan(2000000) - interval);


//Record the last http call
lastHttpCallTime = DateTime.Now;

//Make the http call
response = request.GetResponse();

//dummy statement
//string x = "Hi"; <--------- If i uncomment this it works!
}

//Load to xml document and select iterator to the nodes to be
selected
using (responseStream = response.GetResponseStream())
<-------------------- exception here
{
XPathDocument doc = new XPathDocument(responseStream);
return doc.CreateNavigator().Select(xpath);
}
}
finally
{
if (response != null)
((IDisposable)response).Dispose();
}
return null;
}


At the place i marked above, i intermittently get a malformed xml
exception which says that some tags are not properly closed. However
if i uncomment the dummy statement, it never throws exception.
It's weird but to me it seems that the execution leaves the lock
before the response is fully received.
I'm sure there's a better explanation to this.
Hope someone can get through reading this thread and can help.
.

Relevant Pages

  • ServletException I do not understand
    ... MessageProcessor mp = new MessageProcessor(request, response, true); ... private HTMLEntities htmlEntities; ... private HttpServletRequest request; ...
    (comp.lang.java.help)
  • Re: problem with HttpWebRequest/HttpWebResponse...
    ... Following is the code we are using to get the response object from the request: ... Cache-Control: private ... Immediately after the above response, the next response it shows in Packet Sniffer is: ...
    (microsoft.public.dotnet.framework.aspnet)
  • Re: W2K3 IIS 6.0 ASP.NET Requests Per Second Limits?
    ... allow the page to tell ASP.Net "don't send a response yet until I tell ... this request is "leaked" and will never ... It is when the callback executes with both the async function call ... The thread executing code literally calls into Function1, execute code, ...
    (microsoft.public.inetserver.iis)
  • Re: W2K3 IIS 6.0 ASP.NET Requests Per Second Limits?
    ... >> The way I understand async programming is if u need to do other ... >> request to webservice nothing more can be done until the result ... > The thread executing code literally calls into Function1, ... > the act of sending back the response using data that has been ...
    (microsoft.public.inetserver.iis)
  • ADFS & Sharepoint 2007 Errors
    ... Request or Response. ... claim Group = adfstest readers for user sid ... Sign In Response Dump ...
    (microsoft.public.windows.server.active_directory)
Loading