Re: foreach, IEnumerable and modifying contents
- From: Peter Duniho <NpOeStPeAdM@xxxxxxxxxxxxxxxx>
- Date: Tue, 27 Nov 2007 21:04:13 -0800
On 2007-11-27 20:24:28 -0800, "jehugaleahsa@xxxxxxxxx" <jehugaleahsa@xxxxxxxxx> said:
I have a rather complex need.
Perhaps. Though, I suspect it's more that you've created a complex need, where it wasn't really necessary to do so.
I have a class that parses web pages and extracts all relevant file
addresses. It allows me to download every pdf on a web page, for
instance. I would like to incorporate threads so that I can download N
files separately.
A reasonably common operation.
The obvious solution is a thread pool. However, I need to make sure
that I download the files Async - so I can get percentage and status
information to my interface.
It seems to me that a different "obvious" solution would be to just use the async methods on the HttpWebRequest class, or even just a plain TcpClient or Socket instance, along with a queue. The producer of the queue would add URLs to be downloaded, while the consumer would keep track of how many active downloads are going on (via HttpWebRequest, TcpClient, or Socket).
Every time the producer adds something to the queue, it would signal the consumer. The consumer in response would remove items from the queue, stopping when either the queue is empty or your maximum number of concurrent operations has been reached, whichever comes first.
Upon completion of an item, the consumer would also be signaled, allowing it to pull a new item from the queue.
In the above, I'm thinking of the consumer and producer as individual threads. But you could easily implement it without a thread dedicated to either, with the consumer and producer classes simply being called by whatever thread happens to be managing them at the time. In that case, "signaling" the consumer would be more a matter of just executing the method that attempts to dequeue more download operations.
I have decided that the best way to do this is to have my Download (a
class representing the file to download) to have events raised when
they are finished. I was hoping to have my threads rejoin with the
thread pool when the downloads are finished.
If you use the async methods on the above-mentioned classes, you get the thread pooling behavior for free.
However, I have my Download instances coming out of an
IEnumerable<Download> that is recieved from the WebExtractor class
(which parses the HTML) on-the-fly using "yield return".
This is another reason I think a queue would be better. There's no technical reason you can't implement an asynchronous enumerator, but having done so in this case seems to have overcomplicated the issue. A queue seems like a much more natural fit to me, and wouldn't have the same complicating factors you seem to be running into.
I think I am lacking some basics about Thread Pools. How can I use a
thread pool and have the events fired by the Downloads still reach the
interface?
I think you can avoid the question altogether, but the basic answer is that the idea of a thread pool and having "the events...reach the interface" are orthogonal ideas. Because of the thread pool, you may have thread synchronization issues to deal with. But the basic question of raising an event in a way that some implementer of some interface receives them isn't affected by whether there are multiple threads involved.
Is there are way to add an event handler to an instance
while in a foreach or IEnumerator code block?
You can subscribe to an event at any time you find convenient.
Any help would put me one step closer to being done with my second
release of the software. Thanks in advance!
See above. I recommend abandoning this asynchronous enumerator idea and going with a nice, simple queue.
Pete
.
- Follow-Ups:
- Re: foreach, IEnumerable and modifying contents
- From: jehugaleahsa@xxxxxxxxx
- Re: foreach, IEnumerable and modifying contents
- References:
- foreach, IEnumerable and modifying contents
- From: jehugaleahsa@xxxxxxxxx
- foreach, IEnumerable and modifying contents
- Prev by Date: Re: Play media files
- Next by Date: Re: Play media files
- Previous by thread: foreach, IEnumerable and modifying contents
- Next by thread: Re: foreach, IEnumerable and modifying contents
- Index(es):
Relevant Pages
|
