Re: foreach, IEnumerable and modifying contents

Tech Tip: Click here to run a free scan for Windows Errors and optimize PC performance

On 2007-11-28 07:01:58 -0800, "jehugaleahsa@xxxxxxxxx" <jehugaleahsa@xxxxxxxxx> said:

Well, the first go around, the queue being empty didn't mean I was
done. It occurred quite often that I would finish downloading all my
files before more files were added to the list.

The queue being empty did in fact mean you were done, at least for the moment.

In a typical queue design, you would gracefully deal with an empty queue. A queue that's empty just means there's no work to do. The consumer sits idle (either as an actual thread blocked on an wait event, or just a class that doesn't do anything until some code calls something that adds something new to the queue) until there's more work to do. The logic is the same for the case of starting up some processing as it is for the case of temporarily running out of work to do and then being presented with some more.

If your design didn't support that, then you probably did not separate the logic of the producer, consumer, and client of the queue well enough.

[...]
Here is a scenario: One download finishes and my code begins pulling
the next Download. However, the web extractor is not ready. While
waiting, another download finishes and now a second piece of code
begins pulling the next Download. Now I have two pieces of code trying
to access the same enumerator. Can I be sure that this won't corrupt
my enumerator? If I were to lock the IEnumerator<Download>, would this
cause a deadlock since they are different event handlers?

I can't really comment on an enumerator that you haven't posted code for. Also, I haven't used any custom enumerators in real-world code, so I don't have much experience with them. However, I would say that if you have two pieces of code trying to access the same enumerator, you've got a bug. I would think that each call to GetEnumerator() should return a brand new one, so that different parts of the code don't interfere with each other.

If you do decide to return the same enumerator to different parts of the code, or different instances of the same code, I'd say that at a bare minimum you will need to be VERY careful about how you use the enumerator (and for sure it will need to be written in a thread-safe way to account for this multiple access usage), and it's very likely there's a better way to design the code (like using a queue :) ).

Concurrency isn't that simple for someone who hasn't had to deal with
it. I had plenty of theory in school, including producer/consumer
algorithms. Dealing with events seems similar to threads, but they
take complete control. Threads at least switch context when they hit a
lock.

Events and threads are, as I mentioned, orthogonal to each other. An event is really just a nice syntax for a multi-subscriber callback mechanism. When an event is raised, the handler always executes in the same thread in which it was raised. Multiple threads impose synchronization requirements on your code, and these requirements are the same whether you are using events or not.

That said, I never meant to imply that concurrency was simple. It's not. If anything, my intent is to point out that concurrency is _not_ simple, and that your second design appears to have just made it more complicated than it otherwise needed to be.

If you want to have multiple threads processing things, you _are_ going to have to deal with concurrency. So the question is not whether you can get away from concurrency issues or not; you can't. The question is how complicated are you going to make those issues.

So far, it seems that you've made them very complicated. :)

For fun, I'm thinking about working on a simple download simulation that uses a queue to manage the downloads. If and when it's finished, I'll post the code here in case you or anyone else is interested. Might not be done today, as I've got a busy day, but maybe tomorrow.

Pete

.

Relevant Pages

  • Re: Alt.Binz: Using v242. Should I upgrade?
    ... Newzleech RSS fixed [Link] ... + previous copy of queue file is kept and can be loaded when current queue ... * manual nzb rss download possibile while on rss timer ... redocked tab state not beeing saved when exiting ...
    (news.software.readers)
  • Re: foreach, IEnumerable and modifying contents
    ... I would like to incorporate threads so that I can download N ... The producer of the queue would add URLs to be downloaded, while the consumer would keep track of how many active downloads are going on. ... thread pool when the downloads are finished. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: foreach, IEnumerable and modifying contents
    ... The queue being empty did in fact mean you were done, ... One download finishes and my code begins pulling ... to access the same enumerator. ... object cancelSync = new object; ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Traffic Shaping not working correctly after ipfw coverted to use pfil_hooks API
    ... I'm using 400Kbit/s as the upload limit ... > # Assign all other outgoing traffic to the low-priority queue ... The upload pipe ... > Here is where the problem is if I do a upload and download at the same time: ...
    (freebsd-current)
  • Re: Problem with subprocess in threaded enviroment
    ... address from the queue and download it. ... You don't need a semaphore for that. ... def download(url): ...
    (comp.lang.python)