Re: FIFO lost on Critical Section
- From: Jeroen Mostert <jmostert@xxxxxxxxx>
- Date: Fri, 09 May 2008 01:45:09 +0200
Timothy Jewett wrote:
I need a little advice. Because SP1 of W2K3 took away the fifo feature of the CriticalSection object I need a way to find another way to achieve that feature without a huge performance hit.
Well, finding out why you need this should be good. :-)
What I have is a IOCP queue that handles the socket events for a service
that is a gateway between two networks. I have session objects that
relate to a socket and worker threads that process the socket events.
These events must be handled in the order coming off of the iocp queue.
In the past I would just deal with this by using a Citical Section which
is part of each session object.
I think this is broken to begin with. Although the old CS would release waiters in FIFO order, you cannot actually guarantee the threads will start waiting in FIFO order with this scenario.
Imagine we've got two processors, and two threads waiting in GetQueuedCompletionStatus(), and two events arrive for the same session. Imagine thread A was the last one to start waiting, which means the IOCP will wake it up first (IOCPs wake worker threads in LIFO order). Thread A is now processing event 1, but just as it's figured out which session the event belongs to and it's about to grab the corresponding critical section, it gets preempted by some third-party thread.
Now thread B gets woken up to work on event 2. It's not preempted, so it happily grabs the critical section and processes event 2. Then thread A gets a chance to run again and it finally processes event 1. Whoops: you've just processed the events out of order.
This scenario is not *likely*, since thread A will have a low chance of being preempted right after being woken up, but it's certainly *possible*. Fixing it is not trivial.
The work would have been performed in the order comming off the queue.
Now that is not occuring. I tried to create a single thread to take the
items off the socket queue and place them on and internal session queue
and post to a second queue for the worker threads to process the events.
I control the session queue with a second critical section in the session
object. ( I needed to do this because the InterlockedEntrySList is filo
and I needed fifo ). This all works but I lost 25% of my throughput. Is
there another way to handle work coming from the socket iocp queue ?
Using an InterlockedSList with a CS defeats the point of having a lock-free structure. You could use a proper lock-free FIFO queue instead, but those aren't easy to implement correctly. Or you could use a CS with a *regular* FIFO queue, of course. Also, a "free" FIFO queue is available in the form of a thread's APC queue, which you can post to with QueueUserAPC(). You cannot combine this with GQCS(), so you'd need separate worker threads for the sessions, but you can multiplex sessions over threads if you statically assign affinity (i.e. thread N handles events for sessions ID % N == 0 only). This only works well if sessions have approximately equal loads, though.
--
J.
http://symbolsprose.blogspot.com
.
- Follow-Ups:
- Re: FIFO lost on Critical Section
- From: Timothy Jewett
- Re: FIFO lost on Critical Section
- References:
- FIFO lost on Critical Section
- From: Timothy Jewett
- FIFO lost on Critical Section
- Prev by Date: Re: CreateProcessWithLogonW without UAC popup in Vista
- Next by Date: Re: Quadratic timing behaviour of long pathname access in NT 5.x?
- Previous by thread: FIFO lost on Critical Section
- Next by thread: Re: FIFO lost on Critical Section
- Index(es):
Relevant Pages
|
