Re: Problem with a Socket server program opening/accepting many connections and the GC is running.

• Previous message: Joel Rumerman: "Re: Crystal Report Apply"
• In reply to: Phillip O: "Re: Problem with a Socket server program opening/accepting many connections and the GC is running."
• Messages sorted by: [ date ] [ thread ]

Date: Fri, 9 Apr 2004 21:01:12 -0400

Good, glad you are getting somewhere.

Yes your book is half right - it is 5 pending requests max on Workstation
class OS - including XP Pro. One of the ways Microsoft forces you to buy a
server OS if you want to do high capacity server stuff. Of course if your
clients connect and then stay connected (even if there are 1000's of them)
it might not be the issue you think - as long as they don't all try to
connect simultaneously.

You really should try the threadpool as soon as possible - to see how it
reacts to your large workload. Like I mentioned before, you can not use
BeginAccept in combination with a large workload - you have to use a
dedicated thread to do the Accepting, then pass it off into the Threadpool
via BeginReceive. If your connection requests are even burstier, you might
find you need to use a dedicated thread on Accept, and put the accepted
sockets into a queue where another thread pulls them out and puts them into
the Threadpool - there is a bit of overhead on calling BeginReceive compared
to just adding an object into a queue. This might sound like a lot of work
but it enables you to keep up with a flood of requests. Of course, given
enough requests you can still overwhelm the server. At that point more
hardware is needed - dual processor or more servers with a load balancer in
front.

As far as garbage collection killing things, I have done some (soft)
realtime (100ms cycle) on XP with .NET to error-proof a manufacturing
process. I am handling hundreds of packets/sec of data going in/out to
physical I/O on ethernet, running logic to make decisions, logging to
database etc - runs about 4% CPU usage on a P4-2.4. Due to the large volume
of small objects (buffers etc), a Generation 0 collection occurs every
single second!!! This takes about 0.07ms. A full Garbage collection occurs
every 17 seconds. It takes about 0.7ms. That said, my data size is only
about 26MB total and remains constant. Note that I don't ever call the
Garbage Collector myself - its running this often on its own. This is up
for months at a time, only down for changes - no crashes :-), I have had no
issues with memory leaks or failing to collect garbage or the garbage
collector stalling this process. It doesn't happen.

As long as you don't hang onto object longer than needed I don't think it
will be an issue for you. If you are going to make lots of garbage you need
to have it be short-lived where its cheap to get rid of (Gen 0). The
alternative is to reuse buffers - which end up in Gen 2 eventually. I have
mixed feelings on the buffer reuse idea since Gen 2 collections are so much
more expensive than Gen 0 ones.

That's about it for my thoughts.
Tom

"Phillip O" <spam@vwars.com> wrote in message
news:%23mlHCslHEHA.2744@TK2MSFTNGP10.phx.gbl...
> Tom,
> Yes that helps! I never found any reference-mention of it.
> (I was primarily focused on .net and stuff and had not dived into the
> Winsock underpinnings that deep, except for a book, "Windows Sockets
Network
> Programming" by Bob Quinn and Dave Shute. They listed it as 5..... )
>
>
> We have done a previous version of of Sever socket listen using Winsock
and
> C++.
> In that case we are able to handle "1000's" of connections simultaneously,
I
> am told.
> Now it may be the fact that, in this previous version, the listen was
> cleared out quicker (i.e. by passing it to a worker thread as soon as the
> accept was completed), than what I am doing.
>
> In that case, my next iteration of the test code should help alleviate the
> problem as I will start using asynchronous accept or receive's for the
> sockets to handle the messages and their required processing.
>
> We are potentially looking at many thousands of clients access this
> information at the same time. So I believe this test is a realistic
> simulation of the potential client load. Yes the server will have to do
some
> work, so that I why I was in no hurry to pass the simulated work off to a
> worker thread I figured it was a simulation of loading up the server
process
> and keeping it busy. (I guess I was more right than I thought. :(
>
>
> My boss wrote the previous C++, V6 version as a DNS lookup. He and the
rest
> of us are fairly new to C# and .Net but we like what it offers and are
> committed to develop in this if we can.
> So far our only concern had been the GC turning on, and "Stopping" all
> forward motion, but we seem to be able to handle that.
>
> Thanks again,
> Your help and time is much appreciated!
> Phil Ouellette.
>
>
> "Tom Hall" <nospam@harmonyit.com> wrote in message
> news:eBKI8ERHEHA.3832@TK2MSFTNGP10.phx.gbl...
> > Ok.
> >
> > Error 10061 means
> >
> > No connection could be made because the target computer actively refused
> it.
> > This usually results from trying to connect to a service that is
inactive
> on
> > the foreign host-that is, one with no server application running.
> >
> > In your case, again the number 200 is suspicious in my memory. If you
> look
> > up the Winsock documentation in MSDN you find the unhelpful:
> >
> > The backlog parameter is limited (silently) to a reasonable value as
> > determined by the underlying service provider. Illegal values are
replaced
> > by the nearest legal value. There is no standard provision to find out
the
> > actual backlog value.
> > In this article
> > http://support.microsoft.com/default.aspx?scid=kb;en-us;127144
> >
> > it says the limit on NT4 Server is 200. I'm going to assume this is
also
> > the case for Windows Server 2003 because that's what you are seeing and
I
> > can find no other documentation about it!
> >
> > Therefore your problem is you aren't accepting connections quickly
enough.
> > If your clients blast multiple attempts simultaneously you are probably
> > overrunning the pending queue. When this happens, the next attempt
> > generates a RESET packet to make the client give up. Perhaps the delay
> > should be at the client end! Is what you are attempting realistic of a
> real
> > client load? The server might have to do some actual work not just
accept
> > connection attempts :-)
> >
> > Hope this helps as well
> >
> > Tom
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > "Phillip O" <spam@vwars.com> wrote in message
> > news:OXibES2GEHA.2224@TK2MSFTNGP12.phx.gbl...
> > > Tom,
> > > "Slowed down the server" The client and server are connecting and
> > > disconnecting very quickly.
> > > So I had the server slow down, hence a 10 ms delay, so that the
clients
> > > would block on their connect and not use up all their ports. ( The
> clients
> > > got a connection slower so they by they time they had connect on port
> > 9999,
> > > 1026 would have freed up.)
> > >
> > > I have since removed that delay code because I ran into another
problem:
> > > I am now running the server code as a service on Windows server 2003.
I
> > have
> > > set the listen to 1000
> > >
> > > I have propagated the client code to several machines. and have 50 or
> 100
> > > copies of the program executing to simulate many clients over our
> network
> > > attaching to the server and sending data.
> > >
> > > But I cannot get more than 200 connections, (over all clients), at any
> one
> > > time. (I have not coverted the server for asynchronous accept yet,
but
> I
> > > guess I don't think that would be the problem.)
> > >
> > > Any ideas why Windows server 2003 will only accept 200 clients at any
> one
> > > time?
> > > The clients are getting, 100061 errors, (Connection refused by
server),
> > when
> > > more than 200 programs are started.
> > >
> > > any ideas?
> > > Thanks again for the followup
> > > Phil
> > >
> > >
> > > "Tom Hall" <nospam@harmonyit.com> wrote in message
> > > news:%23eRLynEGEHA.2976@TK2MSFTNGP10.phx.gbl...
> > > > I'm not quite sure what you mean by "slowed down my server by adding
a
> > > 10ms
> > > > delay". Could you elaborate why this fixes the problem as normally
> you
> > > want
> > > > to handle the connection as quickly as possible?
> > > > Or, are you also running into the problem of 100% CPU usage
preventing
> > the
> > > > garbage collector from running as often as it needs to?
> > > >
> > > > Thanks
> > > > Tom
> > > >
> > > > "Phillip O" <spam@vwars.com> wrote in message
> > > > news:%23s5fGOzFEHA.2732@tk2msftngp13.phx.gbl...
> > > > > Thanks.
> > > > > This information helped me to understand and solve the problem.
So
> > far
> > > I
> > > > > have increased the amount of sockets available, and Slowed down my
> > > server
> > > > > (adding a 10ms delay). I now have pool of sockets available and
can
> > > handle
> > > > > about 40 socket requests per second, from all clients.
> > > > >
> > > > > I will be adding some asynch recieve code soon because that is
> turning
> > > > into
> > > > > the next bottle neck. (other clients are starving if a large (>
1MB
> > > > message)
> > > > > is being handled by the server.)
> > > > >
> > > > > The echo back (send back to client) will be removed, I think, when
> > this
> > > > code
> > > > > goes into production.
> > > > > Oddly enough, that seems to be handled offline, because while the
> > server
> > > > is
> > > > > echoing back, other clients requests are being serviced.
> > > > >
> > > > > Thanks again,
> > > > > Phil O.
> > > > >
> > > > >
> > > > > "Tom Hall" <nospam@harmonyit.com> wrote in message
> > > > > news:Or9Px55EEHA.2804@tk2msftngp13.phx.gbl...
> > > > > > Well, 3956 is a suspicious number - quite close to the default
> > maximum
> > > > > > available sockets. I suspect you are running into the following
> in
> > > your
> > > > > > testing:
> > > > > >
> > > > > > Whenever a socket is closed it enters a "Time Wait" state where
it
> > is
> > > > > > unavailable for reuse for up to about 4 minutes (2 times the
MSL -
> > > > Maximum
> > > > > > Segment Life (120secs)=240secs=4 minutes) - this is to avoid the
> > > > situation
> > > > > > where data arrives after the socket closed. If you immediately
> > reused
> > > > the
> > > > > > same port# for a new connection, it could potentially receive
some
> > old
> > > > > data.
> > > > > > So, the TCP stack picks new sockets from the pool between port
> 1024
> > > and
> > > > > 5000
> > > > > > (by default giving 3976 total sockets available).
> > > > > >
> > > > > > By quickly opening and closing many connections, you eat up all
> the
> > > > > sockets
> > > > > > before they can be recycled - its not a .NET specific problem.
> > > > > >
> > > > > > To check, from a command prompt use NETSTAT -a
> > > > > > this will display all connections and listening ports - I'll bet
> you
> > > see
> > > > > > 1000's of sockets with a status of TIME_WAIT
> > > > > >
> > > > > > The following articles might be of interest to you
> > > > > >
> > > > > > Why this occurs
> > > > > > http://support.microsoft.com/default.aspx?scid=kb;en-us;196271
> > > > > >
> > > > > > How to modify the maximum # of sockets available (not really
> > > > recommended)
> > > > > > http://support.microsoft.com/default.aspx?scid=kb;EN-US;149532
> > > > > >
> > > > > > I believe this is one of the reasons that DNS normally uses UDP
> > > packets
> > > > > > unless the answer is too big for a 512 byte packet - you might
> want
> > to
> > > > > > consider it. Also it saves the 3 packets connection setup plus
3
> > > packet
> > > > > > connection teardown overhead.
> > > > > >
> > > > > > Also, I hope your real code uses Asynchronous sockets with the
> > > > Threadpool
> > > > > > and timeouts - a misbehaving client could hang your code below
> > > > > > indefinitely - just power it off at the wrong time - right after
> it
> > > > > > connected but before it could make a request - users have a
knack
> > for
> > > > > these
> > > > > > sorts of thing :-)
> > > > > >
> > > > > > Note that there are problems with BeginAccept if you go the
> > > asynchronous
> > > > > > route - you can get yourself in the situation where you have no
> > > threads
> > > > > left
> > > > > > in the Threadpool to process Accepts because they are all busy
> doing
> > > > work
> > > > > > and you start rejecting connections. In a prior discussion
months
> > ago
> > > > the
> > > > > > consensus was to use a dedicated Accept thread making blocking
> calls
> > > > (such
> > > > > > as your code), but immediately hand off each accepted request
into
> > the
> > > > > > Threadpool for processing - via BeginReceive.
> > > > > > Also of note if running on a non-server O/S (such as XP Pro) the
> > > maximum
> > > > > > accept backlog is 5 pending connections (no matter what you put
> into
> > > the
> > > > > > parameter when you call Listen)! The 6th one will get back an
> > error.
> > > > > This
> > > > > > will sting you if you need to handle many simultaneous requests
> and
> > > are
> > > > > slow
> > > > > > about doing it.
> > > > > >
> > > > > > Hope this helps
> > > > > > Tom
> > > > > >
> > > > > >
> > > > > >
> > > > > > "Phillip O" <spam@vwars.com> wrote in message
> > > > > > news:%23gQF0D4EEHA.3040@TK2MSFTNGP12.phx.gbl...
> > > > > > > Hello,
> > > > > > > I have checked all the groups and messages and cannot find
this
> > > > > > > situation:
> > > > > > >
> > > > > > > I have created a synchrounous (and Asynchronous as a test)
> servers
> > > > that
> > > > > > are
> > > > > > > listening on port 11000 and ready to take a new connection.
> (this
> > > is
> > > > > > > test code for a DNS handler)
> > > > > > >
> > > > > > > I created the Listener socket, then Listen.bind and
> > > Listen.listen(10)
> > > > is
> > > > > > > called, I loop and block on the Listen.accept(); waiting for a
> new
> > > > > > > connection to be started by my clients.
> > > > > > >
> > > > > > > From the Listen.Accept, I get the new socket Handler and
receive
> > > > > > > data, looking for an end marker, then the server echos the
data
> > back
> > > > to
> > > > > > the
> > > > > > > client as an 'ack'.
> > > > > > >
> > > > > > > I then execute Handle.Shutdown(both) and Handle.close and go
> back
> > to
> > > > the
> > > > > > > top of the loop waiting for the next Listen.accept to unblock.
> > > > > > >
> > > > > > > After 3956 succesful connections and (data transfers), my
> clients
> > > get
> > > > a
> > > > > > > 10048 error,
> > > > > > > This error occurs for 70 seconds while the GC is running.
> > > > > > > Then all is well, and my clients programs can connect for
about
> > > > another
> > > > > > 3900
> > > > > > > connections. (total on the server side that is). (clients may
> > only
> > > > get
> > > > > a
> > > > > > > few hundred each).
> > > > > > >
> > > > > > > I have read all the GC stuff, and socket stuff. Tried to
ensure
> > > > > > > nothing is still un-fread or locked up and I have tried
forcing
> > the
> > > GC
> > > > > > > to run.
> > > > > > > Nothing works. I tried ReuseAddress and Linger(true) and
> > > > > > > Linger(false).
> > > > > > > Nothing Helps.
> > > > > > >
> > > > > > > My clients can't wait 70 seconds for the next connect if it
that
> > > > > > > client happens to be #3957.
> > > > > > >
> > > > > > > I have included most of the code below.
> > > > > > > THanks in advance.
> > > > > > > Phillip O.
> > > > > > >
> > > > > > > public static int StartListening()
> > > > > > > {
> > > > > > > // Data buffer for incoming data.
> > > > > > > byte[] bytes = new Byte[1024];
> > > > > > > int retval = 0;
> > > > > > > StringBuilder data = new StringBuilder();
> > > > > > > // Creates CompareInfo for the InvariantCulture.
> > > > > > > System.Globalization.CompareInfo myComp =
> > > > > > > CultureInfo.InvariantCulture.CompareInfo;
> > > > > > > int connectioncount = 0;
> > > > > > > // Establish the local endpoint for the socket.
> > > > > > > // Dns.GetHostName returns the name of the
> > > > > > > // host running the application.
> > > > > > > IPHostEntry ipHostInfo = Dns.Resolve(Dns.GetHostName());
> > > > > > > IPAddress ipAddress = ipHostInfo.AddressList[0];
> > > > > > > IPEndPoint localEndPoint = new IPEndPoint(ipAddress, 11000);
> > > > > > >
> > > > > > > // Create a TCP/IP socket. the Listern socket.
> > > > > > > using(Socket listener = new Socket(AddressFamily.InterNetwork,
> > > > > > > SocketType.Stream, ProtocolType.Tcp ))
> > > > > > > {
> > > > > > > data.Capacity = 100000; // a really big buffer!!!
> > > > > > > // Bind the sockets to the local endpoint and
> > > > > > > // listen for incoming connections.
> > > > > > > try
> > > > > > > {
> > > > > > > listener.Bind(localEndPoint);
> > > > > > > listener.Listen(10);
> > > > > > >
> > > > > > > // Start listening for connections.
> > > > > > > while (connectioncount < 500)
> > > > > > > {
> > > > > > > Console.WriteLine("Waiting for a connection...");
> > > > > > > // Program is suspended while waiting for an incoming
> connection.
> > > > > > > using(Socket handler = listener.Accept())
> > > > > > > {
> > > > > > > data.Length = 0; // reset the data.
> > > > > > > // An incoming connection needs to be processed.
> > > > > > > while (true)
> > > > > > > {
> > > > > > > int bytesRec = handler.Receive(bytes);
> > > > > > > data.Append(Encoding.ASCII.GetString(bytes,0,bytesRec));
> > > > > > > if (myComp.IndexOf (data.ToString(), "<EOF>") > -1)
> > > > > > > { break; }
> > > > > > > }
> > > > > > > // Show the data on the console.
> > > > > > > Console.WriteLine( "Text received : {0}", data.ToString());
> > > > > > > // Echo the data back to the client.
> > > > > > > try
> > > > > > > {
> > > > > > > byte[] msg = Encoding.ASCII.GetBytes(data.ToString());
> > > > > > > handler.Send(msg);
> > > > > > > handler.Shutdown(SocketShutdown.Both);
> > > > > > > handler.Close();
> > > > > > > }
> > > > > > > catch (Exception e)
> > > > > > > {
> > > > > > > Console.WriteLine(e.ToString());
> > > > > > > }
> > > > > > > } // end using handler.
> > > > > > > connectioncount++;
> > > > > > > GC.Collect();
> > > > > > > GC.WaitForPendingFinalizers();
> > > > > > > GC.GetTotalMemory(true);
> > > > > > > } // end while
> > > > > > > try
> > > > > > > {
> > > > > > > retval = 1; // Ending this set on the socket.
> > > > > > > // hopefully the gc will run now
> > > > > > > listener.Close();
> > > > > > > }
> > > > > > > catch (Exception e)
> > > > > > > {
> > > > > > > Console.WriteLine(e.ToString());
> > > > > > > }
> > > > > > > } // end try.
> > > > > > > catch (Exception e)
> > > > > > > {
> > > > > > > Console.WriteLine(e.ToString());
> > > > > > > retval = 2; // really having a problem.
> > > > > > > }
> > > > > > > Console.WriteLine("\nPort {0} CLosed", localEndPoint.Port);
> > > > > > > } // end user listener.
> > > > > > > GC.Collect();
> > > > > > > GC.WaitForPendingFinalizers();
> > > > > > > GC.GetTotalMemory(true);
> > > > > > > return(retval);
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

• Previous message: Joel Rumerman: "Re: Crystal Report Apply"
• In reply to: Phillip O: "Re: Problem with a Socket server program opening/accepting many connections and the GC is running."
• Messages sorted by: [ date ] [ thread ]