Re: Problem with a Socket server program opening/accepting many connections and the GC is running.

From: Tom Hall (nospam_at_harmonyit.com)
Date: 04/02/04 

Date: Thu, 1 Apr 2004 19:32:17 -0500

I'm not quite sure what you mean by "slowed down my server by adding a 10ms
delay". Could you elaborate why this fixes the problem as normally you want
to handle the connection as quickly as possible?
Or, are you also running into the problem of 100% CPU usage preventing the
garbage collector from running as often as it needs to?

Thanks
Tom

"Phillip O" <spam@vwars.com> wrote in message
news:%23s5fGOzFEHA.2732@tk2msftngp13.phx.gbl...
> Thanks.
> This information helped me to understand and solve the problem. So far I
> have increased the amount of sockets available, and Slowed down my server
> (adding a 10ms delay). I now have pool of sockets available and can handle
> about 40 socket requests per second, from all clients.
>
> I will be adding some asynch recieve code soon because that is turning
into
> the next bottle neck. (other clients are starving if a large (> 1MB
message)
> is being handled by the server.)
>
> The echo back (send back to client) will be removed, I think, when this
code
> goes into production.
> Oddly enough, that seems to be handled offline, because while the server
is
> echoing back, other clients requests are being serviced.
>
> Thanks again,
> Phil O.
>
>
> "Tom Hall" <nospam@harmonyit.com> wrote in message
> news:Or9Px55EEHA.2804@tk2msftngp13.phx.gbl...
> > Well, 3956 is a suspicious number - quite close to the default maximum
> > available sockets. I suspect you are running into the following in your
> > testing:
> >
> > Whenever a socket is closed it enters a "Time Wait" state where it is
> > unavailable for reuse for up to about 4 minutes (2 times the MSL -
Maximum
> > Segment Life (120secs)=240secs=4 minutes) - this is to avoid the
situation
> > where data arrives after the socket closed. If you immediately reused
the
> > same port# for a new connection, it could potentially receive some old
> data.
> > So, the TCP stack picks new sockets from the pool between port 1024 and
> 5000
> > (by default giving 3976 total sockets available).
> >
> > By quickly opening and closing many connections, you eat up all the
> sockets
> > before they can be recycled - its not a .NET specific problem.
> >
> > To check, from a command prompt use NETSTAT -a
> > this will display all connections and listening ports - I'll bet you see
> > 1000's of sockets with a status of TIME_WAIT
> >
> > The following articles might be of interest to you
> >
> > Why this occurs
> > http://support.microsoft.com/default.aspx?scid=kb;en-us;196271
> >
> > How to modify the maximum # of sockets available (not really
recommended)
> > http://support.microsoft.com/default.aspx?scid=kb;EN-US;149532
> >
> > I believe this is one of the reasons that DNS normally uses UDP packets
> > unless the answer is too big for a 512 byte packet - you might want to
> > consider it. Also it saves the 3 packets connection setup plus 3 packet
> > connection teardown overhead.
> >
> > Also, I hope your real code uses Asynchronous sockets with the
Threadpool
> > and timeouts - a misbehaving client could hang your code below
> > indefinitely - just power it off at the wrong time - right after it
> > connected but before it could make a request - users have a knack for
> these
> > sorts of thing :-)
> >
> > Note that there are problems with BeginAccept if you go the asynchronous
> > route - you can get yourself in the situation where you have no threads
> left
> > in the Threadpool to process Accepts because they are all busy doing
work
> > and you start rejecting connections. In a prior discussion months ago
the
> > consensus was to use a dedicated Accept thread making blocking calls
(such
> > as your code), but immediately hand off each accepted request into the
> > Threadpool for processing - via BeginReceive.
> > Also of note if running on a non-server O/S (such as XP Pro) the maximum
> > accept backlog is 5 pending connections (no matter what you put into the
> > parameter when you call Listen)! The 6th one will get back an error.
> This
> > will sting you if you need to handle many simultaneous requests and are
> slow
> > about doing it.
> >
> > Hope this helps
> > Tom
> >
> >
> >
> > "Phillip O" <spam@vwars.com> wrote in message
> > news:%23gQF0D4EEHA.3040@TK2MSFTNGP12.phx.gbl...
> > > Hello,
> > > I have checked all the groups and messages and cannot find this
> > > situation:
> > >
> > > I have created a synchrounous (and Asynchronous as a test) servers
that
> > are
> > > listening on port 11000 and ready to take a new connection. (this is
> > > test code for a DNS handler)
> > >
> > > I created the Listener socket, then Listen.bind and Listen.listen(10)
is
> > > called, I loop and block on the Listen.accept(); waiting for a new
> > > connection to be started by my clients.
> > >
> > > From the Listen.Accept, I get the new socket Handler and receive
> > > data, looking for an end marker, then the server echos the data back
to
> > the
> > > client as an 'ack'.
> > >
> > > I then execute Handle.Shutdown(both) and Handle.close and go back to
the
> > > top of the loop waiting for the next Listen.accept to unblock.
> > >
> > > After 3956 succesful connections and (data transfers), my clients get
a
> > > 10048 error,
> > > This error occurs for 70 seconds while the GC is running.
> > > Then all is well, and my clients programs can connect for about
another
> > 3900
> > > connections. (total on the server side that is). (clients may only
get
> a
> > > few hundred each).
> > >
> > > I have read all the GC stuff, and socket stuff. Tried to ensure
> > > nothing is still un-fread or locked up and I have tried forcing the GC
> > > to run.
> > > Nothing works. I tried ReuseAddress and Linger(true) and
> > > Linger(false).
> > > Nothing Helps.
> > >
> > > My clients can't wait 70 seconds for the next connect if it that
> > > client happens to be #3957.
> > >
> > > I have included most of the code below.
> > > THanks in advance.
> > > Phillip O.
> > >
> > > public static int StartListening()
> > > {
> > > // Data buffer for incoming data.
> > > byte[] bytes = new Byte[1024];
> > > int retval = 0;
> > > StringBuilder data = new StringBuilder();
> > > // Creates CompareInfo for the InvariantCulture.
> > > System.Globalization.CompareInfo myComp =
> > > CultureInfo.InvariantCulture.CompareInfo;
> > > int connectioncount = 0;
> > > // Establish the local endpoint for the socket.
> > > // Dns.GetHostName returns the name of the
> > > // host running the application.
> > > IPHostEntry ipHostInfo = Dns.Resolve(Dns.GetHostName());
> > > IPAddress ipAddress = ipHostInfo.AddressList[0];
> > > IPEndPoint localEndPoint = new IPEndPoint(ipAddress, 11000);
> > >
> > > // Create a TCP/IP socket. the Listern socket.
> > > using(Socket listener = new Socket(AddressFamily.InterNetwork,
> > > SocketType.Stream, ProtocolType.Tcp ))
> > > {
> > > data.Capacity = 100000; // a really big buffer!!!
> > > // Bind the sockets to the local endpoint and
> > > // listen for incoming connections.
> > > try
> > > {
> > > listener.Bind(localEndPoint);
> > > listener.Listen(10);
> > >
> > > // Start listening for connections.
> > > while (connectioncount < 500)
> > > {
> > > Console.WriteLine("Waiting for a connection...");
> > > // Program is suspended while waiting for an incoming connection.
> > > using(Socket handler = listener.Accept())
> > > {
> > > data.Length = 0; // reset the data.
> > > // An incoming connection needs to be processed.
> > > while (true)
> > > {
> > > int bytesRec = handler.Receive(bytes);
> > > data.Append(Encoding.ASCII.GetString(bytes,0,bytesRec));
> > > if (myComp.IndexOf (data.ToString(), "<EOF>") > -1)
> > > { break; }
> > > }
> > > // Show the data on the console.
> > > Console.WriteLine( "Text received : {0}", data.ToString());
> > > // Echo the data back to the client.
> > > try
> > > {
> > > byte[] msg = Encoding.ASCII.GetBytes(data.ToString());
> > > handler.Send(msg);
> > > handler.Shutdown(SocketShutdown.Both);
> > > handler.Close();
> > > }
> > > catch (Exception e)
> > > {
> > > Console.WriteLine(e.ToString());
> > > }
> > > } // end using handler.
> > > connectioncount++;
> > > GC.Collect();
> > > GC.WaitForPendingFinalizers();
> > > GC.GetTotalMemory(true);
> > > } // end while
> > > try
> > > {
> > > retval = 1; // Ending this set on the socket.
> > > // hopefully the gc will run now
> > > listener.Close();
> > > }
> > > catch (Exception e)
> > > {
> > > Console.WriteLine(e.ToString());
> > > }
> > > } // end try.
> > > catch (Exception e)
> > > {
> > > Console.WriteLine(e.ToString());
> > > retval = 2; // really having a problem.
> > > }
> > > Console.WriteLine("\nPort {0} CLosed", localEndPoint.Port);
> > > } // end user listener.
> > > GC.Collect();
> > > GC.WaitForPendingFinalizers();
> > > GC.GetTotalMemory(true);
> > > return(retval);
> > > }
> > >
> > >
> >
> >
>
>