Re: Garbage Collection Issues in long-standing services

Willy, thanks for pointing this out, it does make sense concerning the Pinned
array.

In our case, the client should be fulfilling the requests in an immediate
fashion. If this is the case, the pinned array should get unpinned or
compacted since the async call completes pretty quickly, right?

The receive case is done quite well, I do create a 4K buffer and reuse it
for the life of the connection. In most cases, I don't expect an outstanding
begin receive at the end of the connection, but it could happen. I do keep
track of the IAsyncResults and if it is outstanding, I grab the state object
and remove the reference to my wrapper socket class, but I don't remove the
reference to the buffer. I should probably do this, right? The OS still has
a reference to the request object, can I just call EndReceive if it is
outstanding (not completed)? The one issue I see with this is if for some
reason the remote (client) socket hasn't closed (e.g. physical route is
broken, etc.), then EndReceive would block according to MSDN (framework 1.1).

The send case is a little more difficult and I would appreciate your advise
on it. I also have to support SSL and there is a maximum limit of 16K that
can be transfered at a time. I try to make life easy on the application
programmer by buffering this data if the amount is larger that 16K. I have a
class that my wrapper socket class references that contains an ArrayList of
byte arrays, so there could be more than one. Should I go to the exteme of
having a perminent send buffer and then just copy the data to this prior to
sending so that OS has only one reference to a byte array during the length
of the connection?

One other note, sorry about the long set of questions, I did use the CLR
Profiler using a simple application that mimics the references of my
TCPServer. My one worry was that my wrapper socket class did have a
reference to the TCPServer object (which is obviously long-standing) and I
was worried about the GC not collecting due to this. The TCPServer also has
a hashtable that references the wrapper socket object. I did verify that
when I removed the the wrapper socket from the hashtable that the GC wasn't
worried about the reference back to the TCPServer; I could explicitly set
this to null but feel that it is cleaner not having to do this throughout the
code. What's your opinion?


"Willy Denoyette [MVP]" wrote:

> Larry,
>
> The problem with sockets is the unmanaged memory buffers used by the
> underlying Winsock library, whenever you need to transfer a block of data
> (probably a byte array) to and from the unmanaged socket send/receive
> function, these arrays must be pinned. Now, when using asynchronous sockets,
> these arrays may stay pinned for a relatively long period of time, right.
> The problem with pinned objects (the arrays) is that they prevent the GC to
> compact the heap (pinned objects can't move!).
> The result is that, depending of the number of buffers (your State object?),
> you might end with a highly fragmented GC heap that keeps growing, not
> because the GC cannot collect but because he cannot compact.
> So what you need to do is review your design, and try to find out how many
> pinned objects you have in the youngest generations ((Gen0 and Gen1), these
> have the most impact). Try to prevent pinning for young objects by
> (pre)allocating your buffers very early in the process and use the same
> buffers for the whole run of the process, that way they will end in the Gen2
> after a few collector runs and stay there where they don't hurt the GC that
> much.
>
> Hope this helps.
> Willy.
>
>
>
> "Larry Herbinaux" <LarryHerbinaux@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in
> message news:7EFFDE15-6BB8-43DF-8A96-04200AB4A468@xxxxxxxxxxxxxxxx
> > Thanks for the reply.
> >
> > I would agree that I must be holding on to some references, I will try to
> > isolate. If the GC was really being aggressive, then why after everything
> > came back to steady state did the memory in use reduce by half? (800MB -
> > 350MB)
> >
> > I did run some tests on the classes that would give me the most problems
> > and
> > it appeared that the GC was smart enough to solve the issue. Here is a
> > brief
> > synopsis:
> >
> > TCPServer - Has a hashtable holding a class that encapsulates the client
> > socket (CS), including a refrence to a class that handles the application
> > processing (AP). The first thing AP does when it handles an OnConnect is
> > store a reference to CS so that it can use it to send data back to the
> > client.
> >
> > State Object (SO) - This holds a reference to CS and a pointer to a 4K
> > byte
> > array. This is passed on the Async Send / Receive Calls.
> >
> > Connection Termination - After the connection is closed (abnormally or
> > normally), the CS is removed from the hashtable. I've done test to see
> > that
> > once this is performed that the GC is smart enough to deal with the
> > circular
> > reference between CS and AP, so I don't explicitly set each reference to
> > null.
> >
> > Outstanding Async Send / Receive - Depending how the connection is
> > terminated, there is a chance that there can be an Outstanding Async Send
> > and
> > Receive. I store a reference to the most current IAsyncResult for both
> > the
> > Send and Receive in the CS object. When closing, I check to see if these
> > are
> > still outstanding and if so, I retrieve the SO object and set the CS
> > reference to null. I'm not setting the buffer to null and I probably
> > should.
> > I don't expect that this condition is normal, but I do my best at handling
> > it. You can't just stop an asyncronous request which is unfortunate
> > because
> > I'm sure there is a structure being stored in the OS to encapsulate each
> > (e.g. same structure that holds the state object).
> >
> > I will continue to try and isolate. Any ideas based on the above?
> >
> > "Willy Denoyette [MVP]" wrote:
> >
> >> Did you ever checked the GC performance counters (using perfmon) to check
> >> whether this is true? The GC is more aggressive than you imagine.
> >> Just check the Gen0 1 and 2 performance counters and you will see that
> >> the
> >> collector runs, your problem is that you are holding references to
> >> objects
> >> (probably large objects) which you never release, so there is little or
> >> nothing to collect.
> >> By starting another process that allocates memory, your service working
> >> set
> >> gets trimmed by the OS, that all that happens.
> >>
> >> Willy.
> >>
> >> "Larry Herbinaux" <LarryHerbinaux@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in
> >> message news:72FFED84-F740-456E-A986-0BB25C18ED39@xxxxxxxxxxxxxxxx
> >> > I'm having issues with garbage collection with my long-standing service
> >> > process. If you could review and point me in the right direction it
> >> > would
> >> > be
> >> > of great help. If there are any helpful documents that you could point
> >> > me
> >> > to
> >> > help me control the GC, then that would be great also.
> >> >
> >> > The .Net GC does not cleanup memory of our service process unless it is
> >> > forced to by another process that hogs memory.
> >> > · GC Algorithm - This is an issue because if the GC is not forced
> >> > into
> >> > doing
> >> > this, it does not aggressively cleanup until the amount of physical
> >> > memory
> >> > available is very small. I understand why it doesn’t want to
> >> > force
> >> > cleanup
> >> > due to processor efficiency, but it forces applications into conditions
> >> > that
> >> > are not acceptable. It would be nice to be able to hint an upper limit
> >> > for
> >> > an application that helps the GC be more aggressive when required.
> >> > · Race Condition – The GC Algorithm causes race conditions
> >> > because the
> >> > GC is
> >> > not coordinated with our application and our application throws
> >> > OutOfMemoryExceptions. We have very good exception handling that
> >> > guards
> >> > against unhandled exceptions in the main thread and thread pool
> >> > threads.
> >> > The
> >> > problem is that we use memory to log these issues so the handlers are
> >> > probably throwing another OutOfMemoryException. We can handle this,
> >> > but
> >> > the
> >> > point is the OutOfMemoryException cause a transaction to fail.
> >> > · Force GC To Collect - I wrote a Memory Hogger application which
> >> > when
> >> > run
> >> > will reduce the amount of memory used by the service application from
> >> > 800MB
> >> > to 3MB, so this proves that the GC will cleanup the memory when it
> >> > truly
> >> > needs it. I noticed that Memory Usage displayed in the Task Manager
> >> > Process
> >> > tab did not add up to the total amount of memory in use, so this means
> >> > that
> >> > the inactive applications probably moved their heap to swap. One
> >> > concern
> >> > here was that when the Memory Hogger application was terminated, our
> >> > service
> >> > application reclaimed half of its memory and we were not processing
> >> > transactions. Maybe GC just moved it to swap.
> >> > · GC.Collect – Using this is not recommended within the
> >> > application.
> >> > Even
> >> > when this is used, it doesn’t make the GC any more aggressive,
> >> > so I
> >> > agree,
> >> > there is no reason to use it. It would be nice to have the ability to
> >> > make
> >> > the GC more aggressive.
> >> > · CLR Profiler - I’ve used the CLR Profiler to determine what
> >> > memory is
> >> > not
> >> > being collected. Mostly string and byte arrays. Our service handles
> >> > TCP
> >> > Connections asynchronously and we reuse the same byte array when
> >> > receiving
> >> > data for the next asynchronous read. We store a reference to the
> >> > current
> >> > IAsyncResult for both the asynchronous send and receive requests. The
> >> > state
> >> > object holds the byte array which we are currently not setting to null.
> >> > There isn’t a way of canceling the async requests, so we might
> >> > have to
> >> > explicitly set this to null. I will try this to see if makes a
> >> > difference.
> >> > As for the strings, I’m not sure where the problem is here.
> >> >
> >>
> >>
> >>
>
>
>
.

Relevant Pages

  • Re: Garbage Collection Issues in long-standing services
    ... If your buffers are that small why does your working set grows to ... the pinned array should get unpinned or ... > and remove the reference to my wrapper socket class, ... > class that my wrapper socket class references that contains an ArrayList ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: VB-101: Passing Arrays ByVal vs ByRef
    ... changed if an array is passed by Val. ... ' new reference ... As each function creates a new array object, ... 'secondArray' and 'secondArrayCopy'. ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Need help with textboxes
    ... The JavaScript 1.5 Reference already states: ... All forms and their children are stored in an array ... use dot notation or object literals. ... No. Bracket property accessors allow their argument to be any string value. ...
    (comp.lang.javascript)
  • Re: How to give selective access to the methods in a class?
    ... different memory locations at different times. ... The reference still leads to the ... So array resizing ... program pushes certain objects into the circular buffer, ...
    (comp.lang.java.programmer)
  • Re: Perlish map() function
    ... the reference material, the Array constructor was introduced in JavaScript ... evaluation would result in a ReferenceError. ... identifier shows that it is true. ...
    (comp.lang.javascript)