Re: Garbage Collection Issues in long-standing services

Larry,

The problem with sockets is the unmanaged memory buffers used by the
underlying Winsock library, whenever you need to transfer a block of data
(probably a byte array) to and from the unmanaged socket send/receive
function, these arrays must be pinned. Now, when using asynchronous sockets,
these arrays may stay pinned for a relatively long period of time, right.
The problem with pinned objects (the arrays) is that they prevent the GC to
compact the heap (pinned objects can't move!).
The result is that, depending of the number of buffers (your State object?),
you might end with a highly fragmented GC heap that keeps growing, not
because the GC cannot collect but because he cannot compact.
So what you need to do is review your design, and try to find out how many
pinned objects you have in the youngest generations ((Gen0 and Gen1), these
have the most impact). Try to prevent pinning for young objects by
(pre)allocating your buffers very early in the process and use the same
buffers for the whole run of the process, that way they will end in the Gen2
after a few collector runs and stay there where they don't hurt the GC that
much.

Hope this helps.
Willy.



"Larry Herbinaux" <LarryHerbinaux@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in
message news:7EFFDE15-6BB8-43DF-8A96-04200AB4A468@xxxxxxxxxxxxxxxx
> Thanks for the reply.
>
> I would agree that I must be holding on to some references, I will try to
> isolate. If the GC was really being aggressive, then why after everything
> came back to steady state did the memory in use reduce by half? (800MB -
> 350MB)
>
> I did run some tests on the classes that would give me the most problems
> and
> it appeared that the GC was smart enough to solve the issue. Here is a
> brief
> synopsis:
>
> TCPServer - Has a hashtable holding a class that encapsulates the client
> socket (CS), including a refrence to a class that handles the application
> processing (AP). The first thing AP does when it handles an OnConnect is
> store a reference to CS so that it can use it to send data back to the
> client.
>
> State Object (SO) - This holds a reference to CS and a pointer to a 4K
> byte
> array. This is passed on the Async Send / Receive Calls.
>
> Connection Termination - After the connection is closed (abnormally or
> normally), the CS is removed from the hashtable. I've done test to see
> that
> once this is performed that the GC is smart enough to deal with the
> circular
> reference between CS and AP, so I don't explicitly set each reference to
> null.
>
> Outstanding Async Send / Receive - Depending how the connection is
> terminated, there is a chance that there can be an Outstanding Async Send
> and
> Receive. I store a reference to the most current IAsyncResult for both
> the
> Send and Receive in the CS object. When closing, I check to see if these
> are
> still outstanding and if so, I retrieve the SO object and set the CS
> reference to null. I'm not setting the buffer to null and I probably
> should.
> I don't expect that this condition is normal, but I do my best at handling
> it. You can't just stop an asyncronous request which is unfortunate
> because
> I'm sure there is a structure being stored in the OS to encapsulate each
> (e.g. same structure that holds the state object).
>
> I will continue to try and isolate. Any ideas based on the above?
>
> "Willy Denoyette [MVP]" wrote:
>
>> Did you ever checked the GC performance counters (using perfmon) to check
>> whether this is true? The GC is more aggressive than you imagine.
>> Just check the Gen0 1 and 2 performance counters and you will see that
>> the
>> collector runs, your problem is that you are holding references to
>> objects
>> (probably large objects) which you never release, so there is little or
>> nothing to collect.
>> By starting another process that allocates memory, your service working
>> set
>> gets trimmed by the OS, that all that happens.
>>
>> Willy.
>>
>> "Larry Herbinaux" <LarryHerbinaux@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in
>> message news:72FFED84-F740-456E-A986-0BB25C18ED39@xxxxxxxxxxxxxxxx
>> > I'm having issues with garbage collection with my long-standing service
>> > process. If you could review and point me in the right direction it
>> > would
>> > be
>> > of great help. If there are any helpful documents that you could point
>> > me
>> > to
>> > help me control the GC, then that would be great also.
>> >
>> > The .Net GC does not cleanup memory of our service process unless it is
>> > forced to by another process that hogs memory.
>> > �· GC Algorithm - This is an issue because if the GC is not forced
>> > into
>> > doing
>> > this, it does not aggressively cleanup until the amount of physical
>> > memory
>> > available is very small. I understand why it doesn�t want to
>> > force
>> > cleanup
>> > due to processor efficiency, but it forces applications into conditions
>> > that
>> > are not acceptable. It would be nice to be able to hint an upper limit
>> > for
>> > an application that helps the GC be more aggressive when required.
>> > Ã?· Race Condition ââ?¬â?? The GC Algorithm causes race conditions
>> > because the
>> > GC is
>> > not coordinated with our application and our application throws
>> > OutOfMemoryExceptions. We have very good exception handling that
>> > guards
>> > against unhandled exceptions in the main thread and thread pool
>> > threads.
>> > The
>> > problem is that we use memory to log these issues so the handlers are
>> > probably throwing another OutOfMemoryException. We can handle this,
>> > but
>> > the
>> > point is the OutOfMemoryException cause a transaction to fail.
>> > �· Force GC To Collect - I wrote a Memory Hogger application which
>> > when
>> > run
>> > will reduce the amount of memory used by the service application from
>> > 800MB
>> > to 3MB, so this proves that the GC will cleanup the memory when it
>> > truly
>> > needs it. I noticed that Memory Usage displayed in the Task Manager
>> > Process
>> > tab did not add up to the total amount of memory in use, so this means
>> > that
>> > the inactive applications probably moved their heap to swap. One
>> > concern
>> > here was that when the Memory Hogger application was terminated, our
>> > service
>> > application reclaimed half of its memory and we were not processing
>> > transactions. Maybe GC just moved it to swap.
>> > Ã?· GC.Collect ââ?¬â?? Using this is not recommended within the
>> > application.
>> > Even
>> > when this is used, it doesn�t make the GC any more aggressive,
>> > so I
>> > agree,
>> > there is no reason to use it. It would be nice to have the ability to
>> > make
>> > the GC more aggressive.
>> > �· CLR Profiler - Iâ��ve used the CLR Profiler to determine what
>> > memory is
>> > not
>> > being collected. Mostly string and byte arrays. Our service handles
>> > TCP
>> > Connections asynchronously and we reuse the same byte array when
>> > receiving
>> > data for the next asynchronous read. We store a reference to the
>> > current
>> > IAsyncResult for both the asynchronous send and receive requests. The
>> > state
>> > object holds the byte array which we are currently not setting to null.
>> > There isn�t a way of canceling the async requests, so we might
>> > have to
>> > explicitly set this to null. I will try this to see if makes a
>> > difference.
>> > As for the strings, I�m not sure where the problem is here.
>> >
>>
>>
>>


.

Relevant Pages

  • Re: How to give selective access to the methods in a class?
    ... different memory locations at different times. ... The reference still leads to the ... So array resizing ... program pushes certain objects into the circular buffer, ...
    (comp.lang.java.programmer)
  • Re: Vector length in bytes
    ... An array of pointers is likely to be much smaller than the buffers it points to, but that would be the size of the array. ... The buffers OTOH are probably what you want transmit, but if you used a general memory counting algorithm, you'd likely have the size of the array of pointers included. ...
    (comp.lang.java.programmer)
  • Re: High Memory Consumption of Classes and Arrays
    ... > that classes and arrays in dotnet consume very much memory. ... you consider the reference as well). ... a reference to an array which contains the actual items. ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Any help about FIR filter algorithm
    ... > Can anyone tell me how i can get an output array from the following piece ... Please proviode memory allocations for your buffers. ... i am a bit lost about how to store my outputs in array. ...
    (comp.dsp)
  • Re: pinning types via Fixed
    ... > Now, for correctness, shouldn't he be locking the reference to the ... The array cannot be garbage collected since there is still a refererence on ... But it might be moved in memory when not beeing fixed so it is IMHO ...
    (microsoft.public.dotnet.languages.csharp)