Re: Reproducable Memory Leak using simple value types

From: AlexS (salexru2000NO_at_SPAMsympaticoPLEASE.ca)
Date: 06/25/04 

Date: Fri, 25 Jun 2004 13:29:51 -0400

Hi, Grouchy

I check your sample - slightly modified it. One button allocated memory,
another - called GC methods.
Clicking both buttons I got that VM allocation got to 100MB and after button
click with GC calls it jumped down to around 8MB - initial allocation for my
form.

Conclusion is - when you call GC immediately after allocation you don't give
enough time to GC to find out what and where to collect. MS docs are
specific in this respect - GC is not guaranteed to collect all inaccessible
memory. You must allow for sometimes several GC cycles to free all available
memory and even then some might stay. Depends on application - I've seen
some cases when I had huge heap with thin slices of allocations between huge
holes. Then I seen 100MB of VM while in reality only 10MB were "used". Alas,
relocation also requires CPU cycles. So, as you can see style of coding
influences GC behavior in unexpected ways.

Happy coding
Alex

"Grouchy" <Grouchy@discussions.microsoft.com> wrote in message
news:70BF7148-F613-437B-9740-D4F8BFEE1BFB@microsoft.com...
> Smiles,
>
> I have narrowed down the memory leak. The leak seems to only effect
Virtual Memory. Any virtual memory consumed during array allocation or
consumed when populating arrays with variable sized objects is not released.
Physical memory is GC'd correctly but the VM is dead forever.
>
> Would you mind running the snippet at the bottom and giving me your
analysis of it. You should see a ~100MB VM allocation and a ~100MB physical
memory allocation. Minimizing the app releases the physical memory but not
the VM. Repeatedly running the same code will not result in additional VM
allocation. It seems that if an ARRAY is involved with VM allocation, the
new PEAK VM usage is held forever by the application.
>
> private void button1_Click(object sender, System.EventArgs e)
> {
> byte[] array = new byte[100000000];
> array = null;
> GC.Collect();
> GC.WaitForPendingFinalizers();
> }
>
>
>
> "AlexS" wrote:
>
> > Hi, Grouchy
> >
> > I think you might want to take a look under the hood - both for C# code
and
> > W32. Take a look which API functions are used in W32 version to allocate
and
> > free memory and which are used for Queue in .Net. There are differences,
> > which could be considered as "bad" or "good" depending on criteria used.
> > These differences are coming from both compilers and environments.
However
> > they do not mean you can't repeat W32 code in .Net using PInvoke and get
> > same results if you need them. Just note that using .Net components you
> > modify your sample in a way, which makes comparison somewhat imprecise.
> >
> > Also, note that 300% increase was for my application (forms, controls
etc.
> > in addition to sample code), so it says not much about the test loop
itself.
> > When you use .Net components you have to accept that their behavior is
not
> > directly translatable into W32 - GC here is one of entities, which
influence
> > heap and has no direct counterpart in W32. Array is not simple type
anymore
> > too. So, comparison and conclusions should be more "balanced". That's
why I
> > stay unconvinced.
> >
> > At the same time I do not mean that .Net is ideal. Not for such
examples.
> > I've seen some also some unpleasant things with Marshal.PtrToStructure
and
> > others, especially under stress. That's why I think profiling should be
> > mandatory if you have concerns. That's why PInvoke exists. As I never
seen
> > such code in working applications I do continue to think that sample is
> > careless - if you do something wrong in W32 world do you expect that
Windows
> > will correct itself? You accept API limitations and look for a way
around,
> > isn't it so? You did not get expected result with your sample - possibly
you
> > should modify expectations or sample?
> >
> > Finally, me too I would prefer to have your sample working as expected
:-)
> > Maybe new version will do that?
> >
> > Alex
> >
> > "Grouchy" <Grouchy@discussions.microsoft.com> wrote in message
> > news:5793CBF5-3CAC-421E-AC74-A0D90F218DBE@microsoft.com...
> > > So in a nutshell...
> > >
> > > Increasing the initial memory consumption by 300% and never releasing
it
> > is a good thing? "Does it say anything about quality of CLR and GC???"
Its
> > pretty obvious what it says about the quality of the CLR and the GC.
> > >
> > > The code is simple test code of simple types (and a Queue) and it
works
> > flawlessly under straight W32 code without memory consumption.
> > >
> > > There is nothing careless about the allocation. It is a blatant
attempt to
> > force the system to allocate a god aweful amount of memory in a very
short
> > time, in a VERY straight forward manner, and then clean up after itself.
> > There are no objects other than the System.Collections.Queue and
> > System.Value types. This code should not leak.
> > >
> > > Concerning your 10GB buffer into 512k memory. Explain why straight W32
> > code does not have a problem with this. C#/ CLR does.
> > >
> > > I know my example using C# uses a Queue. I have also tested it using
an
> > Array. Same problem.
> > >
> > > Create a W32 project and cut and past the following where into the
> > appropriate areas. You can run this all day and for all practical
purposes
> > it won't leak. J = 10 takes 5 to 10 minutes and consumes roughly 1GIG of
> > virtual memory and 500MB of RAM. But at the end it cleans itself up.
> > > BYTE *array[100000];
> > >
> > > for(int j = 0; j < 10; j++)
> > > {
> > > for(int i = 0; i < 100000; i++)
> > > array[i] = new BYTE[10000];
> > > for(int i = 0; i < 100000; i++)
> > > delete[] array[i];
> > > }
> > >
> > > Bottom line- If the GC worked there would not be a need for CLR memory
> > profiler.
> > >
> > >
> > >
> > >
> > >
> > > "AlexS" wrote:
> > >
> > > > Apart from the fact that this kind of code nobody ever will use in
real
> > > > application, I would like to note that sample is fairly
demonstrative of
> > GC
> > > > behavior. However this doesn't mean it is "black hole" behavior.
E.g. on
> > one
> > > > of my test machines this sample causes immediate memory overflow
> > exception.
> > > > In another case it eats up memory up to the available limit and
stays
> > there.
> > > > In yet another after clicking button 2 memory is released and drops
back
> > to
> > > > approx. 3 times more than initial allocation but in all cases to
much
> > lower
> > > > level than immediately after allocation loop.
> > > >
> > > > Does it say anything about quality of CLR and GC? I doubt. This is
just
> > a
> > > > good example of "miss-management" of memory by programmer. And
another
> > > > example of what happens when allocations are done in careless way.
> > Honestly,
> > > > I don't think there are many other environments, which will survive
this
> > > > kind of testing without problems.
> > > >
> > > > Another question: do you read 10GB file in memory buffer when you
have
> > only
> > > > 512MB available? If you get somewhat "surprising" results do you
> > conclude
> > > > that environment is to blame?
> > > >
> > > > HTH
> > > > Alex
> > > >
> > > > > "Grouchy" <Grouchy@discussions.microsoft.com> wrote in message
> > > > > news:A9BB9866-B54F-428F-80AE-E19D8C6573B3@microsoft.com...
> > > > > > Create a new windows project. Add 2 buttons. System I am
running on
> > has
> > > > > 1GIG of ram. Click Button1 and watch your virtual memory usage
> > increase
> > > > from
> > > > > 7MB to 39MB permanently. Do a round of click1, click2, etc, etc
and
> > watch
> > > > > your virtual memory usage grow permanently. As far as I can
determine,
> > the
> > > > > whole System.Collections namespace is a memory black hole.
> > > > > >
> > > > > > I was ready to jump in bed with .NET but after a few months of
> > testing
> > > > all
> > > > > I can say is "I can miss-management my memory all by myself
without
> > the
> > > > > assistance of the CLR".
> > > > > >
> > > > > >
> > > > > > private void button1_Click(object sender, System.EventArgs e)
> > > > > > {
> > > > > > Queue queue = new Queue();
> > > > > > for(int i = 0; i < 100000; i++)
> > > > > > queue.Enqueue(new byte[10000]);
> > > > > >
> > > > > > queue.Clear();
> > > > > > queue.TrimToSize();
> > > > > > queue = null;
> > > > > > }
> > > > > >
> > > > > >
> > > > > > private void button2_Click(object sender, System.EventArgs e)
> > > > > > {
> > > > > > GC.Collect();
> > > > > > GC.WaitForPendingFinalizers();
> > > > > > }
> > > > >
> > > >
> > > >
> > > >
> >
> >
> >

Loading