Re: How to release heap memory that is marked as 'free'

BoundsChecker is a great product, but pricey; someone pointed you at a codeproject memory
tracker, which I had not seen before; I put in one of my apps last night and in a few
hours had fixed a lot of memory leaks (actually, they were tables that were created at
startup and didn't actually have to be cleaned up, but now they are).

Note that BoundsChecker will *not* help with memory fragmentation problems; your
heapwalker is an important tool in analyzing this, but doesn't help much in solving the
problem. Fragmentation is one of the truly serious problems, and there are a lot of
techniques for avoiding it, some of which are hard to implement in VS6 (but MFC in VS.NET
actually lets you override the default 'new' allocation mechanism, which gives you some
hope of dealing with it). The issue arises largely because of variable-size allocations,
so you can often reduce fragmentation by allocating only fixed-size blocks of a single
size (e.g., 256 bytes), which superficially looks wasteful, but because most blocks are
now the same size, there's always a hole of the right size. Multiples of block size for
larger allocations (e.g., 512, 1024, etc.) reduce (but don't necessarily eliminate)
fragmentation. Check out how CString buffers are allocated; in release mode they are
always 64, 128, 256 or 512 bytes long, or as large as needed for larger blocks.

There was a PhD disseration done by Chuck Weinstock back around 1974 that did a lot of
analysis of this phenomenon; my work which we reported in our book was based on his
original research, and tuned to within an inch of its life (and mine, too...it was hectic
at that point in my life).

To avoid fragmentation, you end up spending a *lot* of time analyzing your heapwalker
data. Since it is generally tedious-to-impossible to analyze these dumps manually, you
end up writing a lot of different programs to study the statistical behavior of the
allocations. Then you have to override the allocator to try to tune it. I've not tried
this in VS.NET, but that's where you have to start looking.
joe

On Sat, 15 Apr 2006 08:36:44 -0400, "Michael Evenson" <mevenson@xxxxxxxxxxxx> wrote:

Joe,

Thanks for the info. It sure looks like what you are saying is truely
the case. I will look for other ways to solve this problem. I am foing to
purchase Compuware Bounds Checker and see if that can shed any light on the
subject. The problem is that the 'leak' if there is one or fragmentation may
be caused by a dll that I have no control ober. I suspect it may be in the
VB Scripting automation that the program is using.

Thanks very much for the info
Mike


"Joseph M. Newcomer" <newcomer@xxxxxxxxxxxx> wrote in message
news:33p0421fon9b11i3ntiqcrfcf1qirja78i@xxxxxxxxxx
Wrong. What SetProcessWorkingSetSize does is screw up your process by
forcing all the
pages out. It doesn't free anything; it just gives the illusion of
freeing something, and
consequently it is actually a Really Bad Idea. I think it was Mark
Russinovich who coined
the term "fraudware" for programs that do this.

Believing any of those numbers leads to erroneous solutions for
non-problems. If you slow
down, it means you have badly fragmented memory, and "returning" memory to
the OS is not
going to solve things. You need to rethink the entire problem that is
causing the
fragmentation, not apply superficial non-solutions to the problem.

SetProcessWorkingSetSize is working as advertised; the problem is that it
isn't even
vaguely related to solving your problem, and its use is inappropriate.

I've spent a nontrivial part of my career writing and optimizing storage
allocators. You
need to understand why all this memory is being allocated. If you have a
gigabyte image,
you are doing a lot of allocations. Whie a heapwalker is very
informative, it cannot be
used in conjunction with any tools such as process viewer, manipuating the
working set, or
any similar tools; it completely invalidates their information. So you
need to understand
exactly where all that storage is going. It sounds to me like chronic
fragmentation
syndrome. And that takes a lot of work to fix. The fix will not ever
involve
SetProcessWorkingSetSize.
joe
On Thu, 13 Apr 2006 10:26:29 -0400, "Michael Evenson"
<mevenson@xxxxxxxxxxxx> wrote:

Tom,

I have found that using the SetProcessWorkingSetSize function does
immediately show the memory being returned in task manager. I set a
breakpoint just before the call and the memory usage for the process is
over
4MB. Immediately after the call, Task Manager shows the memory usage to be
around 130K which is what I would expect. When I then look at the heap
dump,
I still have a very large amount of 'FREED' memory that belongs to my
process's allocated memory pool. Then when I go to allocate a very small
amount of memory (say for a local CString), the memory usage in Task
Manager
shows application has the original 4MB of memory allocated. This 4MB value
eventually grows to over 1GB during program execution and it slows to a
crawl. By the way, I should mention that all of my calls to 'new' to
allocate memory that I have control over inside my application are done at
program startup and NOT within the loop where the memory seems to be
getting
allocated and released.

I am thinking that the SetProcessWorkingSetSize function is not
working
as advertised. Is this possible. I am checking the return code from
SetProcessWorkingSetSize and it is non-zero indicating that it was
successful.

Mike


"Tom Serface" <tserface@xxxxxxx> wrote in message
news:OF6$KOwXGHA.3868@xxxxxxxxxxxxxxxxxxxxxxx
Hi Michael,

I've seen the amount of memory for a process stay at a higher amount,
but
once the memory is freed (especially globally allocated memory) Windows
seems to consume it again for other programs as needed. Are you seeing
other program "not" getting memory after you free it? Do the same
programs work OK if you exit your program first? Could it be that
you've
allocated memory "around" the large block so the OS can't get it because
it's discontiguous? This article was interesting:

http://www.rsinc.com/services/techtip.asp?ttid=3346

Tom

"Michael Evenson" <mevenson@xxxxxxxxxxxx> wrote in message
news:5uGdnT3CS9jSz6PZRVn-vA@xxxxxxxxxxxxxxx

Actually the memory does still belong to the process, but has been
freed.
Apparently Windows does not reclaim the freed memory from the heap
until
it is needed. This is all well and good except that , if Windows cannot
find a large enough contiguous block of memory upon a request for
allocation, it prefers to use virtual memory over doing a garbage
collection to find more memory. The freed memory becomes part of my
process's 'working set'. Eventually, my process consumes so much
memory
in it's working set (which is marked as free since there is no leak)
that
all other process running on the machine have to get their memory from
the virtual memory pool.




Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
--
NewsGuy.Com 30Gb $9.95 Carry Forward and On Demand Bandwidth

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
--
NewsGuy.Com 30Gb $9.95 Carry Forward and On Demand Bandwidth
.

Relevant Pages

  • Re: [00/17] Large Blocksize Support V3
    ... Fragmentation when you multiple chunk sizes ... And grouping pages by mobility works best when the majority of memory is ... cannot large contiguous allocations. ... Lots of small requests are fundamental. ...
    (Linux-Kernel)
  • Re: [PATCH 1/2] MM: Make page tables relocatable -- conditional flush (rc9)
    ... allocations signficantly ... Is there a standard test used to evaluate kernel memory fragmentation? ... These tests are not suitable on machines with very large amounts of memory ...
    (Linux-Kernel)
  • RE: 2003 SBS stalling randomly
    ... A memory leak occurs in an application using the Volume Shadow Copy Service ... Poolmon displays data that the ... The data is grouped by pool allocation tag. ... Press P twice to display allocations from only the paged pool. ...
    (microsoft.public.windows.server.sbs)
  • Re: [Lhms-devel] [PATCH 0/7] Fragmentation Avoidance V19
    ... > fragmentation avoidance does not have a major performance penalty. ... It is complexity that is mostly already handled for us with the zones ... If you don't need to guarantee higher order allocations, ... > be under pressure even though there is memory available elsewhere. ...
    (Linux-Kernel)
  • Re: xmalloc string functions
    ... than a NULL return from malloc(). ... pointer value to null at the point I want to trigger the failure. ... I've also had VMWare report out-of-resource at times when the only resource that was tight was memory, and again it gave me the chance to recover the situation which saved me significant work because I had two VMs running and the state between them was important and took time setting up. ... allocations without reference to other circumstances (number of ...
    (comp.lang.c)
Loading