Re: How to release heap memory that is marked as 'free'

What you need to see is the size of the free blocks. Generally you need to sort these
into "buckets": I have so many blocks of size this, so many of size this+1 (where +1 is
the next allocation quantum), and so on. Then you have to study how many times large
blocks are allocated. Note that if there is some third-party library which is using
LocalAlloc, GlobalAlloc, VirtualAlloc, or HeapAlloc off a private heap then none of these
values will show up in a standard C-library heap-walker, and these could be what you are
suffering from, in addition to fragmentation.

As I said, fragmentation is a very serious problem, and one of the most serious problems
you can hit in long-running apps. Dealing with it is nontrivial, particularly in the
presence of third-party libraries, over which you have no control.

At one time, my allocator was accused of using massive amounts of memory. It turns out
that on each page of the listing, the compiler put out a header with the name of the
logged-in user as part of the page header. It turned out that each time the system API
for "get user ID" was called on that OS, it consumed another 64K block of data which it
did not release. By the end of a long compilation, we had consumed many tens of megabytes
of data space. The solution, of course, was to call the API once and cache the result,
but it took us a couple weeks to track down the actual problem. I spent a *lot* of time
writing diagnostic software that used really weird system calls to retrieve things like
the list of known, allocated pages from the kernel. This one glitch probably cost me 100
hours of programming/debugging/analysis effort. And the problem was actually part of the
OS API for that OS.
joe

On Sun, 16 Apr 2006 13:31:38 -0400, "Michael Evenson" <mevenson@xxxxxxxxxxxx> wrote:

Joe,

I installed a trial copy of BoundsChecker and it found absolutely no
memory leaks. I removed the heap walker calls and put in a heapstat function
that I call instead from within the loop where I feel the memory is getting
fragmented. All the heapstat function does is traverse the heap adding up
the used and freed blocks into separate accumlators. I then output these
values every 100 times through the loop. Both accumulators keep growing in
size (about 7K for the used memory for every 100 records processed - the
freed size seems to stabalize around 500+KB). I am using the ZipArchive
library, ADODB and the VisualBasic Scripting host in the loop. The scripting
host is used to massage the output, ADODB is getting a piece of data from a
database, and ZipArchive is compressing the output record. I thought that
there might be a leak in one of these pieces, but according to
BoundsChecker, there is not. They just seem to be fragmenting the crap out
of the memory. I'm going to have to re-think the memory allocation that I'm
using, but I don't have much control over ZipArchive (but some), and I have
absolutely no control over ADODB or the VBS host engine. The count of the
number of records processed is now over 410,000 per day and growing.

Thanks for all your input. It is greatly appreciated.
Mike


"Joseph M. Newcomer" <newcomer@xxxxxxxxxxxx> wrote in message
news:s8e342580ae1l8e6c0aoaldgk8302let9j@xxxxxxxxxx
BoundsChecker is a great product, but pricey; someone pointed you at a
codeproject memory
tracker, which I had not seen before; I put in one of my apps last night
and in a few
hours had fixed a lot of memory leaks (actually, they were tables that
were created at
startup and didn't actually have to be cleaned up, but now they are).

Note that BoundsChecker will *not* help with memory fragmentation
problems; your
heapwalker is an important tool in analyzing this, but doesn't help much
in solving the
problem. Fragmentation is one of the truly serious problems, and there
are a lot of
techniques for avoiding it, some of which are hard to implement in VS6
(but MFC in VS.NET
actually lets you override the default 'new' allocation mechanism, which
gives you some
hope of dealing with it). The issue arises largely because of
variable-size allocations,
so you can often reduce fragmentation by allocating only fixed-size blocks
of a single
size (e.g., 256 bytes), which superficially looks wasteful, but because
most blocks are
now the same size, there's always a hole of the right size. Multiples of
block size for
larger allocations (e.g., 512, 1024, etc.) reduce (but don't necessarily
eliminate)
fragmentation. Check out how CString buffers are allocated; in release
mode they are
always 64, 128, 256 or 512 bytes long, or as large as needed for larger
blocks.

There was a PhD disseration done by Chuck Weinstock back around 1974 that
did a lot of
analysis of this phenomenon; my work which we reported in our book was
based on his
original research, and tuned to within an inch of its life (and mine,
too...it was hectic
at that point in my life).

To avoid fragmentation, you end up spending a *lot* of time analyzing your
heapwalker
data. Since it is generally tedious-to-impossible to analyze these dumps
manually, you
end up writing a lot of different programs to study the statistical
behavior of the
allocations. Then you have to override the allocator to try to tune it.
I've not tried
this in VS.NET, but that's where you have to start looking.
joe

On Sat, 15 Apr 2006 08:36:44 -0400, "Michael Evenson"
<mevenson@xxxxxxxxxxxx> wrote:

Joe,

Thanks for the info. It sure looks like what you are saying is truely
the case. I will look for other ways to solve this problem. I am foing to
purchase Compuware Bounds Checker and see if that can shed any light on
the
subject. The problem is that the 'leak' if there is one or fragmentation
may
be caused by a dll that I have no control ober. I suspect it may be in the
VB Scripting automation that the program is using.

Thanks very much for the info
Mike


"Joseph M. Newcomer" <newcomer@xxxxxxxxxxxx> wrote in message
news:33p0421fon9b11i3ntiqcrfcf1qirja78i@xxxxxxxxxx
Wrong. What SetProcessWorkingSetSize does is screw up your process by
forcing all the
pages out. It doesn't free anything; it just gives the illusion of
freeing something, and
consequently it is actually a Really Bad Idea. I think it was Mark
Russinovich who coined
the term "fraudware" for programs that do this.

Believing any of those numbers leads to erroneous solutions for
non-problems. If you slow
down, it means you have badly fragmented memory, and "returning" memory
to
the OS is not
going to solve things. You need to rethink the entire problem that is
causing the
fragmentation, not apply superficial non-solutions to the problem.

SetProcessWorkingSetSize is working as advertised; the problem is that
it
isn't even
vaguely related to solving your problem, and its use is inappropriate.

I've spent a nontrivial part of my career writing and optimizing storage
allocators. You
need to understand why all this memory is being allocated. If you have
a
gigabyte image,
you are doing a lot of allocations. Whie a heapwalker is very
informative, it cannot be
used in conjunction with any tools such as process viewer, manipuating
the
working set, or
any similar tools; it completely invalidates their information. So you
need to understand
exactly where all that storage is going. It sounds to me like chronic
fragmentation
syndrome. And that takes a lot of work to fix. The fix will not ever
involve
SetProcessWorkingSetSize.
joe
On Thu, 13 Apr 2006 10:26:29 -0400, "Michael Evenson"
<mevenson@xxxxxxxxxxxx> wrote:

Tom,

I have found that using the SetProcessWorkingSetSize function does
immediately show the memory being returned in task manager. I set a
breakpoint just before the call and the memory usage for the process is
over
4MB. Immediately after the call, Task Manager shows the memory usage to
be
around 130K which is what I would expect. When I then look at the heap
dump,
I still have a very large amount of 'FREED' memory that belongs to my
process's allocated memory pool. Then when I go to allocate a very small
amount of memory (say for a local CString), the memory usage in Task
Manager
shows application has the original 4MB of memory allocated. This 4MB
value
eventually grows to over 1GB during program execution and it slows to a
crawl. By the way, I should mention that all of my calls to 'new' to
allocate memory that I have control over inside my application are done
at
program startup and NOT within the loop where the memory seems to be
getting
allocated and released.

I am thinking that the SetProcessWorkingSetSize function is not
working
as advertised. Is this possible. I am checking the return code from
SetProcessWorkingSetSize and it is non-zero indicating that it was
successful.

Mike


"Tom Serface" <tserface@xxxxxxx> wrote in message
news:OF6$KOwXGHA.3868@xxxxxxxxxxxxxxxxxxxxxxx
Hi Michael,

I've seen the amount of memory for a process stay at a higher amount,
but
once the memory is freed (especially globally allocated memory)
Windows
seems to consume it again for other programs as needed. Are you
seeing
other program "not" getting memory after you free it? Do the same
programs work OK if you exit your program first? Could it be that
you've
allocated memory "around" the large block so the OS can't get it
because
it's discontiguous? This article was interesting:

http://www.rsinc.com/services/techtip.asp?ttid=3346

Tom

"Michael Evenson" <mevenson@xxxxxxxxxxxx> wrote in message
news:5uGdnT3CS9jSz6PZRVn-vA@xxxxxxxxxxxxxxx

Actually the memory does still belong to the process, but has been
freed.
Apparently Windows does not reclaim the freed memory from the heap
until
it is needed. This is all well and good except that , if Windows
cannot
find a large enough contiguous block of memory upon a request for
allocation, it prefers to use virtual memory over doing a garbage
collection to find more memory. The freed memory becomes part of my
process's 'working set'. Eventually, my process consumes so much
memory
in it's working set (which is marked as free since there is no leak)
that
all other process running on the machine have to get their memory
from
the virtual memory pool.




Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
--
NewsGuy.Com 30Gb $9.95 Carry Forward and On Demand Bandwidth

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
--
NewsGuy.Com 30Gb $9.95 Carry Forward and On Demand Bandwidth

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
--
NewsGuy.Com 30Gb $9.95 Carry Forward and On Demand Bandwidth
.

Relevant Pages

  • Re: [PATCH 00/28] Swap over NFS -v16
    ... memory they can consume. ... So we need the extra (skb) ... included in the reserve? ... if the allocation had to dip into emergency reserves, ...
    (Linux-Kernel)
  • Re: CMemoryState & multi-threaded application
    ... released, and interspersed with smaller allocations, *will* cause fragmentation. ... I've sometimes done this by having a thread that wakes up once a minute and traces memory. ... Manager, Process Viewer, etc., since they are largely clueless about storage allocation ...
    (microsoft.public.vc.mfc)
  • Re: Memory leak with CAsyncSocket::Create
    ... read my essay on how storage allocators work. ... Create method is consuming system memory that is not released back to ... The memory consumption is either shown as "Mem Usage" on the Task ... many levels of allocation going ...
    (microsoft.public.vc.mfc)
  • Growing Working Set Size on GUI App
    ... snapshots of the heap at various time and will display the ... memory keeps ... For some reason, the working set ... >when there had been a large amount of allocation between ...
    (microsoft.public.dotnet.framework.performance)
  • Re: OT: C++ overloading operators
    ... dynamic allocation, no matter how many "clever tricks" are used... ... though there's enough memory in the system, ... all these "flexible data types" map into CPU command ... The computing environment is completely ...
    (comp.dsp)