Re: CMemoryState & multi-threaded application
- From: Joseph M. Newcomer <newcomer@xxxxxxxxxxxx>
- Date: Mon, 02 Feb 2009 01:18:23 -0500
Yes, 100K is really, really small! But a repetitive allocation of blocks of ~100K,
released, and interspersed with smaller allocations, *will* cause fragmentation.
Fragmentation management is one of the serious problems that lead to "server decay", and
can be challenging. I've instrumented code by doing a heapwalk, recording the blocks
allocated and freed. It is incredibly tedious, requires writing programs to analyze the
trace output (use Python, Perl, Awk, or Your Favorite Scripting Language), but you can
quite often find the problem patterns. Fixing them can be challenging.
I've sometimes done this by having a thread that wakes up once a minute and traces memory.
You can get quite a lot of megabytes of trace output, and if you really have
fragmentation, you can sometimes find yourself in a very unpleasant place because there
may be no obvious fix for the problem. VS >= 2003 allow you to specify the allocator, but
I've never tried this or looked into too deeply. When I was programming in pure C, I
spent a lot of time creating an instrumented interface to malloc/free and never called
malloc or free directly. This is harder in C++, and much, much harder in MFC.
Note that for earlier versions of VS, if you did not define
#ifdef _DEBUG
#define new DEBUG_NEW
#endif
at the head of each file not generated by the framework, you won't get memory block
tracing for the storage you allocate inside that file.
You may safely ignore any data from any program outside your process, such as Task
Manager, Process Viewer, etc., since they are largely clueless about storage allocation
patterns.
I have generally found CMemoryState relatively useless, since it is based on a premise
which is not always satisfiable: that there is an invariant on the total amount of memory
in use, and a deviation from the amount represents a leak. Over a given interval, this
may be true, but determining that interval can be challenging (for example, the current
program I'm working on allocates over 500K items when working on a given file, but the net
memory usage is fixed at some size (I don't know what). So if I asked, for example, to
list all the memory blocks allocated-since, it would take a VERY long time to print them
out, and they wouldn't be particularly informative. So the only valid snapshots would be
before reading a file, and after, and the indication of fragmentation would be a large
collection of unallocated blocks of small size interspersed among allocated blocks of a
large size, and CMemoryState won't tell you that.
joe
On Fri, 30 Jan 2009 10:20:23 -0800, "Leo Violette" <nineballer@xxxxxxxxxxx> wrote:
Joseph M. Newcomer [MVP]
"Trevor" <trevor@xxxxxxxx> wrote in message
news:%23wo7wkvgJHA.3728@xxxxxxxxxxxxxxxxxxxxxxx
"Leo Violette" <nineballer@xxxxxxxxxxx> wrote in message
news:8E2880C9-4378-4A15-B25A-795EF4BF07D5@xxxxxxxxxxxxxxxx
Can I ask why your friend feels like there is a memory leak? Is there a
leak diagnostic getting dumped during program termination?
If so, I and probably many others here can give you tips on interpreting
that and pointers on pinpointing the leak.
She watches the "Memory Usage" in Task Manager increase over time (it
eventually stabilizes). I have begun to look into the design of the
program and saw some strange things. For example, half a dozen CWinThread
objects are constantly being dynamically created and destroyed to do the
same task over and over. I am beginning to think all of this constant
dynamic memory allocation is causing memory fragmentation. This is a
server program meant to run 24/7. I have suggested to re-design the
threads to be created once and run in a loop instead of being constantly
created/destroyed. I am also asking her to take out all of the
CMemoryState checks since we know those are not helping us much in this
usage. Then I will be able to see if MFC is dumping any memory leak
messages to the output window.
The Task Manager can be mis-leading when used. The fact that it climbs for
a while and then flattens out could be perfectly normal.
Also, what seems like a "high" memory usage to you and her may seem like a
normal memory usage to a lot of people here.
It's like imagining Mr. Newcomer eating Cheerios in the morning, reading
ms.public.vc.mfc in place of the daily paper. Choking on a
bite of cheerios upon reading the subject line: "Whats the best way to read
in a really large text file, like 100K?"
Then opening the message and to his dis-belief seeing complicated loop logic
code reading in 2K at a time into a buffer.
After recovering and getting slapped on the back one of his lovely family
members to clear his throat of those treacherous little O's, he
posts back how Not Big 100K is and how easily a computer built 5 years ago
could handle it, let alone modern hardware and that no
complicated elaborate logic is required to read it. Just allocate 100K and
read in the file all in one chunk and be done with it.
Point is, you didn't say specifically how much memory this application is
using. What sounds like a lot to you, might be fairly typical.
If you are sure you know it's a lot, then so be it. If you aren't you may
want to confirm that before starting to re-architect it.
Also, the architecture you proposed might inadvertently change the way the
program works.
For example, it sounds like the program currently handles up to a half-dozen
tasks simultaneously by constructing them all
at once. Your method of just creating one and running it in a loop would at
best cause serialization which might defeat the purpose
of using threads or at worst, might actually violate some assumptions made
in the code that these threads are happening simultaneously
and the program no longer functions correctly.
Good luck in investigating what is going on and determining if something
needs to be fixed or not. MFC will tell you if new or malloc
isn't be cleaned up. But it won't tell you if handles are leaking. So, you
may want to look for that kind of stuff (incorrect BSTR handling,
incorrect number of calls to CloseHandle, that kind of stuff).
Leo.
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
.
- Prev by Date: Re: CMemoryState & multi-threaded application
- Next by Date: read fail: sharing violation
- Previous by thread: Re: CMemoryState & multi-threaded application
- Next by thread: read fail: sharing violation
- Index(es):
Relevant Pages
|
