Re: Calculating Virtual Memory Fragmentation

When does an allocation fail because of an fragmentation pattern?
a.) There is no contiguous virtual memory space large enough for the
allocation request?
b.) There is no contiguous space within a heap large enough for the
allocation request?

Either? Which one is more likely to occur first in a typical,
frequent-allocation, poorly-designed, fragmentation-inducing system?

A highly fragmented heap can simply grow in terms of pages and service
allocation requests, right? In this case, if there's tons of physical
memory, allocation would eventually fail when the heap is huge and
there is no remaining either virtual or physical/pagefile space and
another page can't be added. Within the heap, memory load reported for
the application could remain constant as fragmentation increases and
corresponding ineffeciency and actual amount of memory being FUBAR'ed
by the application increases?

What formula/algorithm can I use to calculate largest free VM block and
fragmentation percentage given the data that VirtualQuery gives me.
Early in the application life I get 270 blocks, 50MB committed, 20MB
free, 10MB reserved. Since VirtualQuery returns the whole range of
pages with the same protection & state, is it enough to just look at
every free block and remember the biggest one? What would this report
of VM frag represent if the total VM usage is only 80MB, and thus there
are still 2048-80 = 1968MB of VM pages still available to pull in? Is
fragmentation simply the existence of free VM blocks in the VM space,
as in, they shouldn't be in there at all in a theoretically perfect
world? Does VM fragmentation not really matter until
VMCommit+VMFree+VMReserve gets close to 2GB (VM non-kernel address
space w/o the /3GB switch)?

Thanks for everything,
-ken

Skywing [MVP] wrote:
1. The entire region of pages returned has the same protection and state
attributes, so you can skip ahead the size of the region for the next query.
2. The heap takes up 4k (or larger) regions and subdivides them into
smaller, more managable blocks. So even thought an entire heap segment
might be `committed` at the OS/VM-level, only the parts that the heap
manager has handed out to applications are truly `in-use.

BTW, check out the `!address -v' command in WinDbg; it has useful facilities
for diagnosing address space consumption (including reservations vs
commits).

--
Ken Johnson (Skywing)
Windows SDK MVP
http://www.nynaeve.net
"ravenous_wolves" <ravenous.wolves@xxxxxxxxx> wrote in message
news:1169505296.539000.205210@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I'm seeing a problem in an long runtime application that I'm working
which which I believe to be virtual memory fragmentation. A change was
made which greatly increased the # of fast alloc/release cycles and the
system MTBF dropped significantly and the failure profile was unable to
allocate memory despite physical memory still being available.

SO, I'm trying to quantify this effect and give some type of # which
can report how fragmented the virtual address space of our process is.
I've used a MS tool (DebugDiag) which actually reports this, but MS is
unable to get this tool to work on our application (generating a dump
locks the OS).

I've got two ideas, but I'm stuck in terms of mapping my
programmer-level concept of virtual memory to the real low-level
implementation details of Windows.
1. Use VirtualQueryEx, and start at address 0x0 and work my way up to
0x80000000, and keep notes on the sizes of free blocks and the sizes of
committed blocks. One question is, if VirtualQueryEx returns and says a
"range of pages" from N to N + 2MB is committed, does that mean that a
2MB region is completely committed and 100% unfragmented, or does that
mean that the set of pages is all committed, and each page has it's own
fragmentation characteristics?

2. DebugDiag reports virtual memory fragmentation on a per-heap basis.
I understand how to enumerate all the heaps in my process with
GetProcessHeaps and HeapWalk. This seems right in terms of per-heap
fragmentation, but I don't understand virtual memory fragmentation to
be a per-heap concept, but rather a per-process concept. Furthermore,
the Region field of PROCESS_HEAP_ENTRY indicates that part of the block
is committed and part is uncommitted, which makes me confused as to
what is being enumerated.

Sorry if that doesn't make much sense, if anyone has some brief
guidance on what kind of API calls and calculation would need to be
done to quantify virtual memory fragmentation that would be great....

Thanks,
-ken


.

Relevant Pages

  • Re: XP keeps freezing up
    ... Volume fragmentation ... Try Ctrl+Alt+Delete to select Task Manager and click the Performance ... the boxes before Peak Memory Usage and Virtual Memory size. ... You can access Event Viewer by selecting Start, Control Panel, ...
    (microsoft.public.windowsxp.perform_maintain)
  • Re: scf /scannow & INFO in Event Viewer
    ... Diskeeper has completed a defragmentation run on this ... Little or no fragmentation detected on this volume. ... use / free / unused memory ... total virtual memory allocated ...
    (microsoft.public.windowsxp.general)
  • Re: Exchange, Virtual Memory, too much RAM? Event ID 9582
    ... Troubleshoot Virtual Memory Fragmentation in Exchange ... How to Modify the Store Database Maximum Cache Size ...
    (microsoft.public.backoffice.smallbiz2000)
  • Re: Exchange, Virtual Memory, too much RAM? Event ID 9582
    ... Troubleshoot Virtual Memory Fragmentation in Exchange ... How to Modify the Store Database Maximum Cache Size ...
    (microsoft.public.exchange2000.admin)
Loading