Re: Calculating Virtual Memory Fragmentation

Both can happen, although the latter obiously only applies if you are using the heap manager.

For finding the largest free chunk, yes, take the largest free region that is returned by VirtualQuery. Of course, this considers all heap segments in use even if they have sub-blocks that are not yet in use, since VirtualQuery knows nothing of process heaps.

VM fragmentation usually only causes problems when you have mostly exhausted the available address space. The obvious excepton is if your program requires very large contiguous chunks of VM...

--
Ken Johnson (Skywing)
Windows SDK MVP
http://www.nynaeve.net
"ravenous_wolves" <ravenous.wolves@xxxxxxxxx> wrote in message news:1169580200.444802.289590@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
When does an allocation fail because of an fragmentation pattern?
a.) There is no contiguous virtual memory space large enough for the
allocation request?
b.) There is no contiguous space within a heap large enough for the
allocation request?

Either? Which one is more likely to occur first in a typical,
frequent-allocation, poorly-designed, fragmentation-inducing system?

A highly fragmented heap can simply grow in terms of pages and service
allocation requests, right? In this case, if there's tons of physical
memory, allocation would eventually fail when the heap is huge and
there is no remaining either virtual or physical/pagefile space and
another page can't be added. Within the heap, memory load reported for
the application could remain constant as fragmentation increases and
corresponding ineffeciency and actual amount of memory being FUBAR'ed
by the application increases?

What formula/algorithm can I use to calculate largest free VM block and
fragmentation percentage given the data that VirtualQuery gives me.
Early in the application life I get 270 blocks, 50MB committed, 20MB
free, 10MB reserved. Since VirtualQuery returns the whole range of
pages with the same protection & state, is it enough to just look at
every free block and remember the biggest one? What would this report
of VM frag represent if the total VM usage is only 80MB, and thus there
are still 2048-80 = 1968MB of VM pages still available to pull in? Is
fragmentation simply the existence of free VM blocks in the VM space,
as in, they shouldn't be in there at all in a theoretically perfect
world? Does VM fragmentation not really matter until
VMCommit+VMFree+VMReserve gets close to 2GB (VM non-kernel address
space w/o the /3GB switch)?

Thanks for everything,
-ken

Skywing [MVP] wrote:
1. The entire region of pages returned has the same protection and state
attributes, so you can skip ahead the size of the region for the next query.
2. The heap takes up 4k (or larger) regions and subdivides them into
smaller, more managable blocks. So even thought an entire heap segment
might be `committed` at the OS/VM-level, only the parts that the heap
manager has handed out to applications are truly `in-use.

BTW, check out the `!address -v' command in WinDbg; it has useful facilities
for diagnosing address space consumption (including reservations vs
commits).

--
Ken Johnson (Skywing)
Windows SDK MVP
http://www.nynaeve.net
"ravenous_wolves" <ravenous.wolves@xxxxxxxxx> wrote in message
news:1169505296.539000.205210@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> I'm seeing a problem in an long runtime application that I'm working
> which which I believe to be virtual memory fragmentation. A change was
> made which greatly increased the # of fast alloc/release cycles and the
> system MTBF dropped significantly and the failure profile was unable to
> allocate memory despite physical memory still being available.
>
> SO, I'm trying to quantify this effect and give some type of # which
> can report how fragmented the virtual address space of our process is.
> I've used a MS tool (DebugDiag) which actually reports this, but MS is
> unable to get this tool to work on our application (generating a dump
> locks the OS).
>
> I've got two ideas, but I'm stuck in terms of mapping my
> programmer-level concept of virtual memory to the real low-level
> implementation details of Windows.
> 1. Use VirtualQueryEx, and start at address 0x0 and work my way up to
> 0x80000000, and keep notes on the sizes of free blocks and the sizes of
> committed blocks. One question is, if VirtualQueryEx returns and says a
> "range of pages" from N to N + 2MB is committed, does that mean that a
> 2MB region is completely committed and 100% unfragmented, or does that
> mean that the set of pages is all committed, and each page has it's own
> fragmentation characteristics?
>
> 2. DebugDiag reports virtual memory fragmentation on a per-heap basis.
> I understand how to enumerate all the heaps in my process with
> GetProcessHeaps and HeapWalk. This seems right in terms of per-heap
> fragmentation, but I don't understand virtual memory fragmentation to
> be a per-heap concept, but rather a per-process concept. Furthermore,
> the Region field of PROCESS_HEAP_ENTRY indicates that part of the block
> is committed and part is uncommitted, which makes me confused as to
> what is being enumerated.
>
> Sorry if that doesn't make much sense, if anyone has some brief
> guidance on what kind of API calls and calculation would need to be
> done to quantify virtual memory fragmentation that would be great....
>
> Thanks,
> -ken
>


.

Relevant Pages

  • Re: Calculating Virtual Memory Fragmentation
    ... When does an allocation fail because of an fragmentation pattern ... Ite depends of the kind of allocation. ... For heap allocations greater then 0xFFFE * sizeof, ... To calculate the largest available virtual-address, ...
    (microsoft.public.win32.programmer.kernel)
  • Re: Heap fragmentation (2nd attempt)
    ... fragments the heap and measure the time it take to allocate memory. ... heap fragmentation depends on the malloc package much ...
    (comp.arch.embedded)
  • Heap fragmentation
    ... in University I learnt about heap fragmentation, ... over this procedure I cycle a hundred times. ... int main{ ...
    (comp.realtime)
  • Re: How to release heap memory that is marked as free
    ... fragmentation and you don't control the code then you can do little ... You could use a memory mapped file to ... I removed the heap walker calls and put in a heapstat function ... using, but I don't have much control over ZipArchive, and I have ...
    (microsoft.public.vc.mfc)
  • Re: Virtual Memory shooting up to 2 GB
    ... HeapValidate / HeapCompact do behave as you might expect, ... heap internal fragmentation and heap external fragmentation. ... Please help us with some information as to how can Virtual Memory ...
    (microsoft.public.win32.programmer.kernel)