Re: Still confused why working set larger than virtual memory

Hi Ondrej,


I have studied your sample code, and modify it to show my points. In your
test program, the working set is low because of each time after you use the
portion of the memory map file, you close it.

In my scenario, I keep it open. Here is my code based on your code and you
can see working set is much higher than virtual bytes, this is the situation
I described in my question. Any ideas?

(Here is my code, if there is anything wrong with my code, please also feel
free to let me know.)

#include <windows.h>
#include <stdio.h>

int main(int argc, char* argv[])
{
LARGE_INTEGER start,end;
LARGE_INTEGER freq;
QueryPerformanceCounter(&start);
QueryPerformanceFrequency(&freq);

MEMORYSTATUS memstat;
void* map;
memstat.dwLength = sizeof(memstat);
GlobalMemoryStatus(&memstat);

// basic file mapping test (512 MB)
long long size = 512*1024*1024;

HANDLE mapping =
CreateFileMapping(NULL,NULL,PAGE_READWRITE|SEC_COMMIT,(DWORD)(size>>32),DWORD(size),NULL);
if (mapping)
{
// create and destroy temporary views
SYSTEM_INFO sysInfo;
GetSystemInfo(&sysInfo);
const int allocSize = sysInfo.dwAllocationGranularity;

GlobalMemoryStatus(&memstat);

void *mem = new char[allocSize];
memset(mem,0x11,allocSize);
for (int i=0; i<10; i++)
{
for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
map =
MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map)
{
memcpy(map,mem,allocSize);
// UnmapViewOfFile(map);
}
}

if (map)
{
UnmapViewOfFile(map);
}

GlobalMemoryStatus(&memstat);


for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
map =
MapViewOfFile(mapping,FILE_MAP_READ,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map)
{
for (int t=0; t<allocSize; t++)
{
if (((char *)map)[t]!=0x11)
{
OutputDebugString("Memory read failed\n");
}
}

// UnmapViewOfFile(map);
}
}
if (map)
{
UnmapViewOfFile(map);
}

GlobalMemoryStatus(&memstat);
} // for (int i=0; i<10; i++)

QueryPerformanceCounter(&end);

GlobalMemoryStatus(&memstat);

printf("Time %.3f\n",
double(end.QuadPart-start.QuadPart)/double(freq.QuadPart));
CloseHandle(mapping);
delete[] mem;
GlobalMemoryStatus(&memstat);
} //if (mapping)

return 0;
}


regards,
George

"Ondrej Spanel" wrote:

> 1.
> Your described sample is interesting. Have you experienced any
situations
> when working set (counter in Perfmon) is larger than virtual memory
(counter)?

I do not know. I have no experience with PerfMon. I do not know if what
I wrote is really an answer to your question, however as it seems to be
related, I wrote it hoping it might help you in some way.

> 2.
> I think in your below case, you mentioned the file map pages are not
counted
> as part of the working set. How do you know that? Which
tool/technique you
> are using to check some specific page belongs to working set or not?

I know this because even when using very large mapped files this way,
the working set stays very low. I measure process working set size using
SysInternals Process Explorer or by default Windows Task Manager.
Moreover, in some applications I use GetProcessMemoryInfo from
PSAPI.DLL, which way I can get the same values as those tools do, but
programatically. The same tools also report the number of page faults,
however they do not distinguish between hard and soft ones. In case you
are interested, I attach my test source which I used for my file mapping
measurements. When running this source, I see following:

- the CPU load is 100 % of 1 CPU, which means no hard page faults (with
hard page faults the CPU would be idle while pages are loaded)
- I can see I page fault per 4 KB page access
- I can see very little memory reported as a working set
- I can see the page file space reserved on CreateFileMapping, and
physical memory being used after MapViewOfFile/UnmapViewOfFile, as
evidenced by GlobalMemoryStatus results

I suppose file pages which are currently mapped into a view are counted
as a part of both the working set and used virtual memory, however I did
not perform any experiments in this sense. This sample demonstrates how
to have physical memory allocated without using any virtual addresses
for it, by creating paging file backed file mapping and creating view
only as necessary.

Regards
Ondrej


Source follows:
-----------------------------------------
#include <windows.h>
#include <stdio.h>

int main(int argc, char* argv[])
{
LARGE_INTEGER start,end;
LARGE_INTEGER freq;
QueryPerformanceCounter(&start);
QueryPerformanceFrequency(&freq);

MEMORYSTATUS memstat;
memstat.dwLength = sizeof(memstat);
GlobalMemoryStatus(&memstat);

// basic file mapping test (512 MB)
long long size = 512*1024*1024;

HANDLE mapping =
CreateFileMapping(NULL,NULL,PAGE_READWRITE|SEC_COMMIT,(DWORD)(size>>32),DWORD(size),NULL);
if (mapping)
{
// create and destroy temporary views
SYSTEM_INFO sysInfo;
GetSystemInfo(&sysInfo);
const int allocSize = sysInfo.dwAllocationGranularity;

GlobalMemoryStatus(&memstat);

void *mem = new char[allocSize];
memset(mem,0x11,allocSize);
for (int i=0; i<10; i++)
{
for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
void *map =
MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map)
{
memcpy(map,mem,allocSize);
UnmapViewOfFile(map);
}
}
GlobalMemoryStatus(&memstat);


for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
void *map =
MapViewOfFile(mapping,FILE_MAP_READ,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map)
{
for (int t=0; t<allocSize; t++)
{
if (((char *)map)[t]!=0x11)
{
OutputDebugString("Memory read failed\n");
}
}
UnmapViewOfFile(map);
}
}
GlobalMemoryStatus(&memstat);
}
QueryPerformanceCounter(&end);

GlobalMemoryStatus(&memstat);

printf("Time %.3f\n",
double(end.QuadPart-start.QuadPart)/double(freq.QuadPart));
CloseHandle(mapping);
delete[] mem;
GlobalMemoryStatus(&memstat);
}


return 0;
}
--------------------------------

George napsal(a):
Thanks Ondrej,


Two more comments,

1.

Your described sample is interesting. Have you experienced any situations
when working set (counter in Perfmon) is larger than virtual memory (counter)?

2.

I think in your below case, you mentioned the file map pages are not counted
as part of the working set. How do you know that? Which tool/technique you
are using to check some specific page belongs to working set or not?


regards,
George

"Ondrej Spanel" wrote:

true, such RAM does not have related virtual memory address, how could the
current process utilize or even address (re-use to avoid hard page fault) it
in the future?
There is one special case with filemapping - you may have pages
allocated in a physical memory (be it RAM or page file), which have no
virtual address mapped. You can address such memory by creating a view
to it, use it, and then unmap again. To identify the location you use
the position in the file, and the memory is still committed even when
you do not have any virtual address mapped to it. Each time after you
map the view, you will get a soft page fault on first access, not the
hard one.

However, while this technique is definitely interesting, I never seen
such pages to be reported as a part of the working set (I am using
Process Explorer though, not PerfMon as you do).

(You might perhaps also find my reply to your older topic "File map
performance" of some interest)

Regards
Ondrej


.

Relevant Pages

  • Re: Still confused why working set larger than virtual memory
    ... With the SysInternals Process Explorer the column is called Working Set. ... What you removed is exactly the "smart trick" to have memory "accessible", ... as you map views you never unmap. ... I attach my test source which I used for my file mapping measurements. ...
    (microsoft.public.vc.language)
  • Re: Still confused why working set larger than virtual memory
    ... I have modified my program to show even if there is no leak, the working set ... Some comments, In my program, I use two ways to hold file map. ... // basic file mapping test ... What you removed is exactly the "smart trick" to have memory ...
    (microsoft.public.vc.language)
  • Re: Still confused why working set larger than virtual memory
    ... > when working set is larger than virtual memory? ... > I think in your below case, you mentioned the file map pages are not counted ... I can see very little memory reported as a working set ... I suppose file pages which are currently mapped into a view are counted as a part of both the working set and used virtual memory, however I did not perform any experiments in this sense. ...
    (microsoft.public.vc.language)
  • Re: Still confused why working set larger than virtual memory
    ... I measure process working set size using ... as a part of both the working set and used virtual memory, ... page map file does not have virtual address but in RAM when you access it? ... I attach my test source which I used for my file mapping ...
    (microsoft.public.vc.language)
  • Re: working set is larger than virtual bytes?
    ... In the definition of working set, it is the set of pages from virtual memory ... Some comments, In my program, I use two ways to hold file map. ... // basic file mapping test ...
    (microsoft.public.vc.language)