Re: Still confused why working set larger than virtual memory

> I am using Windows Server 2003. What OS are you using? In Windows Server

I am performing my tests on WinXP SP 2. With the SysInternals Process Explorer the column is called Working Set. I guess with the task manager some "user friendly" name is used (I tried to use PerfMon, however as I am running Czech version of WinXP, I gave it up very seeon, because all counter names are translated into Czech, and make no sense to me at all, because they are translated in a very strange way)

> I have studied your sample code, and modify it to show my points. In your
> test program, the working set is low because of each time after you use the
> portion of the memory map file, you close it.

What you removed is exactly the "smart trick" to have memory "accessible", but not "addressable". By using virtual addresses only temporarily, you have the memory reserved to you, but you have no permanent virtual addresses for it. Off course, using such memory is limited - e.g. you cannot have any pointers pointing into such memory. Approach like this is definitely necessary when you want to work with huge mapped files (>> 1 GB), as they would not fit into the virtual address space of the process.

> In my scenario, I keep it open. Here is my code based on your code and you
> can see working set is much higher than virtual bytes, this is the situation
> I described in my question. Any ideas?

My explanation for this would be the mapped portions of files are counted as part of "working set", but not as "virtual bytes". However without having exact definition or seeing the code OS uses to compute the values, it is only a qualified guess, nothing more. The definition I was able to find by short Googling "Shows the size, in bytes, of the virtual address space that the process is using." does not tell much. The results might also be OS dependent. When I tried it on my computer, I have always seen "Virtual Size" higher then "Working Set".

> (Here is my code, if there is anything wrong with my code, please also feel
> free to let me know.)

Your code leaks virtual address, as you map views you never unmap. While for purposes of demonstrating the "working set is much higher than virtual bytes" it probably does not matter, you would definitely not want to do it like this in a production code. Each view you map needs to be unmapped at well at some point. Even in this experiment this leak causes virtual space exhausted once you reach ~2 GB of mapped views, with all MapViewOfFile calls failing since that point.

Regards
Ondrej

George napsal(a):
Hi Ondrej,


I have studied your sample code, and modify it to show my points. In your test program, the working set is low because of each time after you use the portion of the memory map file, you close it.

In my scenario, I keep it open. Here is my code based on your code and you can see working set is much higher than virtual bytes, this is the situation I described in my question. Any ideas?

(Here is my code, if there is anything wrong with my code, please also feel free to let me know.)

#include <windows.h> #include <stdio.h>

int main(int argc, char* argv[]) { LARGE_INTEGER start,end; LARGE_INTEGER freq; QueryPerformanceCounter(&start); QueryPerformanceFrequency(&freq);

MEMORYSTATUS memstat; void* map;
memstat.dwLength = sizeof(memstat); GlobalMemoryStatus(&memstat);

// basic file mapping test (512 MB) long long size = 512*1024*1024;

HANDLE mapping = CreateFileMapping(NULL,NULL,PAGE_READWRITE|SEC_COMMIT,(DWORD)(size>>32),DWORD(size),NULL); if (mapping) { // create and destroy temporary views SYSTEM_INFO sysInfo; GetSystemInfo(&sysInfo); const int allocSize = sysInfo.dwAllocationGranularity;

GlobalMemoryStatus(&memstat);

void *mem = new char[allocSize]; memset(mem,0x11,allocSize); for (int i=0; i<10; i++) { for (long long offset=0; offset<=size-allocSize; offset+=allocSize) { map = MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize); if (map) { memcpy(map,mem,allocSize); // UnmapViewOfFile(map);
} }

if (map) { UnmapViewOfFile(map);
}

GlobalMemoryStatus(&memstat);


for (long long offset=0; offset<=size-allocSize; offset+=allocSize) { map = MapViewOfFile(mapping,FILE_MAP_READ,(DWORD)(offset>>32),(DWORD)offset,allocSize); if (map) { for (int t=0; t<allocSize; t++) { if (((char *)map)[t]!=0x11) { OutputDebugString("Memory read failed\n"); } }

// UnmapViewOfFile(map); } } if (map)
{
UnmapViewOfFile(map);
}

GlobalMemoryStatus(&memstat); } // for (int i=0; i<10; i++)

QueryPerformanceCounter(&end);

GlobalMemoryStatus(&memstat);

printf("Time %.3f\n", double(end.QuadPart-start.QuadPart)/double(freq.QuadPart)); CloseHandle(mapping); delete[] mem; GlobalMemoryStatus(&memstat); } //if (mapping)

return 0;
}


regards,
George

"Ondrej Spanel" wrote:

> 1.
> Your described sample is interesting. Have you experienced any situations
> when working set (counter in Perfmon) is larger than virtual memory (counter)?

I do not know. I have no experience with PerfMon. I do not know if what I wrote is really an answer to your question, however as it seems to be related, I wrote it hoping it might help you in some way.

> 2.
> I think in your below case, you mentioned the file map pages are not counted
> as part of the working set. How do you know that? Which tool/technique you
> are using to check some specific page belongs to working set or not?

I know this because even when using very large mapped files this way, the working set stays very low. I measure process working set size using SysInternals Process Explorer or by default Windows Task Manager. Moreover, in some applications I use GetProcessMemoryInfo from PSAPI.DLL, which way I can get the same values as those tools do, but programatically. The same tools also report the number of page faults, however they do not distinguish between hard and soft ones. In case you are interested, I attach my test source which I used for my file mapping measurements. When running this source, I see following:

- the CPU load is 100 % of 1 CPU, which means no hard page faults (with hard page faults the CPU would be idle while pages are loaded)
- I can see I page fault per 4 KB page access
- I can see very little memory reported as a working set
- I can see the page file space reserved on CreateFileMapping, and physical memory being used after MapViewOfFile/UnmapViewOfFile, as evidenced by GlobalMemoryStatus results

I suppose file pages which are currently mapped into a view are counted as a part of both the working set and used virtual memory, however I did not perform any experiments in this sense. This sample demonstrates how to have physical memory allocated without using any virtual addresses for it, by creating paging file backed file mapping and creating view only as necessary.

Regards
Ondrej


Source follows:
-----------------------------------------
#include <windows.h>
#include <stdio.h>

int main(int argc, char* argv[])
{
LARGE_INTEGER start,end;
LARGE_INTEGER freq;
QueryPerformanceCounter(&start);
QueryPerformanceFrequency(&freq);

MEMORYSTATUS memstat;
memstat.dwLength = sizeof(memstat);
GlobalMemoryStatus(&memstat);

// basic file mapping test (512 MB)
long long size = 512*1024*1024;

HANDLE mapping = CreateFileMapping(NULL,NULL,PAGE_READWRITE|SEC_COMMIT,(DWORD)(size>>32),DWORD(size),NULL);
if (mapping)
{
// create and destroy temporary views
SYSTEM_INFO sysInfo;
GetSystemInfo(&sysInfo);
const int allocSize = sysInfo.dwAllocationGranularity;

GlobalMemoryStatus(&memstat);

void *mem = new char[allocSize];
memset(mem,0x11,allocSize);
for (int i=0; i<10; i++)
{
for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
void *map = MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map)
{
memcpy(map,mem,allocSize);
UnmapViewOfFile(map);
}
}
GlobalMemoryStatus(&memstat);


for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
void *map = MapViewOfFile(mapping,FILE_MAP_READ,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map)
{
for (int t=0; t<allocSize; t++)
{
if (((char *)map)[t]!=0x11)
{
OutputDebugString("Memory read failed\n");
}
}
UnmapViewOfFile(map);
}
}
GlobalMemoryStatus(&memstat);
}
QueryPerformanceCounter(&end);

GlobalMemoryStatus(&memstat);

printf("Time %.3f\n", double(end.QuadPart-start.QuadPart)/double(freq.QuadPart));
CloseHandle(mapping);
delete[] mem;
GlobalMemoryStatus(&memstat);
}


return 0;
}
--------------------------------

George napsal(a):
Thanks Ondrej,


Two more comments,

1.

Your described sample is interesting. Have you experienced any situations when working set (counter in Perfmon) is larger than virtual memory (counter)?

2.

I think in your below case, you mentioned the file map pages are not counted as part of the working set. How do you know that? Which tool/technique you are using to check some specific page belongs to working set or not?


regards,
George

"Ondrej Spanel" wrote:

true, such RAM does not have related virtual memory address, how could the current process utilize or even address (re-use to avoid hard page fault) it in the future?
There is one special case with filemapping - you may have pages allocated in a physical memory (be it RAM or page file), which have no virtual address mapped. You can address such memory by creating a view to it, use it, and then unmap again. To identify the location you use the position in the file, and the memory is still committed even when you do not have any virtual address mapped to it. Each time after you map the view, you will get a soft page fault on first access, not the hard one.

However, while this technique is definitely interesting, I never seen such pages to be reported as a part of the working set (I am using Process Explorer though, not PerfMon as you do).

(You might perhaps also find my reply to your older topic "File map performance" of some interest)

Regards
Ondrej

.

Relevant Pages

  • Re: Still confused why working set larger than virtual memory
    ... portion of the memory map file, ... can see working set is much higher than virtual bytes, ... I attach my test source which I used for my file mapping ...
    (microsoft.public.vc.language)
  • Re: Still confused why working set larger than virtual memory
    ... I have modified my program to show even if there is no leak, the working set ... Some comments, In my program, I use two ways to hold file map. ... // basic file mapping test ... What you removed is exactly the "smart trick" to have memory ...
    (microsoft.public.vc.language)
  • Re: Still confused why working set larger than virtual memory
    ... > when working set is larger than virtual memory? ... > I think in your below case, you mentioned the file map pages are not counted ... I can see very little memory reported as a working set ... I suppose file pages which are currently mapped into a view are counted as a part of both the working set and used virtual memory, however I did not perform any experiments in this sense. ...
    (microsoft.public.vc.language)
  • Re: Still confused why working set larger than virtual memory
    ... I measure process working set size using ... as a part of both the working set and used virtual memory, ... page map file does not have virtual address but in RAM when you access it? ... I attach my test source which I used for my file mapping ...
    (microsoft.public.vc.language)
  • Re: Binary data in multiple adjacent files
    ... Mapping it doesn't eat any working set except, possibly, for the ... It's not faulted into physical memory. ... You also get a whole "page fault cluster" faulted in. ... grid sections immediately north, south, northeast, northwest, southeast ...
    (comp.os.vms)