Re: Still confused why working set larger than virtual memory
- From: Ondrej Spanel <OndrejSpanel.news@xxxxxxxxxxxxx>
- Date: Mon, 14 Jan 2008 09:39:37 +0100
> 1.
> Your described sample is interesting. Have you experienced any situations
> when working set (counter in Perfmon) is larger than virtual memory (counter)?
I do not know. I have no experience with PerfMon. I do not know if what I wrote is really an answer to your question, however as it seems to be related, I wrote it hoping it might help you in some way.
> 2.
> I think in your below case, you mentioned the file map pages are not counted
> as part of the working set. How do you know that? Which tool/technique you
> are using to check some specific page belongs to working set or not?
I know this because even when using very large mapped files this way, the working set stays very low. I measure process working set size using SysInternals Process Explorer or by default Windows Task Manager. Moreover, in some applications I use GetProcessMemoryInfo from PSAPI.DLL, which way I can get the same values as those tools do, but programatically. The same tools also report the number of page faults, however they do not distinguish between hard and soft ones. In case you are interested, I attach my test source which I used for my file mapping measurements. When running this source, I see following:
- the CPU load is 100 % of 1 CPU, which means no hard page faults (with hard page faults the CPU would be idle while pages are loaded)
- I can see I page fault per 4 KB page access
- I can see very little memory reported as a working set
- I can see the page file space reserved on CreateFileMapping, and physical memory being used after MapViewOfFile/UnmapViewOfFile, as evidenced by GlobalMemoryStatus results
I suppose file pages which are currently mapped into a view are counted as a part of both the working set and used virtual memory, however I did not perform any experiments in this sense. This sample demonstrates how to have physical memory allocated without using any virtual addresses for it, by creating paging file backed file mapping and creating view only as necessary.
Regards
Ondrej
Source follows:
-----------------------------------------
#include <windows.h>
#include <stdio.h>
int main(int argc, char* argv[])
{
LARGE_INTEGER start,end;
LARGE_INTEGER freq;
QueryPerformanceCounter(&start);
QueryPerformanceFrequency(&freq);
MEMORYSTATUS memstat;
memstat.dwLength = sizeof(memstat);
GlobalMemoryStatus(&memstat);
// basic file mapping test (512 MB)
long long size = 512*1024*1024;
HANDLE mapping = CreateFileMapping(NULL,NULL,PAGE_READWRITE|SEC_COMMIT,(DWORD)(size>>32),DWORD(size),NULL);
if (mapping)
{
// create and destroy temporary views
SYSTEM_INFO sysInfo;
GetSystemInfo(&sysInfo);
const int allocSize = sysInfo.dwAllocationGranularity;
GlobalMemoryStatus(&memstat);
void *mem = new char[allocSize];
memset(mem,0x11,allocSize);
for (int i=0; i<10; i++)
{
for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
void *map = MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map)
{
memcpy(map,mem,allocSize);
UnmapViewOfFile(map);
}
}
GlobalMemoryStatus(&memstat);
for (long long offset=0; offset<=size-allocSize; offset+=allocSize)
{
void *map = MapViewOfFile(mapping,FILE_MAP_READ,(DWORD)(offset>>32),(DWORD)offset,allocSize);
if (map)
{
for (int t=0; t<allocSize; t++)
{
if (((char *)map)[t]!=0x11)
{
OutputDebugString("Memory read failed\n");
}
}
UnmapViewOfFile(map);
}
}
GlobalMemoryStatus(&memstat);
}
QueryPerformanceCounter(&end);
GlobalMemoryStatus(&memstat);
printf("Time %.3f\n", double(end.QuadPart-start.QuadPart)/double(freq.QuadPart));
CloseHandle(mapping);
delete[] mem;
GlobalMemoryStatus(&memstat);
}
return 0;
}
--------------------------------
George napsal(a):
Thanks Ondrej,.
Two more comments,
1.
Your described sample is interesting. Have you experienced any situations when working set (counter in Perfmon) is larger than virtual memory (counter)?
2.
I think in your below case, you mentioned the file map pages are not counted as part of the working set. How do you know that? Which tool/technique you are using to check some specific page belongs to working set or not?
regards,
George
"Ondrej Spanel" wrote:
true, such RAM does not have related virtual memory address, how could the current process utilize or even address (re-use to avoid hard page fault) it in the future?There is one special case with filemapping - you may have pages allocated in a physical memory (be it RAM or page file), which have no virtual address mapped. You can address such memory by creating a view to it, use it, and then unmap again. To identify the location you use the position in the file, and the memory is still committed even when you do not have any virtual address mapped to it. Each time after you map the view, you will get a soft page fault on first access, not the hard one.
However, while this technique is definitely interesting, I never seen such pages to be reported as a part of the working set (I am using Process Explorer though, not PerfMon as you do).
(You might perhaps also find my reply to your older topic "File map performance" of some interest)
Regards
Ondrej
- Follow-Ups:
- References:
- Re: Still confused why working set larger than virtual memory
- From: Ondrej Spanel
- Re: Still confused why working set larger than virtual memory
- From: George
- Re: Still confused why working set larger than virtual memory
- Prev by Date: Re: output hex bytes to binary file
- Next by Date: Heap or Stack
- Previous by thread: Re: Still confused why working set larger than virtual memory
- Next by thread: Re: Still confused why working set larger than virtual memory
- Index(es):
Relevant Pages
|
Loading
