Re: Still confused why working set larger than virtual memory

George napsal(a):
Server, perhaps he can see it there). In my code attached in this thread what I see is even when I mapped and unmapped 512 MB of files, using 512 MB of physical memory and using no virtual addresses,

I still do not know how could the mapped file pages do not have virtual address, but have physical address? In my mind, everything we can access in current process should be in current process's address space. Could you provide more information please?

I will answer for the last time, however I think you should be able to see this for yourself already, as all information is in the sample.

I can map the page, will it with data, and unmap it. The next time I map the same part of the file, it still contains the same data I filled it before, demonstrating it resided somewhere (I also explained why I am sure they resided in the physical memory, not in the page file). Off course, each time you want to access the data, it needs to have a virtual address, but the trick is it does not need to have any permanent virtual address, and it is able to exist even without a virtual address, as it exists as a part of the file mapping (as for whether it has a physical address meanwhile, it is not that interesting - it probably has while it resides in the physical memory, in case it is swapped out, it does not have it any more, but this is not something I need to care about, OS takes care of this).

This also exactly what Alexander Nickolov is talking about: even when you unmap your file views, the pages still may reside in the memory and may be a part of the working set (the working set is a set of pages recently used - it may contain even pages which do not have any virtual address mapped right now, and this is useful because on mapping them again you get only soft fault, not a hard one).

MB of physical memory and using no virtual addresses, I cannot see any significant working set - the working set reported by both

You can not see any significant working set because you free the memory at once when you complete using it. So, it is too fast and counter can not "see" it. :-)

I am not freeing the memory, I am only unmapping its virtual addresses. The memory is "allocated" (committed) using CreateFileMapping and not released until I release the mapping.

You can see my program (new version) posted above to see if we hold all memory for a while and close all handles altogether at the end, you can find the working set is much larger.

No, I am sorry, this is not what I see with WinXP SP 2. Perhaps your OS is counting workins set size in a different (more logical) way.

What I see is:

- while I am mapping more and more memory, both Virtual Size and Working Set grow, Virtual Size keeping a little bit higher than Working Set
- once I unmap the memory, both Virtual Size and Working Set size go down, Virtual Size still keeping a little bit higher than Working Set

If you see a working set in this situation to stay above 512 MB, it is perfectly logical, as explained above, and it means the numbers your OS is reporting are more reliable then mine.

Regards
Ondrej


Any comments?


regards,
George

"Ondrej Spanel" wrote:

Alexander Nickolov napsal(a):
What pertains to your question is the fact you may have mapped,
used and unmapped many views. The pages you used in those views
would be in your working set, but they will not use virtual addresses.
In that case your working set may become larger than your virtual
memory address space. I think that's what everybody else was
trying to tell you so far.

I would definitely agree with you in theory, however I am afraid it is not that easy to demonstrate it in practice.

What you write definitely makes sense with the "encyclopaedia" definition of the working set, however it is not what I see with Process Explorer on WinXP SP 2. (I do not know what George sees on Win 2003 Server, perhaps he can see it there). In my code attached in this thread what I see is even when I mapped and unmapped 512 MB of files, using 512 MB of physical memory and using no virtual addresses, I cannot see any significant working set - the working set reported by both GetProcessMemoryInfo and Process Explorer (which most likely uses GetProcessMemoryInfo as well) stays only a few MB. I would expect those pages to be counted as a part of the working set, exactly from the reasons you outlines here, however this is not what I see. (As for why I am sure the data stay in physical memory, see my other replies in this thread)

I guess the best way to explain this would be to know the exact OS definition of the "working set" value reported by GetProcessMemoryInfo. The docs say "Current working set size, in bytes.", which really does not reveal much.

(By the way, the pages mapped are also not counted in the PagefileUsage member of the PROCESS_MEMORY_COUNTERS, which I also find unexpected, because I create the file mapping object backed by the paging file).

Regards
Ondrej

.

Relevant Pages

  • Re: Performance problem on shared memory over 2GB on XP x64
    ... working set, it is placed onto the modified page list. ... If you no longer need the mapping object and want to destroy ... If you need to preserve memory contents for later use, ... before unmapping it. ...
    (microsoft.public.win32.programmer.kernel)
  • Re: Still confused why working set larger than virtual memory
    ... > physical memory and counts as a part of the working set of current process ... I do not know what definition of working set which OS uses when reporting this value. ... If you want to prove this, you should be able to construct a simple code using VirtualQuery or VirtualAlloc, or by monitoring the global virtual space usage using GlobalMemoryStatus - and no, I have no intention to construct such code for you. ... I think you guys mean when we unmap some pages, since they are still in the physical memory and counts as a part of the working set of current process and at the same time they do not have virtual address, so we can see working set is larger than virtual bytes. ...
    (microsoft.public.vc.language)
  • Re: Still confused why working set larger than virtual memory
    ... has virtual address, and when we unmap it, it does not have virtual memory ... but it resides in memory. ... working set shirnks to very little value. ... MB of physical memory and using no virtual addresses, ...
    (microsoft.public.vc.language)
  • Re: Still confused why working set larger than virtual memory
    ... With the SysInternals Process Explorer the column is called Working Set. ... What you removed is exactly the "smart trick" to have memory "accessible", ... as you map views you never unmap. ... I attach my test source which I used for my file mapping measurements. ...
    (microsoft.public.vc.language)
  • Re: Binary data in multiple adjacent files
    ... Mapping it doesn't eat any working set except, possibly, for the ... It's not faulted into physical memory. ... You also get a whole "page fault cluster" faulted in. ... grid sections immediately north, south, northeast, northwest, southeast ...
    (comp.os.vms)
Loading