Re: Let's discuss XP memory management (paging, pagfile, etc.)
- From: "Vagabond Software" <vagabondsw-X-@xxxxxxxxxxxx>
- Date: Sun, 26 Feb 2006 22:35:19 -0800
"Jim Lewandowski" <jlewand@xxxxxxxxxxxxxxxxxxx> wrote in message
news:QDuMf.61318$PL5.18744@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I've been scouring newsgroups for the past week to see what's going on out
there with regard to XP's memory management. I've also read Russinovich's
Windows Internal book. This is a long post for those so-inclined to not
like to read verbose posts.
FWIW, my background comes from 25 years of machine/assembler-level coding
on 4 different hardware platforms. Memory algorithms and memory
management, low-level disk I/O and optimization (including buffering
algorithms) are my forte'.
I've seen not ONE newsgroup post from anyone with specific examples. All
the newsgroup/regular web stuff I read talks about conceptuals. I'd like
to move the discussion to conceptual EXAMPLES (to start) and eventually
move to real-life examples (by discussing conceptual examples we can then
hopefully agree on WHAT types of real-life tests to perform).
I've also seen MVPs constantly point to various links/URLs which do NOT
address the specific questions being asked: NEVER. The URLs/links in
question NEVER answer the 3 questions below.
1) CUSTOM page file not STICKING to what the user requests
2) WHY XP seems to not want to USE a lot of RAM (I have 1G on my machine)
even when HALF of it is still available (i.e. pagefile usage grows)
3) WHAT the actual memory management algorithms ARE (I have a hunch which
we'll discuss at the end)
***
SOME of the below is assumed to work a certain way based on my experience
with the large IBM mainframe OS (which is 1000x more complex than Windows)
and has been evolving since the 60's (NASA was a big user in the early
days).
***
Gigabyte/Gig/GB and Megabyte/Meg/MB and kilobyte/KB/K are all powers of 2
(i.e. K is 1024, MB is 1024*1024).
VIRTUAL MEMORY
Virtual memory is like having MORE memory than what your RAM (not disk,
not anything else, but RAM) will hold. The reasoning behind this is that
programs use only a SUBSET of their memory ALLOCATION request. Thus, for
the 80% of the memory pages that they DON'T access (whether DATA or CODE),
those can either be (discussion later) discarded COMPLETELY out of memory
OR put onto a paging file if there's stress for RAM usage/needs.
XP (Windows, the CPU chips, however you want to slice and dice this) is
designed to allow each PROCESS to use a 4 GB address space (2 to the power
32) which is numbered in hex from 00000000 to FFFFFFFF. However, only
HALF of that address space is USABLE by joe-average application program.
IOW, let's discard system-level (kernal, drivers) from the discussion at
this point. FTR, the USER accessible VIRTUAL address space area resides
from 00000000-7FFFFFFF. Let's not discuss the special case of 3 GB user
address spaces (of the 4 GB total when the system only gets 1 GB).
ALL referenced storage to an average non-system/driver/kernal program is
VIRTUAL in nature. EACH address space (process) has it's OWN
00000000-7FFFFFFF area of VIRTUAL storage to play with (2 GB). IOW,
process A has a virtual address at 0x00001000 as does process B and
process C. Those are eventually mapped (backed) by REAL RAM to allow the
CPU to execute them or the data bus to ADDRESS/access them.
WORKING SET
The working set for a process is ONLY VIRTUAL storage/memory that is
backed in REAL RAM at this instant in time. If you add up all the exe/dll
storage for a process AND its data storage, this number is usually MUCH
small than the AGGREGATE (total) VIRTUAL storage in use within the address
space. IOW, within the WORD application program, if you don't use any of
the HELP or touch ANY functions other than start typing away in the
default font, the working set (ACTUAL code executed within the mass of exe
and dlls) CAN BE quite small (i.e. it WILL be small if XP is under a RAM
shortage and starts trimming pages from RAM).
PAGING FILE
There is some disagreement as to whether the XP/Windows paging file will,
by design, ever hold NON-modified pages (i.e. exe and dll code). We'll
discuss this later in this post as we get to conceptual examples that will
hopefully illuminate how things work AND how to test HOW we THINK things
work.
FTR, if you specify a CUSTOM-sized pagefile and then upon boot see a 1.5x
RAM-sized one, it is likely that your initial (primary) size was TOO SMALL
and XP took it upon himself to discard the CUSTOM settings. There are
supposedly some Norton Anti-Virus issues related to the CUSTOM setting of
a pagefile NOT being used by XP. In my case, with NAV disabled on boot, a
too-small CUSTOM primary/minimum setting still reverted to 1.5x RAM.
After boot, about 20 MB of my pagefile is in use ALREADY (this implies XP
writes some nucleus/kernal code out there that will likely NEVER be used
for this boot sequence). This is MY experience and yours is likely to be
the same.
CONCEPTUAL EXAMPLE to illustrate the above terms
Program A.exe loads from your C: drive and is 100K in size. It will
reside in its own address space (PROCESS A) and consume VIRTUAL addresses
00000000-00018FFF. Program A then asks XP, I'd like 100MB of VIRTUAL
storage in one contiguous VIRTUAL chunk. This VIRTUAL memory allocation
consumes VIRTUAL addresses from 00019000-06418FFF. NONE of this virtual
memory allocation (100MB residing on VIRTUAL addresses > 00019000) has
been TOUCHED in any way by the application. XP does NOTHING other than
account for it (i.e. PROCESS A has used 0-06418FFF VIRTUAL so the next
VIRTUAL address I will hand PROCESS A will be 06419000) AND increment the
PF usage (XP Task Manager) by the SAME storage amount (100M) amount EVEN
THOUGH none of it is IN REAL RAM YET!
Viewing the SYSTEM MONITOR in XP (Admin tools > Performance > System
Monitor) and viewing PROCESS (ALL FIELDS) shows for Process A:
PRIVATE BYTES (100K + 100M)
PAGE FILE BYTES (100K + 100M)
The 2 terms are synoymous save for the exception, IIRC, of the SYSTEM
address space.
This newly asked-for memory is NOT written to the pagefile. The PF USAGE
in TM is only accumulating the POTENTIAL virtual memory in use by ALL
processes. It's a poor-man's way of showing YOU the user what the ACTUAL
memory demands of your system are.
In the original program A.exe, ONLY code within the first 4K has been
executed at this point in time. However, the WORKING SET is STILL 100K
(A.exe) because XP hasn't done any storage trimming, etc.
The working set currently sits at 100K (A.exe). As A.exe starts TOUCHING
(i.e. modifying and/or reading) VIRTUAL addresses within its 2 Gigabyte
(Gig, GB) address space, those result in a machine exception (i.e. virtual
address Q for a given address space is NOT backed in REAL RAM) which
results in XP allocating a 4K memory page in REAL RAM with the appropriate
system-level housekeeping to keep track of WHICH REAL RAM page contains
virtual address Q for process A. Let's say A.exe only touches the first
100K of this 100MB that it has asked for. Thus, the working set at this
instant would be 200K (100K for A.exe + 100K of touched/POPULATED WITH
DATA virtual pages). This would show as WORKING SET in System Monitor for
Process A. Note: SM shows things in DECIMAL (unless noted IN SM's field
descriptors).
Questions to ask ourselves:
If XP loads a 100K .exe program into storage, it DOES have to page-fault
100K of REAL RAM to do so. Is this exe/dll REAL RAM "modified" as to
pagefile potential?
If a process populates 100K of data storage, to XP are BOTH equal in terms
of "modifiability"? IOW, exe code rarely changes on the fly
(self-modifying code is not likely with C/C#/C+/C++ programmers).
Storage had to be "backed in REAL RAM" to hold BOTH the exe code pages AND
the then-asked for DATA pages. IF, as I've read, exe/dll pages are NEVER
paged to a pagefile (only MODIFIED pages are ever written to the
pagefile), again, aren't BOTH the exe/dll code pages AND the DATA pages
"modified" in that they were changed/populated with something other than
binary 0's?
IF XP won't page exe/dll code, it would then have to re-read the exe/dll
and then re-back in REAL RAM/page fault REAL RAM to then hold them. This
also begs the question, in our example, of the original 100K of exe code,
is it BLOCK-paged (i.e. removed from RAM if there's a storage
crisis/shortage as far as XP is concerned)? I can't believe XP would only
discard REAL RAM pages, as AN example for our case, 1, 35, 100, 103, and
106 then ONLY repopulate THOSE pages (i.e. XP would read the exe disk
block/page by disk block/page and ONLY re-back in REAL RAM pages
1,35,100,103,106). It would be FAR simpler to have them paged to the
pagefile (after all that's what it is there for - to hold MORE virtual
addresses than what RAM CAN hold). What difference does it make whether
they go to the pagefile when they're on disk already? Well, it would eat
up more pagefile territory, but exe/dll storage is quite small relative to
DATA storage (typically).
I've noticed at work (XP Pro) and at home (XP Home), the % PAGE FILE IN
USE grows quite quickly at certain points in time. It almost seems that
XP is PREEMPTIVELY paging when likely YOUR workload on YOUR PC doesn't
change much from boot sequence to boot. It COULD be that this is because
in 2001 when XP was released, disk drives that hold page files were slow
(5400 RPM or lower) and they wanted to stay "ahead of the RAM demand
curve" by paging older, modified, but NOT recently referenced pages to the
pagefile to clear REAL RAM for "potential" upcoming uses. This seems
possible but it would seem a WASTE of time to preemptively PREDICT what
future workload an OS might experience. Completely, utterly a waste of
time. Predicting DISK ACCESS patterns for a specific file: very doable
based on "hints" from open time. Predicting when and IF you'll start
MSWord seems wasteful.
I've attempted to use Sysinternals.com (Russinovich) FILEMON to trace
pagefile.sys I/Os and it's hard to catch them at those points in time when
the % PF in usage DOES grow quickly.
REAL-LIFE EXAMPLE
PF usage in Task Manager shows 250MB (VERY normal +/0 a few MB). I open a
20 MB JPG and surprisingly see it grow to 850 MB (how a 20 MB JPG could
take that much RAM for Photoshop is beyond me). My AVAILABLE (Task
Manager PHYSICAL MEMORY K) memory never dropped below 150 MB. I then
opened another JPG (xxx) and resized the resolution and Photoshop just sat
there for 30 seconds. TM PF usage did NOT increase. Physical Memory
AVAILABLE never DECREASED. Filemon showed no pagefile accesses. CPU was
NOT screaming away (some in SYSTEM, little in Photoshop). Photoshop never
came back so I tried to flush it using END PROCESS in TM and it still took
quite a while. When the desktop eventually was shown again, it took
FOREVER to repaint the screen (like a VERY slow TV picture paint,
resolution line by resolution line).
Again, AVAILABLE memory never dropped below 150 MB and PF usage never
climbed ABOVE 850 MB. Makes NO sense whatsover.
Someone asked WHY anyone would want to play with the pagefile size. I
have 2 reasons:
1) when I backup my PC using 3rd party software, it doesn't allow EXCLUDE
files so a 1.5 GB pagefile.sys file takes about 1/4 of the backed up
image.
2) my C: partition is only 10 GB and I don't want 1.5 GB of that eaten up
by a pagefile that's useless since THAT disk space might be needed for
burning a CD or defragging the C: partition.
If one does NOT have a pagefile, the only real drawback seems to be the 20
MB that's parked there RIGHT after boot? If that were to remain in REAL
RAM in my case, that would be NO problem when typically I have 750 MB of
REAL RAM left. If I exclude ALL that pagefile writing when my REAL RAM is
STILL above a few hundred MB, frankly I don't see ANY downside. If I ever
open a massive JPG in photoshop, not having a pagefile could be a problem.
However, in the instance when I did the real-life example above, the PF
usage went to 99.5% (100 MB) and never seemed to force an ADDITIONAL
pagefile allocation. Very, very mysterious.
This post is just for starters. I'll gladly welcome discussion (whether
terse and confrontational or otherwise). Eventually, we'll get into cache
fast-write options (for your disk drive) and how they impact the page
file. As well, rarely if ever would only ONE 4K page from REAL RAM be
written to the pagefile in cases of RAM shortage/stress. A whole track's
worth would seem the most worthwhile as little additional delay would
occur (cache fast-write, etc.). This can be turned on via SYSTEM > DEVICE
MANAGER > DISK DRIVES (policies tab).
JL
I don't understand the point of this discussion. First of all, you say that
your page file is used when you have memory available. I am now three
months into the second generation of desktop computer where my pagefile is
yet to be touched. We need to figure out why your computer is using the
pagefile where mine is not.
carl
.
- References:
- Let's discuss XP memory management (paging, pagfile, etc.)
- From: Jim Lewandowski
- Let's discuss XP memory management (paging, pagfile, etc.)
- Prev by Date: Re: Encrypt Contents to Secure Data
- Next by Date: RE: Let's discuss XP memory management (paging, pagfile, etc.)
- Previous by thread: Let's discuss XP memory management (paging, pagfile, etc.)
- Next by thread: RE: Let's discuss XP memory management (paging, pagfile, etc.)
- Index(es):
Relevant Pages
|
