Re: Att. Alex Nichol -VM cont.

From: Joshua Bolton (JoshuaBolton_at_discussions.microsoft.com)
Date: 12/27/04 

Date: Mon, 27 Dec 2004 11:29:04 -0800

Hi Cquirke:

> Have you got a URL for that?
Yes. There are two of them and they are in the first post of this thread
>
> "AFAIK the original 386 design hard-coded virtual memory paging in 4k
> blocks, which fits 4k clusters quite nicely." and "This virtual paging
> isn't related to segment size, BTW".
Do you have a URL for this? Anything from MS saying 4kb clusters are
optimal for pagefile.sys? A source for the reason behind this?

My first point is that from the Intel and Microsoft urls I find both
manufacturers support for large memory pages [2 or 4meg pages]. No where can
I find documentation that says a 4kb page in memory is written to the hard
drive ie. pagefile.sys in 4kb incriments. A simple documented statement from
Intel or MS would say something like "pagefile.sys is written in such a
manner that a 4kb memory page is equal to a 4kb cluster written on the disk".
 In all of your study have you ever read a statement like that? I haven't.
So where is the basis for saying since 4kb paging in memory that 4kb clusters
are optimal for pagefile.sys?

My arguements to the contrary start with how going from billionths of a
second in memory to thousands of a second access time on a disk, the pagefile
writes [memory to disk] are going to stack up while waiting for the disk to
become available to be written to.

"thus transparent to what we are dealing with (transferring from RAM to HD)."

The time spent transferring data/code from memory to disk IS revelvant when
you consider that cluster size can make a huge difference to any file.
Pagefile.sys is a file.

For example if you do video editing you do NOT want a 4kb cluster size.
This is due to the fact you are working with relatively large files 1-2gigs
and larger. Math shows you the reason. A 2gig file on 4kb clusters takes up
500 clusters. A 2gig file on 512kb clusters takes up 4 clusters. If
contigous and the drives being equal the pickup rate should be equal. But
rarely is a file contigous and this is where the true "tire meets the road"
consequences of fragmentation come in. 4 reads vs 500 reads makes the larger
cluster size a faster retrieval option for large files.

If you have a gig pagefile, would it be optimal to have larger cluster sizes?

I am left with the question of how is pagefile.sys written to? How is it
internally organized?

Lets say the OS wants to "pageout" to disk 128kb of code and data from
memory [this can be a cumlitive of 32 4kb pages]. It sends it somewhere [a
buffer] to await the disk write or marks it as available to be paged out. In
the meantime of waiting for the disk to become available another 256kb of
working set memory needs to be paged to disk due to a different program/os
request. These two disk writes are sent to the disk controller. The disk
controller computs and sets their order in the disk cache to optimize the
disk head movement and its subsquent writes. 384kb of data/code is written
to the pagefile.

Lets assume [since we can't seem to find any documentation to support this
aspect] that we are dealing with only 4kb paging from memory to 4kb clusters.
 You physically can't go from a 4kb memory page to a 4kb cluster any faster
then the SLOWEST link. Which in this case is the hard drive. It would slow
head movement if the disk controller waited for each 4kb page to do a disk
write so its going to stack them up. Just like a capacitor does before it
releases its energy. Data is written as a collection of 4kb pages [assuming
paging to disk is ONLY done in 4kb segments].

So how is this data/code organized in the pagefile?

Lets look at the reverse of this process of writting from memory to disk.
We wrote 384kb of data to pagefile.sys. The memory manager determines it
wants back 64kb of that paged out memory back in its working set. The MM
makes the disk request and waits for the data to be returned. After all the
MM is in billionths of a time slice and the drive controller is only living
in the thousandths time slice world. The data is read back into memory.

Where does the cluster size of 4kb help in the disk read operation? The disk
head could have read two 32kb blocks of data just as fast as thirtytwo 4kb
blocks if they are configuous. Now there is a 64kb hole in the 384kb block
of data. How is this space optimized for the next block of data to be
written?

So where is the reasoning that since paging in Ram is done in 4kb pages that
this translates into 4kb cluster sizes are optimal for paging to disk?

Everything I can find concerning paging and disk operations points to larger
cluster sizes would optimize paging not smaller ones. Especially if you were
using larger pagefiles.

The last point is that both Intel and MS documentation talk about large
memory pages. The intel URLs talk about how programmers can use switches in
their programming to utilize different memory models. In the Microsoft URL
concerning XP in the section titled "Windows XP: Kernel Improvements", MS
talks about XP's use of "large memory pages of 4meg". According to this
article if you have more then 255megs of ram you are automatically going to
use large memory pages.

quote from article
"on such systems, these files (and other core operating system data such as
initial nonpaged pool and the data structures that describe the state of each
physical memory page) are mapped with 4MB "large pages" (as opposed to the
normal 4KB page size on the x86 processor set). By mapping this core
operating system code and data with 4MB pages"

and
"The Windows XP change was needed because of the continual increase in
typical memory configurations of PC systems. The minimum memory to use large
pages is now more than 255MB, instead of more than 127MB"

The article does not talk about paging to disk at all.

If the Intel processor supports the three memory management models, MS
programming suppports large memory pages then how can it be assumed ALL
memory manager operations are done in 4kb pages? Wouldn't the OS's memory
manager stratagy change depending on the application being run? How do we go
from 4kb pages in RAM to be equal to 4kb clusters on disk? It's like saying
4lbs of apples are equal to 4lbs of oranges. They are equal in weight but not
in content.

If you come across a link addressing these questions it sure would be great
to have. Thanks

Thank you everyone for participating in this discussion. Happy New Year to
all!

Relevant Pages

  • Re: dataset Performence Issue
    ... of data to hold in memory in some applications. ... Network and disk costs see to that. ... First of all, Dataset is guess what - Managed Code, and to send all the ... You need paging mechanisms etc. which are ...
    (microsoft.public.dotnet.framework.adonet)
  • Re: Explorer Eating Memory When Viewing VERY Large Folders
    ... I meant on the disk. ... add up to quite a bit of wasted space, especially with the larger cluster ... In a virtual memory system like Windows XP, the ideal size of disk clusters ... An NTFS partition of almost any size you are likely to meet will use this, ...
    (microsoft.public.windowsxp.perform_maintain)
  • Proposed Assembler Commands
    ... ACM Automatically Clear Memory ... BKCRDR Backspace Card Reader ... BKSPD Backspace Disk ... EIAO Execute In Any Order ...
    (sci.electronics.design)
  • Re: Problems reclaiming VM cache = XFree86 startup annoyance
    ... :I'm not sure what is overloading the disk other than the large amounts ... :of paging that are initiated the moment I invoke "startx". ... :allocable memory), with the subtle distinction being that "Cache" ... :pages used to hold disk blocks now discarded, then why not allocate ...
    (freebsd-stable)
  • Re: Slow performance
    ... Leaving your computer on 24/7 means that if any programme has a memory ... The Disk Defragmenter report shows you have minimal free disk space. ... select Properties, General, Advanced and check the box before Compress ... Volume fragmentation ...
    (microsoft.public.windowsxp.perform_maintain)