Re: Index.dat file is almost 9MB, and it takes a long time to writ

From: Danor (danor456L_at_passport.com.(donotspam))
Date: 10/31/04 

Date: Sun, 31 Oct 2004 15:59:01 -0800

"V Green" wrote:

>
> "Shenan Stanley" <news_helper@hushmail.com> wrote in message
> news:eLvT0F3vEHA.3228@TK2MSFTNGP12.phx.gbl...
> > crissssss.......... wrote:
....
> >
> > Uhm.. Read again.
> >
> > I did.
>
> No you didn't.
>
> You just told them to go find out for themselves and gave them a link.
> Not the same thing at all.
>
> You could have answered the question yourself, in your own words.
>
> Although you may view this as a personal attack,
> I'm just curious as to why this method (passing a reference to other
> material with little or no real interaction) seems to constitute the bulk
> of "answers" on this, or for that matter, any IT related NG. It's like
> rubber-stamping things. KLUNK-"OK, here's your answer." - "NEXT!"
>
> Do folks really not have the time to spare for a little typed response
> of their own invention?
>
> Or is it wrong of me to be appreciative of the personal insights gained
> from those who answer my questions with their experience on the same
> issue of which I have inquired?
>
> Anyway, won't waste more of your time-the OP was happy you solved
> their issue, and that's what matters most, I guess.
>
I started my Reply to this last night (Cymbal Man freq.'s was last post),
but when I read V Green's second subsequent post (above), decided I'd stick
this in anyways for the benefit of Crissssss........'s and anyone else who's
interested's benefit.
Go and get yourself a tall one, folks...

Hello, Crissssss..........

It seems that several posts have dodged, jinked, twisted or completely
ignored what your question was. And since you restated it more than once,
I'll try to answer *that* for you and stay out of the falderal (although IMO,
you have legitimate reason to more-than-just-wonder about the true usage MS
employs out of the growing index.dat files discussed - or any web-operator
for that matter that knows how to read index.dat files - but that's a
diatribe for another forum...).

The simple answer to your question "...would (they) become enormous and slow
down anyones computer...?", is: well, in a word, yes,,, *eventually*. But how
many days are in an eventually, and just how soon - if ever - do or will
*you* need to do something?

(This is quite a long-but-most-informative spill, but if one felt the need
to ask this question, it is evident they are unaware of what follws)

First, though, there is one subjective adjective in your question that
requires more clarification for this to be understood in a practical and
meaningful sense (for you, and anyone else who would ask this): "Enormous".
Just how big is enormous?
If the .dat file in question (or truly *any* other file/type, as far as this
answer goes) is now 1Mb in size and it grows to 10Mb in just one year, that
means it is reasonable to project it will likely grow to 100Mb in ten years
(twice the reasonable expectation of the lifespan of any recent or current
retail/consumer HDD).
Further, if this file resides in a volume that has 6000Mb (e.g. 6Gb) or more
of Free Disk Space (FDS), then a 100Mb file is hardly worth much effort to
adjust your system over. BUT, if the disk drive volume FDS is only 600Mb,
then this 100Mb is a more-significant amount.
Of and by itself, though, even that 100Mb may not be considered "enormous"
on this scale. So if we were then to adjust slightly our previous
base-to-growth estimate of 1Mb to say 3Mb - the "end" filesize becomes
300Mb... well that just puts a whole new light onto "enormous" where
something may definitely need doing.

A major factor that may significantly impact this issue is:
How many Hard Disk Drives (HDDs) are there in the PC? (Or, to be precisely
accurate, how many disk *volumes*? - clarity on this distinction is below) if
more than one, then most of what follows is of somewhat *lesser* import, but
Not by ANY means of *no* import at all.

The heart of the answer to your question surrounds - or is surrounded BY - a
number of central and related issues that have long existed in Windows OS's,
some unique features of NT that XP has adopted, plus a few wrinkles that XP
brings into play itself. At the core of all of it, though, is this thing
called a "Swap file" - in XP the file is: pagefile.sys.
Pagefile.sys has many names that are bantied about by Techs and
deeply-entrenched computer enthusiasts: "page file", "swapfile", "paging
memory", and "Virtual Memory". MS Help and Knowledgebase documentation
typically refer to it using the term "Virtual Memory", but also use the 1st
two as well ("paging memory" is very old school which I haven't seen for
years, but is still correct).
The swapfile is what Windows (and many other OS's) use to "extend" it's
physical RAM memory capabilites, so an understanging of RAM is also called
for. The primary purpose of RAM is the place where the CPU's "HoneyDoThisNow"
list (e.g. program) is continusly updated and added to. The CPU canNOT
execute a program from wherever it lives out on a disk volume (the C:\Program
Files... or any other place); those reasons are too complex and way beyond
the present scope, just trust me in that it CAN'T function in any efficient
capacity that way. A program that is to be run by the CPU, *must* be placed
on the HoneyDoThisNow list kept in RAM.
Virtual Memory - this RAM "extension" - was invented because most programs
by themselves typically are bigger than the total available RAM capacity of a
PC, so to put an entire program's "HoneyDoThisNow List" into RAM to start
with would: a)leave no other room for other programs, b) be a hugely
inefficient method - if not outright impossible - for performing several
Tasks/windows at the same time (it could not be effectively done).
So, to accomplish multi-tasking and also be most efficient by having only
very short and quick to complete "HoneyDoThisNow Lists" in the queue for the
CPU, programs are broken up into lots of little chunks that are copied out to
known and precisely retrievable loactions in the swapfile whenever that
program is launched.
A simplistic way of how this RAM/pagefile works is to think of it as using a
sticky-note attached on to the bottom of the Main Primary sticky note, and
then more sticky-notes daisy-chained onto those making a long list of
sticky-notes, where each individual process of the program's performance is
on one and only one sticky-note, and then only the particular sticky-note
that is immediately needed is detached, brought up to your glasses so you can
read it, then put back in its place for later reference).
That's how Windows uses the "small" amount of RAM in PCs for multi-tasking,
say: 1 My Computer window, 4 IE browser windows, one for Notepad, another for
eMail, an EXCEL Spread***, and a chat in Windows Messenger, all while
you're listening to the London Philharmonic play Beeethoven's 5th through
your CD player in simultaneous, seamless, crash-free syncrocity... by copying
out the majority of a program's pieces that it knows it may eventually need,
to a *precisely* pre-known spot of the disk (e.g. pagefile.sys) so it can
retreive it *really* fast and won't have to wait for it to be "found" then
read from
where-ever-the-actual-little-piece-of-it-really-lives-out-in-the-suburbs-of-HDD-land.
Every one of the open Windows described above (plus many other programs:
likely every one in your System Tray - that area where the clock is - and
some others), has one of these sets of sticky-notes - each it's own color
paper! - out in pagefile.sys waiting to be called in to active duty when
needed.
A practical example of how this *concept* actually works is this (attn any
programmer actually reading this, the following is for describing VM/PF
*concept*, not actual operation, ok?): think of being in a word processor
type program (which one doesn't matter as long as it is Windows-based). You
want to cut-and-paste a sentence of the file from point A to point B, which
is a specific and pre-defined kind of "Edit" (as on the "Edit" menubar
pulldown).
If you take a look at the Edit menubar item in Notepad, for example, you see
a list of choices:
 - Undo (may or may not ge greyed-out: if you have already done a previous
action that is Undo-able, then this choice is available);
 - if *any* appropriate data/text/object whatever, has been previously cut
or copied onto the Clipboard (the 'place' where something goes whenever you
Cut or Copy, no matter "how" you cut or copy), then Paste is an available
option;
 - if you have a cut/copy/or deleteable entity selected/highlithed (like we
have a section of text now selected), then Cut, Copy and Delete are all
available;
 - Find, Find next, Replace, Go To, Select All, and Time/Date are also all
available options on the Edit menu pulldown...
You may choose from all of these available options what you want the word
processor to "do" next with this currently active selection (of text here,
but could be anything else, like say an inserted .jpg picture file object).
Each one of these above listed "options" requires a different specific set
of "HoneyDoThisNow" set of exact instructions to the CPU, and each one is
therefore directly and specifically related to one particular sticky-note in
the long chain of sticky-notes that Windows collected together (which *is*
the entire word processor program) and loaded into the swapfile when you
launched "Notepad/whatever" in the first place. Each sticky-note has the
small but exact list of specific instructions for the CPU to carry out in
order to complete its particular task/singular operation (like "Cut",
"Paste", "Delete" or "Undo").

Back to our example: *after* having pre-selected the section of text to
move... off the top of my head I can think of at least three different ways
that you can accomplish the "cut" part of the 3-step version of this
operation (a 4th method is some programs allow you to just hover over the
selection and left-click-drag it to where you want it to be, but Notepad
doesn't support this action, and most people tend to Cut, click to point B,
then Paste):
 Method 1) Mouse up to and click on "Edit" on the menubar, then select "Cut";
             2) or simply type CTRL+X as the "Cut" item on the above menubar
pulldown suggests;
             3) or mouse up to and click the "Scissors/Cut" Toolbar button
Once the sentence has been selected, but before you actually do 1, 2, or 3
above, the CPU can be efficient by performing pieces of other programs, so it
does just that until it hears from the word processing program what you want
it to do for you next (i.e. its listening to the port that your Chat session
is using if any new data has arrived to send it to that Window and "ding" you
to alert you that it has, in fact, arrived).
Notepad/whatever meanwhile is sitting there waiting for you to type/click on
something else so it can know what to do *then* with the selection you just
made and thereby determine which sticky-note has the right set of
instructions for the operation you just ordered, to send the identity of that
sticky-note to the CPU's "HoneyDoThisNow List" so the CPU will put it in its
queue to then get done.

All this is so during the time from having selected the text to the time you
actually employ method 1, 2, or 3, the OS would have been doing other
sticky-notes for other processes (reading from the swapfile into actual RAM -
the only place where the CPU reads instructions from), this "long" time, BTW,
will have been a *really long* time to the computer, even if you did option 2
- the fastest human possibility and took only 1/10th of a second between
text-select and CTRL+X; for a 2.?GHz CPU that 1/10th of a second represents
2?0,000 operations it could be performing something else while "waiting" for
your CTRL+X command to actually arrive... so, then, how long did it actually
take you to do *your* last 2?0,000 operations of *your* life, eh?)
It's partly because micro-processors (e.g. CPUs and other chips on the
motherboard) are so fast and can do SO MUCH in between human keystrokes and
mouse clicks that this "swapfile" thing exists, so that the computer can
manage multiple tasks (windows) at the same time within a relatively small
area of acutal memory storage (RAM).
For Windows to know *exactly* where the instructions it needs to perform at
any moment for *whatever* Task/window is yelling for it to do something are
being kept in order to have the specific sticky-note that those instructions
are actually on be brought up to its glasses (into RAM) ASAP so it can read
and execute them while it is waiting for you to complete *your* 2?0,000+
equivalent operations at your speed, is what the swapfile accomplishes.
After you complete the "Cut" step, you then click to where you want to put
it (this process to us humans is just a simple click, but in fact it is truly
represented by its own individual sticky-note set of instructions for the
CPU's "HoneyDoThisNow List"), and then followed by, of course, the final
Paste sticky-note process.

Although NT & XP are far superior in the implementation and management of
the swapfile than Win9x, they both still need some "elbow room" in the disk
volume(s) that "they" occupy (plurals explained below) in order to function
at peak efficiency. And, after all, that is what we users want, isn't it? A
PC that is faster than a speeding...NASCar?
But, if the volume that they reside in becomes gloated, or severly
defragmented or worse, *both*, then Windows performance just nosedives right
into the ground.
It is because of these factors that the gloating of files can impact anyones
Windows performance, which is the (very reasonable) heart of your question.

What can you do about it? Several things. Regularly defragment your volumes
that contain a swapfile for one. But before I get into others, if your
situation is one of a real need for positive action to correct for these
gloating index.dat/or-whatever file(s), then it is appropriate for you to
have a basic understanding of volumes on HDD's because you will need to
utilize *that* knowledge to plan your own individual response.

A lot of people refer to the disk drive (i.e. the entire storage capacity
area of the drive of and by itself) as the meaning of what a volume is (e.g.
"if you have one HDD, you only have one storage volume").
But that is not correct.
A single HDD can be - and often is - partitioned to hold many volumes (even
volumes for different OS's all on the same physical drive).
You're better off to think of a HDD like one of those entertainment centers
that has little holes on the inside vertical panels that you stick pins in
that the shelves rest on. *You* decide what holes to use to make the open
spaces the right size for your TV, DVD player and VCR, etc., to make the best
use of the fixed space the entertainment center provides. In other words, you
"partition" the space by setting the shelves, thus making one "volume" for
your TV, another for the VCR, another for the Stereo, and smaller ones for
the tapes and DVDs, etc.
Another way to describe volumes is as different pens on a farm: pigs are in
one, chickens another, goats, yet another, then the cows.... Each pen is a
different size (or capacity...) each for its own kind of critters, but they
still all live on the same farm just seperated from each other by
chicken-wire or fencing, in their own defined little regions).
You don't really need to know about volumes to use a PC, but if you want to
take a real interest in the *Performance* of your PC, it's a must.

To help you, though, Windows XP has a great tool to see what the PC does
have in the way of volumes: its called Disk Management.
Anywhere you can find a My Computer icon (the Start Menu, Desktop,
whatever), Rt-click that MC icon and select Manage from the Context Menu that
pops up.
When the Computer Management console opens, look for Disk Management in the
left window pane and click that.
The Disk Management info then appears in the right window-pane (for best
viewing results, expand this window to full-size).
Disk Management does not concern itself with floppy drives A: & B: but every
other drive letter which you have a device (that is configured in Device
Manager) attached to should show up.
The new window pane is split into top and bottom halves, but for the
purposes of this exercise, you'll get the most out of the bottom half.
The left-most column shows you all physical HDDs (and other devices like
CD-ROM, DVD, Zip, etc) that are somehow attached to the system; to the right
of each of those boxes is a graphical representation of the contents of that
device.

In the case of a HDD, Disk Management will show you a box for each volume on
the drive. If there is only one big wide box, then there is only one volume
(be careful when looking at the edges/outline of the box, because complex
volume structures can be a little tricky to "see" in this graphic; I suspect
though that you will only see one volume on your HDD drive 0, as you are not
likely to have any of these complex volumes.... yet!;-().
If any HDD has only one formatted volume but also still has areas of unused
space (*these* unformatted as-yet-unused areas are properly referred to as
partitions; an area which has been file-system formatted - like FAT or NTFS -
is no longer a partition, it has become a volume). Partition(s) - if present
- will be designated as Unused areas and will also each be represented by a
(different-colored-outline) graphical box.

Although unlikely, it is possible that a tiny area of unused space exists at
the very beginning of HDD 0 - the first HDD - where your C: volume is likely
to be; and then the C: volume, possibly followed by another empty partition.
Don't ever try to access/use this "partition".

Disk Management will tell you how many volumes you have and on what drives
they live.

If you have Unused partition(s)/space on your present HDD, you may select it
and format it into a new volume, give it a drive-letter assignment, and then
that space becomes an available data storage resource to your PC. This may
very will be an aspect of your total response to the gloating/pagefile.sys
performance issues under discussion.

Knowledge of:
 - how many and what speed capabilies the HDD(s) have *and* how they are
connected to the PC (e.g. PIO/UDMA mode & operational xfer speeds);
 - what volume(s) (and what file system they are each formatted with)
 - how much physical RAM is in place (and how willing you are to add to that
- see below)
is *all* important in evaluating your choice of what method(s) to employ to
diffuse/eliminate gloating enough, or to increase Windows performance.

If you have multiple NTFS volumes available on your system (either from
additional HDDs or multiple volumes on one HDD), your options are wider:
 - XP allows the pagefile.sys swapfile (MS calls it Virtual Memory in the
appropriate dialog-boxes you need to click through) to be "moved" from the C:
volume (its default location) to any other available volume *** CAUTION: Do
NOT choose external detachable volumes such as USB drives, Zip, etc., as they
may not always be attached);
   to accomplish this, go to Control Panel > System > Advanced tab >
Performance section [Settings] > Advanced tab > Virtual Memory [Change]
   here you can see all the Drive [volume labels] and the present space
allocated to paging (swap) files on each of the different volumes;
 - XP *further* allows you to set a paging file on more than one volume...
e.g. "spread it around". Windows will decide where on the volume to put the
pagefile.sys file, but you will have to tell it on what volume(s) you want
one placed.
   This is a very effective method to offload the swapfile-based disk I/O
traffic from competing with the additional Windows system volume I/O load
that is usually placed solely on the primary IDE Master channel in most
typical Windows installations. By placing (the) swapfile(s) in different
volume(s), you will spread the HDD I/O load to at least the alternate Slave
device or to the Secondary IDE channel altogether, thus significantly
improving disk I/O for Windows OS support.

 - Windows XP (Pro at least, I don't know if this is available in Home Ed,
but I suspect it is) also allows you to turn off disk paging altogether and
it will then use physical memory instead! Vastly suprior in paging I/O speed.
   To accomplish this just make sure that:
      you have a *large* amount of physical RAM installed: probably no less
than 1Gb but certainly no less than 768Mb;
      go to the Virtual Memory console as described above and set all
volumes to "No paging file"
   Although Windows will allow you to set no paging file on all volumes
regardless of the amount of RAM actually installed, with less than the 768Mb
mentioned, you will see numerous popup messages from windows complaining that
the Virtual Memory size is too low, or windows will simply crash altogether.
What else is new?
   *********** I would strongly urge against this method if you have LESS
than 512Mb of installed RAM, as you may not be able to get Windows back up on
its feet at all to re-activeate paging files without at least 512Mb of RAM,
unless you can and want to deal with Safe Mode ****************

If you have only one HDD, but multiple volumes on that drive, you can split
or move the paging file to the other volume(s) as described above.

If you have only one HDD and only one present volume but available/Unused
area(s) - partitions... - you can create new volume(s) in that space and
place paging file(s) there.

If you have only one HDD and it presently uses all available space, but its
a huge drive (by todays presently available HDD capacities I would classify
80Gb and up as "huge" for typical home-use PC implementations; however,
bigtime video and media applications do NOT fall under this heading) you may
just go along by defragmenting regularly and not worrying about anything at
all untill your available FDS falls below 10% of the total capacity.

If you have only one HDD with one volume, but still want to make it "higher
performance" by creating smaller and more efficient volumes, you will need a
third-party software partitioning tool like Partition Magic (which I've used
for years, is simple and easy for any User of any degree of expertise to use)
if you want to avoid reinstalling Windows from scratch.

Good luck, and as Yogurt said: "May the Schwarz be with you!"