Re: NTFS drive capacity/utilization percentage?
From: Beanbag (Beanbag_at_discussions.microsoft.com)
Date: 06/18/04
- Next message: Anders: "Re: Printing Windows Explorer file lists?"
- Previous message: NTNEWS: "DFS and FRS File Replication in Windows 2003"
- In reply to: usenetacct_at_lycos.com: "Re: NTFS drive capacity/utilization percentage?"
- Next in thread: Al Dykes: "Re: NTFS drive capacity/utilization percentage?"
- Reply: Al Dykes: "Re: NTFS drive capacity/utilization percentage?"
- Messages sorted by: [ date ] [ thread ]
Date: Fri, 18 Jun 2004 00:42:01 -0700
"usenetacct@lycos.com" wrote:
> sure, we need to keep files created by jobs online for several months
> & that may be going to a year. This is currently 120,000 new files
> per day & could go up dramatically very soon with a new client. Size
> of them ranges from 4MB to 1KB, mostly on the smaller end (<50KB). We
> constantly delete files to make room for new ones, so each day we're
> deleting anywhere from 120,000 to 500,000 of these files while adding
> those I mention above. They're stored in a directory structure based
> on the
> year-month-day-hour-minute-job#. When I say millions of files, I am
> being literal. Tens of millions, to be accurate.
>
> We also have a single directory that contains the original files the
> work is based on, & we have a weeks' worth of data in that, which is
> ~150,000 files in that single directory. We delete old data from here
> every day to make room for the new also. This system runs 24/7 and we
> cannot take it down to defragment, run chkdsk, or anything else. If
> we have disk corruption, we have to format & restore from tape,
> because the sheer number of files means running chkdsk takes many days
> to run.
>
> At the moment, we can store 3 months of data in on an 880GB array only
> if we literally let the system run constantly at extremely high
> utilization rates. We may need to expand it to allow a years worth.
> At current volumes, that would mean a single 4 terabyte volume. If we
> continue growing at current rates (very fast), 4 terabytes is only the
> beginning.
>
> thanks
>
Is there a special reason why you store all data on a single volume? Otherwise I would advise to divide your data up and store it on several smaller volumes. This probably minimizes the effect of data corruption on your work and should give you the ability to do a fsck rather than recover the data from tape. It will also increase your overall file system performance and conforms to the Microsoft performance tuning guidelines.
There are defragmentation tools available that do a much more sophisticated job than the built-in defrag. You can do online defragmentation or defragment with less than 15% free space on your volume, schedule jobs and much more. Try OO Defrag (www.oo-software.com) for example.
When you fill your volumes up to 99% this will result in MFT fragmentation in addition to file fragmentation which slows down the file system, particularly with such a large amount of files. See MS KB article 174619 for details.
I would not recommend to use compression. This will probably not result in a faster file system, rather in a slower one.
When you use Windows Server 2003 you can tune the file system via several registry parameters, you can disable the update of the Last Access Time attribute or the generation of short names in the 8.3 naming convention. I don't know if these parameters are available under Windows 2000. Check the Windows Server 2003 Performance Tuning Guidelines for details.
Cheers
Frank
- Next message: Anders: "Re: Printing Windows Explorer file lists?"
- Previous message: NTNEWS: "DFS and FRS File Replication in Windows 2003"
- In reply to: usenetacct_at_lycos.com: "Re: NTFS drive capacity/utilization percentage?"
- Next in thread: Al Dykes: "Re: NTFS drive capacity/utilization percentage?"
- Reply: Al Dykes: "Re: NTFS drive capacity/utilization percentage?"
- Messages sorted by: [ date ] [ thread ]
Relevant Pages
|
