Splitting Files into Multiple Folders for Read Performance
- From: BlackStarPro <jreid@xxxxxxxxxxxxxxxxxxx>
- Date: Thu, 8 May 2008 08:31:14 -0700 (PDT)
Hi,
I am a .NET developer for a company that has been producing upward of
100,000 PDF reports per month for several years, which has now turned
into a huge problem. We are releasing a new reporting system that
allows for dynamic reports and custom dashboards, however they want
ALL their legacy PDF reports to still be available through our new
reporting system.
We decided the best thing would be to build a simple app that iterated
through the millions of files and folders and re-organize them onto a
different (better performing) HD array. Right now the files are in a
very elaborate folder structure, the names of the folders are
essentially parameters for organizing the files. Our idea was to have
our application pull the files out and when doing so break its
original path up into parameters and write the parameters into a
database, then simply rename the PDF file something unique and store
them in a simpler folder structure, also keeping track of its new
location.
I had heard that for read performance reasons a folder shouldn't have
more than 2000 files in it. Is there anyone that can verify this?
Using that knowledge (which I don't know if true or not) we could have
main folders incrementally numbered (1,2,3,4, etc.) each containing
2000 uniquely named files.
example:
c:\1\ (would hold files 1-2000.pdf)
c:\2\ (would hold files 2001-4000.pdf) etc.
Our app would just keep track of the old folder parameters (structure)
and a reference to its new location.
Currently we are seeing major performance issues when we try to deal
with the millions of files and folders that are in there right now.
Can anyone recommend a better solution or concur with the above
solution? Any suggestions would help.
.
- Follow-Ups:
- Re: Splitting Files into Multiple Folders for Read Performance
- From: Pegasus \(MVP\)
- Re: Splitting Files into Multiple Folders for Read Performance
- Prev by Date: Max recommened SAN disk size
- Next by Date: Re: Splitting Files into Multiple Folders for Read Performance
- Previous by thread: Max recommened SAN disk size
- Next by thread: Re: Splitting Files into Multiple Folders for Read Performance
- Index(es):
Relevant Pages
|
Loading
