Re: more then 105000 folders in one folder

From: Ken Varn (nospam)
Date: 12/02/04 

Date: Thu, 2 Dec 2004 09:01:44 -0500

Our application experienced something similar to your problem. Not sure if
our application is entirely related to how you write files, but this may
help. We may store over 4,000,000 files per folder using multiple folders.
The performance on our system would degrade over time. Usually when the
drive was about 60% full.

We opened a call with Microsoft about the same performance issues that you
described. Our system performance was going in the tank whenever we
starting getting a lot of files on the drive. NTFS is supposedly capable of
handling this, but nonetheless, the performance problems resulted. MS told
us that we may be causing a problem with over allocation of small clusters
and this could lead to the performance problems. In order to avoid the
problem, they suggested, when creating a file, to set the maximum size of
the file prior to writing the data. Believe it or not, this helped a lot.
Here is a brief rundown of their solution which corrected the problem:

PROBLEM
========
Disk subsystem degrades with writing millions of files

RESOLUTION
============
Open the file
Set the filesize to the expected size using SetEndOfFile.
Do the write

Rather than just
Open the file
Do the write

Hope this helps. 

-- 
-----------------------------------
Ken Varn
Senior Software Engineer
Diebold Inc.
EmailID = varnk
Domain = Diebold.com
-----------------------------------
"Marco J.L." <luyendyk@yahoo.com.news> wrote in message
news:OUbqSYv1EHA.3408@tk2msftngp13.phx.gbl...
> Hello,
>
> we have an performance issue, sometimes our server has a lack of
> performance.
> Operating System Windows 2000 and Windows 2003 (both give the performance
> problemens)
> When opening the Explorer and only the drive which contains the files (not
> the folder)  the explorer is using 50 - 90 % of the CPU and the memory is
> fluctuating between 60 and 110 kb,  it fluctuates continouisly.
>
> I think that the problem is caused by this :
> We've an application which uses in ONE single folder more then 105.000
> folders, each folder contains ONE single file.
>
> I told it to our programmers but they don't see any problems with it. I've
> some questions :
> ** 1 ** Has Microsoft a recommendation about the maximum number of files
in
> one folder?
> ** 2 ** Which performance counters are the best to use to monitor if this
is
> a problem?
>
> thank you in advance,
>
> Marco J.L.
> luyendyk@yahoo.newscom
>
>
>

Relevant Pages

  • Re: more then 105000 folders in one folder
    ... The maximum number of objects in an NTFS partition is 4,294,967,295. ... "Marco J.L." ... > We've an application which uses in ONE single folder more then 105.000 ... > folders, each folder contains ONE single file. ...
    (microsoft.public.win2000.setup)
  • Re: Checking if file being accessed
    ... how does that "give me up to 15 seconds to finish writing the ... loops. ... being put in the folder), the this should give me an error and go to the ... accessed (which I know because the FileStream error happens after the ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Writers notebook, valuable tool or waste of time?
    ... You're supposed to spend some time each day just writing in your writer's notebook. ... snippets of overheard conversation, a memory, a description of how you're feeling at that moment, a 10 minute free-write -- whatever flits through your mind as you're sitting there with the notebook open. ... I used to use cardboard folder, but now I use see-through plastic ones. ... Invent a character who owns these things. ...
    (rec.arts.sf.composition)
  • Re: What is a writable CD folder?
    ... On the Windows XP Native CD Writing. ... George Hester ... >> Is this folder put on a CDR by Windows XP's native CD writing or is ...
    (microsoft.public.windowsxp.general)
  • Re: more then 105000 folders in one folder
    ... did you or could you disable DOS short file names? ... Marco J.L. wrote: ... > We've an application which uses in ONE single folder more then 105.000 ...
    (microsoft.public.win2000.setup)
Loading