Re: Explorer Eating Memory When Viewing VERY Large Folders

Alec

Please see inline:

"Alec S." <@> wrote in message
news:uojvVCi%23GHA.3480@xxxxxxxxxxxxxxxxxxxxxxx

Snip

I meant on the disk. When you've got thousands of small files, the slack
space?the unused part of the last cluster of each file?can
add up to quite a bit of wasted space, especially with the larger cluster
sizes that are used these days (8KB clusters are the
minimum for partitions>50GB). Of course I'm talking FAT here, but if I
remember correctly, NTFS has waste as well.


The comments by Alec Nicholl present a balanced view:

Considerations of economy and performance

In a virtual memory system like Windows XP, the ideal size of disk clusters
matches the internal ?page size? used by the Intel processors ? 4 kilobytes.
An NTFS partition of almost any size you are likely to meet will use this,
but it is only used in FAT32 up to an 8 GB partition. Above that, the
cluster size for FAT increases, and the wastefulness of the ?cluster
overhang? grows. (For a table of the varying default cluster sizes used by
FAT16, FAT32, and Win XP?s version of NTFS, for partitions of varying sizes,
click here.)

On the other hand NTFS takes much more space for holding descriptive
information on every file in that file?s own block in the Master File Table
(MFT). This can use quite a large proportion of the disk, though this is
offset by a possibility that the data of a very small file may be stored
entirely in its MFT block. Because NTFS holds significant amounts of these
structures in memory, it places larger demands on memory than does FAT.

Searching directories in NTFS uses a more efficient stucture for its access
to files, so searching a FAT partition is a slower process in big
directories. Scanning the FAT for the pieces of a fragmented file is also
slower. On the other hand, NTFS carries the overhead of maintaining the
?journalized? recovery.

http://aumha.org/win5/a/ntfs.htm

Adrian does not say how his disk is formatted or the size of the disk /
partition. He also does not say whether the folder is on his computer or on
another computer in the network!

If a user is viewing files in a folder and does not close a file
after seeing the contents then this will also waste available memory.

Explorer is pretty bad in the way it handles files. It leaves open
handles, allocated memory, etc. That's why the memory usage of
explorer.exe goes up and up over a session, and files (particularly
graphics files) become locked and cannot be deleted.


Not my forte. Do open handles, allocated memory, etc impact significantly on
system performance when the system does not resort to using virtual save for
specific tasks requiring virtual memory?

snip


I was referring to your question "You could look at your problem from the
opposite direction! Why do I
need to have so many large files?" It may not be up to the user at all;
it is not necessarily the design that the OP has created,
but rather the way that a program they use ships, and they are getting
peeved by the usage caused by the thousands of files that the
program has.


True. Again Adrian does not mention what programme created the files and
unfortunately neither of us thought to ask.


--
Alec S.
news/alec->synetech/cjb/net



--

Regards.

Gerry
~~~~
FCA
Stourport, England

Enquire, plan and execute
~~~~~~~~~~~~~~~~~~~


.

Relevant Pages

  • Re: FAT to NTFS
    ... Bill Ridgeway added these comments in the current discussion du ... there is said to be an improvement of speed when using NTFS ... gig CPU and only 512 meg memory and a relatively slow HD and a ... reformat the original disk and either copy or clone it back ...
    (microsoft.public.windowsxp.hardware)
  • Re: Att. Alex Nichol -VM cont.
    ... I find documentation that says a 4kb page in memory is written to the hard ... manner that a 4kb memory page is equal to a 4kb cluster written on the disk". ... So where is the basis for saying since 4kb paging in memory that 4kb clusters ...
    (microsoft.public.windowsxp.general)
  • RE: Incorrect Disk Space reading
    ... How to Locate and Correct Disk Space Problems on NTFS Volumes in Windows XP ... cannot find the cause or locate the folders and files that cause the NTFS ... The Cluster Size Is Too Large ...
    (microsoft.public.windowsxp.hardware)
  • Att. Alex Nichol -VM cont.
    ... your recommendation that 4k cluster sizes is optimal for the pagefile. ... Intel architecture involves three memory management models and they are FLAT ... Section 3.6 Paging ... page swapped out to disk is done in 4k incriments. ...
    (microsoft.public.windowsxp.general)
  • Re: Cluster hang after disk-check failure
    ... Actually, "Error 64" is usually nothing to do with disk, but an indication ... that you are running out of system resources, such as memory. ... responding after a disk-check failure on a cluster file share. ... The disk resource was on-line and there were no indications of any disk ...
    (microsoft.public.windows.server.clustering)
Quantcast