Help: Large Incoming FTP Files Using Entire Server Memory

• Previous message: Dave: "Users keep losing FTP access"
• Messages sorted by: [ date ] [ thread ]

Date: Fri, 11 Jun 2004 13:59:14 -0700

IIS FTP Group:

I'm scripting an FTP process to backup large disk image files from a Linux
server (VMWare ESX 2.1) to a Windows 2000 Server (Dell NAS) using the FTP
server in IIS 5.0. While I have no problem getting things to work (I've
used IIS extensively), I'm encountering process behavior which severely
degrades performance on the Windows server.

The files we're FTPing are about 1-4GB in size. As soon as an FTP put
process starts to the Windows server, available system memory drops quickly
from ~800MB down to around 5MB (yes, 5MB!) as IIS apparently tries to cache
the entire contents of the FTP file in memory before committing it to disk.
I confirmed this by stopping an FTP process mid-put (after ~2GB had been
sent) and found that no portion of the file wrote to disk -- the file
disappeared from the directory and available system memory returned to
normal after about 5 seconds.

The strange thing is that the excessive memory use can't be accounted for in
Task Manager by looking at the running processes (inetinfo.exe doesn't grow
beyond a few MB), so the system must be performing the cache operation
behind the scenes. In any event, when the memory gets down to 5MB on a 1GB
server, page faults explode, the disks go wild, FTP throughput goes from
25MB/s to 3 MB/s and every aspect of the server slows to a crawl until the
file push is complete.

I've looked around, but found no simple remedy for this issue. Does anyone
know of a way to instruct IIS to flush memory cache to disk in increments or
otherwise control memory use of an FTP process involving large files?
What's the secret reg key?!?!? ;-) It's unbelievable that I can take down a
decently powerful 1U server (P4, 1GB RAM, 4 IDE drives in hardware RAID5
running on a gigabit ethernet network) with a simple FTP push on account of
the FTP server trying to cache the entire contents of each file in memory!

Arggghhhhh...
Bradley

• Previous message: Dave: "Users keep losing FTP access"
• Messages sorted by: [ date ] [ thread ]