Re: Choosing location for large temp files?

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance



On Sun, 02 Nov 2008 12:04:36 -0700, MC <for.address.look@xxxxxxxxxxxxxxxxxxxxxxx> wrote:

I'm developing an application that needs a large (up to 4 GB) amount of temporary file space on a local (not network-attached) disk. What are some good tactics to use in a C# program to make sure the swap space is suitable? There can be some interaction with the user to choose the swap file location, but the more the program can find out about it, the better.

Can you be more specific about what criteria defines "suitable"?

I would think that the two major questions are: capacity, and i/o speed. But, from a file-system coherence point of view, if we assume this is truly a temporary file, deleted when the application exits normally, then you may prefer to store the file in the user's own "Application Data" subdirectory somewhere, as long as that meets whatever other criteria you have.

Wherever you put the file, you should be careful, of course, to provide features in your program to try to recover from errors that might have prevented the deletion of the temp file; otherwise, the disk could be consumed quickly.

To give the file system the best chance to optimize how the file is stored on disk, you'll want to seek to the file's last extent when creating it so that the file system can know how large the file is going to be and can allocate space for it. If the file system has logic in it to maintain contiguous file structure when possible, it will then be able to do that. Otherwise, you may see performance suffer as the file gets fragmented from repeated extensions as you decide you need more and more space.

Extending the file immediately on creation also has the advantage that it allows you to confirm capacity right away. That way, you won't run into problems later not being able to append to the file when you need to.

As for i/o speed, once you've created the file, you can run some profiling code to write and read from the file, timing access to the file to ensure adequate performance. For some applications, this is not so important and you might just wind up warning the user that if they don't show you a faster volume to use for the file, throughput will be lower. For other applications, such as video capture or DVD burning, i/o speed may be critical and you might want to refuse to use the volume altogether if it can't keep up with your minimum speed requirements.

That's what _I_ think of when I think "suitable". But if the above doesn't seem to address your question, you might want to clarify what "suitable" means to you. :)

Pete
.



Relevant Pages

  • Re: mdconfig unable to allocate memory
    ... > one ensure that all the memory is being used for suitable file caching ... > disk, really; I assumed that it would immediately start dumping ... > anything I put in it, into swap, causing disk access. ... file system with 4GB of RAM without a problem, ...
    (freebsd-current)
  • Summary: Root cause of swap space full
    ... The only way I could see that this particular file system causing the ... w/ no disk mounted under it. ... I'm assuming that the filesystem you mention is a VxFS filesystem. ... can't think of any reason that filling it would cause a swap issue. ...
    (SunManagers)
  • Re: what to do about "cannot dump to dumpdev hd(1/41): space for
    ... hard disk showing 255 hds, ... If the the swap was ... If a panic reports 0 blocks of dump space, ... Changing the starting block of a valid file system ...
    (comp.unix.sco.misc)
  • Re: running FreeBSD 4.10 with 16MB RAM
    ... >> That's a first to read about a lack of inodes on swap. ... > The Disk is 3GB in size. ... Whereever you have a file system you may run out ...
    (comp.unix.bsd.freebsd.misc)
  • disktab for VPC Disk
    ... I'm trying to optimize the disk access of NEXTSTEP on Virtual PC 5.0.4. ... Maximum size of NeXT file system is 2GB. ... programs in the initial 8 sectors of the device. ... The number of cylinders per cylinder group in a file system. ...
    (microsoft.public.mac.virtualpc)