Re: NTFS block allocation policy (how does it work?)

From: Pat [MSFT] (patfilot_at_online.microsoft.com)
Date: 02/27/05 

Date: Sat, 26 Feb 2005 21:23:30 -0800

NTFS zeros the file whenever you extend it. If you just use WriteFile() to
extend the file, the same zero'ing occurs. This is for C2 security
requirements, to prevent folks from extending their files over other peoples
files and taking control of the data.

If you extend a file a little at a time, then for every extension:
Seek to cluster location
Zero out enough space for the addition
Seek back to the start of the write
Write the file data

(so, 2 seeks for the write)

Also, if something else has written to the adjoining cluster in the interim
since the last write, then the file will fragment. This happens all the
time in logging applications.

So, if you pre-allocate the file, you only pay for the zeroing 1 time. This
will prevent any other files from intruding on your nicely sequential
allocation and optimize write speeds. If you only pay the price once, then
the the runtime performance is the same as bypassing the zeroing. The
clusters do not get zero'd until you call SetEndOfFile(). So the order is:

CreateFile
SetFilePointer (size of file you intend to create)
SetEndOfFile (commit the file)
SetFilePointer(beginning)
start writing

No more zeroing will occur unless you overrun the file size allocated.
Also, fragmentation will be minimized. The zeroing happens very, very fast
b/c there is no seeking. A single seek to the start of the file, then
sequential writes so the writes occur at the disk speed.

You can bypass the zeroing of files by using the SetFileValidData(). This
requires an elevated privilege to run (SE_MANAGE_VOLUME_NAME), so depending
on the security context of your application you may have problems; also that
is only available on WinXP and later so if you are targeting Win2k systems
it won't work. Whereas the SetFilePointer can be executed by any user
context that has access to the file. Since (from your description) the
process creates the file as well, this won't be a problem.

Pat

"George M. Garner Jr." <gmgarner@newsgroup.nospam> wrote in message
news:eU9bWe9GFHA.3484@TK2MSFTNGP12.phx.gbl...
> Pat,
>
>> If you know at least the approximate size of the file, the best thing to
>> do would be to pre-allocate the file, then go back and write to it.<
>
> Could you be more specific as to how to allocate the file. I see the
> function SetFileValidData. But that assumes that you already have
> allocated the file. Elsewhere I have read that one should not use
> SetFilePointer(Ex) because of the added overhead of writing zeros to the
> file. Is there a way to allocate the file without incurring the overhead
> of immediately zeroing out the clusters. I know the pagefile does this
> but I do not see an api that permits it.
>
> Regards,
>
> George.
>

Relevant Pages

  • Re: NTFS block allocation policy (how does it work?)
    ... NTFS zeros the file whenever you extend it. ... So, if you pre-allocate the file, you only pay for the zeroing 1 time. ... Is there a way to allocate the file without incurring the overhead ...
    (microsoft.public.development.device.drivers)
  • Re: NTFS block allocation policy (how does it work?)
    ... NTFS zeros the file whenever you extend it. ... So, if you pre-allocate the file, you only pay for the zeroing 1 time. ... Is there a way to allocate the file without incurring the overhead ...
    (microsoft.public.windows.file_system)
  • Re: NTFS block allocation policy (how does it work?)
    ... NTFS zeros the file whenever you extend it. ... So, if you pre-allocate the file, you only pay for the zeroing 1 time. ... Is there a way to allocate the file without incurring the overhead ...
    (microsoft.public.win2000.file_system)
  • Re: realloc but not copy
    ... reuse if possible or allocate in a different part of RAM if resizing is ... but be able to avoid the rebuild if an "extend" is possible. ...
    (comp.lang.c)