Re: Data loss appending data to file
- From: "Alexander Grigoriev" <alegr@xxxxxxxxxxxxx>
- Date: Fri, 2 Dec 2005 07:48:14 -0800
I wonder if you have some crappy antivirus running with "scan on access"
mode active.
I also wonder if there is a problem with FILE_FLAG_SEQUENTIAL_SCAN flag for
write access.
"Yannick Fortin" <yannick.fortin@xxxxxxxxxxxxxxxxx> wrote in message
news:%23xds4xq9FHA.2816@xxxxxxxxxxxxxxxxxxxxxxx
> Thank you Vladimir, Terry and Kellie for your help. The latest tests have
> some interesting results, in light of what you all suggested.
>
> It seems the data does not disappear after all, it is written at the
> *beginning* of the destination file. Problem is, there was a call to
> SetFilePointer before WriteFile to put the file pointer at the end. Why
> does it write at the beginning then? And why only sometimes?
>
> Let me put back the code with additional comments and code that show
> what happens when it fails. In this example, the destination is 8000
> bytes, the source is 2000 bytes and the buffer is 32768 bytes.
>
> // Open the destination and seek to the end
> destHandle = CreateFile(outFile, GENERIC_WRITE, 0, nil, OPEN_ALWAYS,
> FILE_FLAG_SEQUENTIAL_SCAN, 0);
> SetFilePointer(destHandle, 0, nil, FILE_END);
> // ** SetFilePointer returns 8000, so the file pointer is at the end
> // ** of the destination.
>
> // Open the source file and get its size
> srcHandle = CreateFile(srcFile, GENERIC_READ, FILE_SHARE_READ,
> nil, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, 0);
> srcSize = GetFileSize(srcHandle, nil);
>
> // copy the data
> // ** At this point, I expect the file pointer to be at the end of the
> // ** target file, and the beginning of the source. Since the source is
> // ** smaller than the buffer, the loop will be executed once and the
> // ** target will now be 10000 bytes, with the source file appended.
> while(srcSize > 0)
> {
> ReadFile(srcHandle, buffer, BUFSIZE, numRead, nil);
> WriteFile(destHandle, buffer, numRead, numWritten, nil);
> dec(srcSize, numRead);
> }
>
> chk1 := SetFilePointer(lu_destHandle, 0, nil, FILE_CURRENT);
> chk2 := SetFilePointer(lu_destHandle, 0, nil, FILE_END);
>
> // ** The two check values were expected to be the same, 10000. However,
> // ** chk1 is 2000 and chk2 is 8000. This means that the data *was*
> // ** written, but at the beginning of the file. Opening the file
> // ** and looking at its content confirms that the source was written
> // ** at the beginning, not the end.
>
> // close the files
> CloseHandle(srcHandle);
> CloseHandle(destHandle);
>
>
>> Btw, take a look at description of FILE_FLAG_SEQUENTIAL_SCAN in
>> MSND: "Indicates that the file is to be accessed sequentially from
>> beginning to end. The system can use this as a hint to optimize file
>> caching."...
>
> True, but it also says that if the file pointer is moved, "correct
> operation is still guaranteed" even if caching is not optimized. It does
> not seem to be the case.
>
>> Try to open those files with FILE_FLAG_NO_BUFFERING or call
>> FlushFileBuffers
>> before CloseHandle's.
>
> The data is written, but at the beginning instead of the end, so that's
> not an issue.
>
>> I agreed with Scherbina. If we don't flush cache before closing file, the
>> data in cache will be lost probably.
>
> AFAIK, from a user-mode perspective, the data written to the file cache
> *is* written in the file. Or at least it is supposed to. I would be very
> surprised that doing WriteFile/CloseHandle without flushing the buffers
> between the 2 operations would cause data to be lost. That's just not how
> operating system cache works.
>
> In any case, this is not the issue because the data *is* written, but at
> the beginning.
>
>> 1). You need to include the flag FILE_ATTRIBUTE_NORMAL to both
>> files (source & destination).
>
> OK, will do.
>
>> 2). You need to lock the destination file before writing the
>> data, and unLocking it after the writing.
>
> I understand that it is preferrable, but since the file is only opened
> for read operation and explicitly denies other writes, is that really
> necessary? I'll just remove the FILE_SHARE_READ to have exclusive access
> to the file.
>
>> 3). You need to use the API SetFilePointer() during the main
>> do-while-loop to set the EOF correctly.
>
> Why? According to the documentation of WriteFile:
>
> If hFile was not opened with FILE_FLAG_OVERLAPPED and
> lpOverlapped is NULL, the write operation starts at the
> current file position and WriteFile does not return until
> the operation has been completed. The system updates the
> file pointer upon completion.
>
> To me, this says that that SetFilePointer is not necessary after a
> synchronous WriteFile. The file pointer should be moved to right after the
> last byte written. It is necessary however if you want to write
> non-sequentially, which is not the case here.
>
> Anyways in my case, SetFilePointer *is* called before the loop. Since
> the test files that my customer uses are smaller than the buffer, there is
> no need to set the file pointer in the loop. And yet, the problem happens.
>
> I'll try to replace the sequential scan flag by FILE_ATTRIBUTE_NORMAL
> and see what happens. The good news is, this last discovery (data is not
> lost but written at the beginning) means that another problem that we
> have, data being written at the beginning rather than the end of a file,
> is in fact the same problem.
>
>
> Yannick
.
- References:
- Data loss appending data to file
- From: Yannick Fortin
- Re: Data loss appending data to file
- From: Scherbina Vladimir
- Re: Data loss appending data to file
- From: "TerryFei"
- Re: Data loss appending data to file
- From: Yannick Fortin
- Data loss appending data to file
- Prev by Date: Re: Data loss appending data to file
- Next by Date: Re: KM > UM
- Previous by thread: Re: Data loss appending data to file
- Next by thread: Re: Data loss appending data to file
- Index(es):
Relevant Pages
|
