Re: asynchronously writing to a file, a cheap way?
- From: nickdu <nicknospamdu@xxxxxxxxxxxxxxxx>
- Date: Wed, 16 Sep 2009 12:28:01 -0700
But I think I want lazy writes, as long as I don't bring the OS to a hault.
The lazy writes gives me the async behavior I'm looking for I assumed.
--
Thanks,
Nick
nicknospamdu@xxxxxxxxxxxxxxxx
remove "nospam" change community. to msn.com
"Alexander Grigoriev" wrote:
Use FILE_FLAG_NO_BUFFERING to avoid lazy writes..
"nickdu" <nicknospamdu@xxxxxxxxxxxxxxxx> wrote in message
news:ADE4AED2-27F4-45FC-B818-19E8BA910A1E@xxxxxxxxxxxxxxxx
The hybrid approach sounds intriguing, but no straightforward
implementation
comes to mind. In fact I was thinking of something similar in which I
stick
with the file backed memory mapped approach but have it be circular.
I did code up something which writes 16K 64K buffers at 1 MB offsets of a
sparse file, which (if I did my math correctly) equates to 1 GB of data
written. By the way, I set the sparse file size to 1 TB. The application
on
my single proc laptop runs in just a couple seconds with the overhead of
all
the writes being virtually zero microseconds. Of course after the app
finishes I see the available memory very low and it slowly increasing as I
guess the OS writes the pages to disk. I was surprised that the file
consumed 3GB of disk space. From the docs I've read the granularity of
sparse files is 64K, which is why I chose 64K as the buffer size. While I
did expect the "sparseness" to consume some additional overhead, I didn't
expect the 1GB I wrote to consume 3GB.
If I run this application again right afterwards it pretty much locks up
the
computer (by the way I'm running XP Pro SP3) for several minutes: the
mouse
pointer is frozen, no updates to task manager values, ctrl-alt-del doesn't
work, etc. I'm a bit surprised I can get the OS into this state. Not
sure
if the kernel is still responsive at this time and it's just explorer
that's
giving the appearance of being locked up, though I doubt it. I think the
OS
itself is in bad shape at this point. Eventually things come back to
life.
Why would the OS let itself get to such a state?
Is there a way for me to set some settings such that the OS doesn't give
away all it's memory and get into this unresponsive state?
--
Thanks,
Nick
nicknospamdu@xxxxxxxxxxxxxxxx
remove "nospam" change community. to msn.com
"Pavel Lebedinsky [MSFT]" wrote:
Having the producer write to disk and the consumer read from disk
seems like a lot more overhead (in the case where the consumer can keep
up
with the producer) than using memory mapped files.
Not necessarily - if you use cached IO, consumer that is able to keep
up will most likely read cached data from memory. The situation would
actually be similar to the memory mapping approach, except that
mapping/unmapping of views and flushing of dirty pages would be handled
by the cache manager.
In your writer thread case it seems that you also stand a greater
chance
of
losing a message if the producer abruptly crasher after calling my API
which
allows them to write these messages.
Data loss due to a process crash can only happen if data is cached
somewhere in the process private memory. Once it is written to a file
(using either WriteFile or mapped IO) it can only be lost if the system
crashes.
I have an additional question regarding mapped views. Is there a way
to
"clean" a view such that the OS doesn't write the changes if it hasn't
done
so already? In the case where the consumer can keep up with the
producer
it
would be nice for the consumer to be able to "clean" the view after it
has
processed it so that no write is needed. It would then unmap the view
and
the memory could be returned to the system.
Generally, no. MEM_RESET can be used for pagefile-backed sections
but it doesn't work for regualr files. FSCTL_SET_ZERO_DATA works
on files but (from a quick look at the code) only if there are no user
mapped views or section objects for the file.
As far as I can tell, FSCTL_SET_ZERO_DATA should work if there is
dirty data in the system cache, so it might be possible to use it with
WriteFile to achieve something similar to what you want. However, the
consumer would probably have to run with very little delay in order to
prevent the cache manager from flushing data to disk.
It might be easier to use some sort of hybrid approach. For example,
have a relatively small shared memory area that is always mapped,
and use it as a circular buffer. If this buffer gets full, switch to a
separate
sparse file (or a set of regular files).
--
Pavel Lebedinsky/Windows Kernel Test
This posting is provided "AS IS" with no warranties, and confers no
rights.
- Follow-Ups:
- Re: asynchronously writing to a file, a cheap way?
- From: Alexander Grigoriev
- Re: asynchronously writing to a file, a cheap way?
- References:
- Re: asynchronously writing to a file, a cheap way?
- From: nickdu
- Re: asynchronously writing to a file, a cheap way?
- From: Alexander Grigoriev
- Re: asynchronously writing to a file, a cheap way?
- Prev by Date: Develop a virtual keyboard device driver
- Next by Date: Re: asynchronously writing to a file, a cheap way?
- Previous by thread: Re: asynchronously writing to a file, a cheap way?
- Next by thread: Re: asynchronously writing to a file, a cheap way?
- Index(es):
Relevant Pages
|
