Re: ReadFile and Buffersize

Jos,

However, it is difficult to determine the correct buffersize
to use to get the optimal result.

I think you should read data in chunks that do not exceed the amount of
memory that you can lock. Imagine what happens if the buffer size is
too large. For example, imagine that the maximal amount of memory the
system can commit for this particular operation is, say,
1M, and you read in 2M chunks. The system will read 1M of data from the
disk (i.e from your file) into memory..... and then, in order to free
memory, necessary for reading a second megabyte from the file, write it
back to the disk again (this time to the pagefile)!!!!! In other
words, if the buffer is too large, you may end up moving data from your
target file to the page file. It is hardly going to improve the
performance, don't you think????

Anton Bassov

Jos Scherders wrote:
Hi,

I haven't tried that yet but I doubt if I will gain by doing that because
it's that what the cache manager is also doing (more or less) ? I am
investigating if reading it using ReadFileEx or a IOCompletionPort will
improve things. However, it is difficult to determine the correct buffersize
to use to get the optimal result.

Jos.

"Chris Becke" <chris.becke@xxxxxxxxx> wrote in message
news:%23Vt5QwA%23GHA.1220@xxxxxxxxxxxxxxxxxxxxxxx
As an alternative, have you tried memory mapping to get (parts) of the
large
file in and process it that way?


"Jos Scherders" <thrower@xxxxxxx> wrote in message
news:uAh9aR79GHA.360@xxxxxxxxxxxxxxxxxxxxxxx
Hi all,

I am investigating a performance issue reading a large file (sequential
reading using ReadFile).

Scenario 1:

Use loop to read the file using 2Mb buffers.

Scenario 2:

Open file, seek 0, read 2Mb, close
Open file, seek 2Mb, read 2Mb, close

What I noticed is that scenario 2 is significantly faster. Can someone
explain this behavior ?

Also, is perform the best tool to get to the bottom of this problem ?

Thanks all,
Jos Scherders





.

Relevant Pages

  • Re: Discovering variable types...
    ... >memory it points to is on the heap. ... sequentially reading data, if one is randomly reading records, then a ... >project is what's prompting me to improve disk access. ... from a memory buffer I can do it in about a second. ...
    (comp.lang.pascal.delphi.misc)
  • Re: Decoding strategy
    ... may be awkward to replace memory mapping with FileStream and measure ... the OS simply cannot correct predict what to buffer for you. ... just reading normally. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: fast file processing optimization question
    ... Look into the option to map the file into memory (you can do it either for ... > As you're reading huge chunks of data, you should at least use setbuf to ... > 5 MB buffer if you don't need it? ... > the raw Windows API CreateFile and friends. ...
    (microsoft.public.vc.mfc)
  • Re: Continuously concatenating binary data
    ... If your user have small amount of memory or record large ... It would be no differents than stream approach. ... In fact stream give you better control on amount of memory ... >> beginning how large this buffer would need to be. ...
    (comp.lang.cpp)
  • Dynamically-sized memory buffer?
    ... when the amount of data is not known in advance? ... DynamicMemBuffer) that uses a linked list of fixed-size bytebuffers ... DynamicMemBuffer, a new internal buffer is created, added to the ... stream held in memory. ...
    (comp.lang.java.programmer)
Quantcast