Re: File map performance

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

George napsal(a):
I have tried that map a large file to access is faster than map several small files. I am not sure whether you agree that it is a general pattern -- my test experience on this is limited to just a couple of test cases.

When accessing mapped files some things are the same as with normal file operations: e.g. on most devices you need to consider seeks as well, unless the data are already cached, therefore accessing one contiguous block is likely to be faster than random access. If it is one file or several files should not matter much per se, but with one file you are very likely to get the contiguous data. (It also depends on what time are you interested in: real world time, CPU time ...)

Your idea of mapping a part of the file is great. I am wondering whether you could recommend me some tutorials about the best practices of file mapping, since I think if I develop straight forward way of mapping portion of a file which is needed currently, close un-needed mapping, such kinds of straight forward way is not of the best performance.

I am not aware of any reference, but by coincidence I just spent most of today experimenting with file mappings, and I can therefore share a few measurements and observations (timings taken on WinXP SP 2, Intel Quad Core 2.4 GHz, 3 GB RAM).

- MapViewOfFile takes about 3 microseconds
- CreateFileMapping takes about 17 microseconds

When you map a view, unmap the view, and then map the view of the same area again, the data are still in memory unless you are running out of RAM. You will get a page fault when accessing the data again, however only a minor one (no disk access, only mapping the memory into the process address space), not a major one (which would be full page swap). (Cf. http://en.wikipedia.org/wiki/Page_fault)

If the straight forward way of mapping and unmapping is good enough for you, that depends on your target requirements for data processing speed.
My experience is OS is very good at handling this, it keeps the data in the physical memory as long as you could reasonably expect, and for my purposes such straight forward way is fast enough.

(One slightly related note: what I like about file mapping a lot is it allows to use more physical memory than allowed by available virtual address space, which can come handy on 32b OS).

Regards
Ondrej

----------------
Ondrej Spanel
www.bistudio.com



regards,
George

"Igor Tandetnik" wrote:

George <George@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
I have several physical file and I want to use file map
(MapViewOfFileEx) to map the file into memory to improve performance.
Each file is about several hundred M bytes. All the memory mapped
files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is
very fast;
2. the performance to access the files which are opened later is
slower and slower (the performance to access the 10th file is very
bad).
You are running out of physical memory, so the system has to swap the data to and from the hard drive.

Do you need to work with all the data all the time? Do you know that you can map just parts of the file into memory?
--
With best wishes,
Igor Tandetnik

With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead. -- RFC 1925



.

Relevant Pages