Re: physical storage on disk
- From: Joseph M. Newcomer <newcomer@xxxxxxxxxxxx>
- Date: Wed, 04 Apr 2007 14:33:10 -0400
No, you allocate VIRTUAL memory by VirtualAlloc, and additionally mark it as nonpageable,
which is almost always a bad decision in almost all cases. While it gives you the
illusion that you are improving performance (yours) you are screwing the performance of
nearly everything else, resulting in overall system degradation.
AWE is irrelevant unless your machine has more than 4GB of installed memory. A 128MB
machine would not be using this. And while it is fine to produce a program that can use
massive amounts of extended memory, this becomes part of the specification of the program.
"Must have at least 4GB of physical memory" or "must have > 8GB of physical memory". But
don't expect that you have produced a program that will run on anything smaller. So if
you have a truly massive physical memory, a rare situation except on file servers, you are
in one of the very few exceptions to that "almost always a bad decision in almost all
cases". But don't expect it is going to run efficiently, or even at all, on anything
smaller. Actually, since you have a GB of image, you can safely assume that it will not
work acceptably on anything smaller than a 2GB machine, and probably not very well on a
machine that small. 4-8GB might be a reasonable choice of machine size.
So in a restricted situation you can just ignore the issues of paging-aware design because
you can create a situation in which paging-aware design doesn't matter. Don't expect such
a design to generalize to situations in which paging is a central performance issue.
Key here is that you MUST, when dealing with situations like this, REWRITE your algorithm
to be paging-aware, and write one that minimizes paging traffic. This has been an issue
since the 1960s when we first started using virtual memory, and is a fundamental issue of
large data manipulation. I wrote my first performance analyzer in 1969 to measure
inter-page function call patterns so we could link our program to form clusters of related
pages of code. You can run away and hide from paging issues in the short term (such as by
locking down physical memory) but to scale up you have to work with the notion that paging
is real, memory is limited, and your code has to change to accomodate this situation.
Consider what you would have to do to build an algorithm that worked on a 4GB image on
Win32. The answer, of course, is to run Win64. That's fine, but don't expect your code
will run on Win32.
I spent most of my life working on VM systems that had no way to lock pages down, so I
never consider this as a possible solution. Given that a page fault is six orders of
magnitude slower than an L2 cache miss, a serious amount of effort needs to be expended in
writing paging-aware code when performance is an issue. Bitmaps and DSP algorithms are
key candidates here. Note that if you write your own image-segment-swapping code, you are
likely to be substantially less efficient than the built-in paging algorithms. Think, at
the outset, you will be lucky to be seven orders of magnitude slower than the built-in
paging algorithms. Write paging-aware code, use the normal paging mechanism, and you may
be only a factor of 10 slower than the locked-in-memory mechanism when you start. Then
you can start refining the agorithms.
Note that if you work with a file instead of a pure-VM-based solution, then YOU are going
to be responsible for writing the paging algorithms. Given the nature of your question,
you probably have little or no experience doing this. I've done it many times, and it is
very challenging to do in the first place, and you STILL have the issues of writing
paging-aware code, even if you are doing the paging yourself. So the first thing to do is
write paging-aware code that works well on small machines without using features like
VirtualAlloc's ability to lock pages into memory. Once you've factored the code so it is
paging-aware, it is substantially easier to modify for your own paging strategy.
joe
On Wed, 4 Apr 2007 07:11:15 +0900, "William" <port@xxxxxxxxxxxxxxxxxx> wrote:
I know I can allocate physical memory(RAM) by AWE. That isJoseph M. Newcomer [MVP]
VirtualAlloc(,,MEM_RESERVE | MEM_PHYSICAL, PAGE_READWRITE,). It allows my
application to get RAM that is never swapped by the operating system to or
from disk.
The problem is that I need huge memory and I hope my app can run on low
memory configuration PC. I just tried to create 4000*30000, 256color bitmap
by MSPaint.exe on a 256MB RAM Windows2000 while monitoring the system
memory(physical and virtual memory) status by using GlobalMemoryStatus().
When I change the bitmap size from 4000*4000 to 4000*30000, physical memory
goes down to almost zero at first, then takes a long long time(60s) to
swaping it to pagefile and physical memory goes up to about 10MB.
As I can tolerate a low speed to access disk(pagefile) but can not tolerate
a long waiting time, so I hope I can allocate directly virtual memory(disk)
as my DIB bitmap buffer by some functions. I don't know how.
William
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
.
- References:
- physical storage on disk
- From: William
- Re: physical storage on disk
- From: Joseph M . Newcomer
- Re: physical storage on disk
- From: William
- physical storage on disk
- Prev by Date: Re: MFC and threads
- Next by Date: Problem with CTreeCtrl::SetImageList()
- Previous by thread: Re: physical storage on disk
- Next by thread: Re: physical storage on disk
- Index(es):
Relevant Pages
|
