Windows XP Shared memory woes

Hi There,

I have an application that uses quite a bit of memory to maintain a
database. This is a real-time, latency-sensitive app that needs this memory.
Let's take that as a caveate and not suggest "use less memory". My machine
has 16 gig of memory - I only want about 5 gig of that - a pittance, really.
;) I'm running on 64-bit XP using dev studio 2005 verstion 8.0.50727.42
(RTM.050727-4200).

The application uses three shared memory regions. Two of them are one gig in
size. One of them is 3 gig in size. I have enlarged my working set to about 6
gig to encompass the shared regions, my program code, a few internal structs
and the like. Upon startup, the app begins to "allocate" buffers from the 3
gig region while processing transactions from a ring buffer (one of the one
gig regions). The "main database" lies in the other one gig region and is
accessed randomly as transactions fly in. At first the app runs very smoothly
and quickly. It pages in about 2 gig very quickly and runs for about a half
hour very well. Once the 3 gig buffer region begins to be consumed (once it's
about 80% used) the app slows to a crawl.

Using process explorer, and several of my own breadcrumbs, I find that I am
"stuck" accessing a pointer most of the time (a pointer into the 3 gig
region). The 3 gig region, by the way, is mapped into the "highest" address
space - I hope that makes sense. Process explorer shows this as my stack:

ntoskrnl.exe!ExAcquireSharedWaitForExclusive+0x115
ntoskrnl.exe!ExAcquireSharedWaitForExclusive+0x545
ntoskrnl.exe!MmProbeAndLockPages+0x5ce
ntoskrnl.exe!KeSynchronizeExecution+0x41c
ntoskrnl.exe!ExfReleasePushLock+0x1f
ntoskrnl.exe!ObReferenceObjectByPointer+0x16d
ntoskrnl.exe!LsaFreeReturnBuffer+0x2913
ntoskrnl.exe!IoFreeController+0x547
ntoskrnl.exe!PsReturnProcessNonPagedPoolQuota+0x82f
ntoskrnl.exe!NtAllocateVirtualMemory+0x4b7
timeseries.EXE+0x5c200

This looks to me to be memory management stuff. I'm mildly disturbed by the
"allocateVirtualMemory" entry point because I'm NOT allocating virtual memory
- I'm accessing a pointer of a mapped shared region.

Process explorer suffers mightily while trying to examine my process. It
frequently sits there for quite a long time before it actual does anything.

I have tried using Virtuallock to hold the pages in my working set to
improve performance, but alas, this does not help. The virtuallock call takes
MINUTES to complete and the performance is still swirling deeply in the
toilet.

I have a small service that creates the shared region and "holds it" in
memory as I stop and start this program and one other that map these regions
to examine the current data and store new data.

Questions:

Since this is shared memory, SHOULD I be locking it in my working set? If
so, why does the performance still stink?

What can I do to improve the memory access of this very large shared region?
Increase my working set more?

Process explorer only seems to suffer when my app is also suffering.. could
my system simply need some tuning? If so, what do I tweak to make this
machine better use it's honking 16 gig of RAM?

--- More info ---

It did NOT seem to help to have the original creator of the region (the
little service I mentioned before that "holds open" the shared memory) lock
the pages down. On the contrary, this resulted in my processes totally
locking up. I couldn't kill it with taskmanager or process explorer. I had to
reboot.

I NEED this to work...it SHOULD work. It just doesn't and I don't know why.
Please help me to understand. I have programmed on VMS systems and this
shared memory stuff - the non-paged pool references etc. look very VMS-like.
No surprise really since it was probably designed by the illustrius David
Cutler, the father of VMS. If there is some way to reserve this memory and
keep it "as my own" - that is, keep XP from paging it or charging my process
for it or whatever, please tell me.

Thanks for your time.
.

Relevant Pages

  • Re: Windows XP Shared memory woes
    ... shared memory is a memory-mapped file. ... One of them is 3 gig in size. ... Using process explorer, and several of my own breadcrumbs, I find that I ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Vista 64 System needs Extended warm-up
    ... Note that with the total 8 Gig of memory I had the "normal" problem of not being able to install Windows until I had stripped the memory down to only 4 Gigs in the first 2 slots. ... Installation went OK with only 2 sticks of memory, but I needed to do a repair installation a couple of times to remedy a "forever boot" that included a full completion of POST to FF. ...
    (microsoft.public.windows.vista.general)
  • Vista 64 System needs Extended warm-up
    ... Note that with the total 8 Gig of memory I had the "normal" problem of not being able to install Windows until I had stripped the memory down to only 4 Gigs in the first 2 slots. ... Installation went OK with only 2 sticks of memory, but I needed to do a repair installation a couple of times to remedy a "forever boot" that included a full completion of POST to FF. ...
    (microsoft.public.windows.vista.general)
  • Re: Gateway FX6800-01e----Is this computer ok?
    ... Carole Clark wrote: ... The Graphic card you will ... Two of the people told me that yesterday, and today I said I should by more memory and 2 of the techs said that was plenty, like 6G of DDR2. ... A gig is a gig is a gig, ...
    (comp.sys.ibm.pc.games.rpg)
  • Re: Windows XP Shared memory woes
    ... What is shared memory? ... One of them is 3 gig in size. ... Using process explorer, and several of my own breadcrumbs, I find that I ...
    (microsoft.public.dotnet.framework.performance)
Loading