Re: Windows XP Shared memory woes

I'm going to plead complete ignorance here, and ask a few very basic
questions:

What is shared memory? To me, shared memory means a memory mapped file, or
something similar. Based on the context you're using it in, it doesn't sound
like you're talking about a Memory Mapped file.

What language are developing in? Are you in Native C++ / MFC, Managed /
Mixed C++, or C#?

Some basic thoughts, just based on your call-stack:
- Allocations in whatever technology you're using look to require exclusive
ownership of a Mutex of some sort. If you've got multiple processes /
threads reading & writing to this area, you're going to need to do some
smarter memory management. Pre-allocate large pools of objects, and such.

A classic (and silly) trick: Turn your page file size waaay down. Try "0".
That way nothing is paged out.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP
http://www.coversant.com/blogs/cmullins

"GregP" <GregP@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:F99ACAD2-61FE-46DB-B34C-66B9A9D585DC@xxxxxxxxxxxxxxxx
Hi There,

I have an application that uses quite a bit of memory to maintain a
database. This is a real-time, latency-sensitive app that needs this
memory.
Let's take that as a caveate and not suggest "use less memory". My machine
has 16 gig of memory - I only want about 5 gig of that - a pittance,
really.
;) I'm running on 64-bit XP using dev studio 2005 verstion 8.0.50727.42
(RTM.050727-4200).

The application uses three shared memory regions. Two of them are one gig
in
size. One of them is 3 gig in size. I have enlarged my working set to
about 6
gig to encompass the shared regions, my program code, a few internal
structs
and the like. Upon startup, the app begins to "allocate" buffers from the
3
gig region while processing transactions from a ring buffer (one of the
one
gig regions). The "main database" lies in the other one gig region and is
accessed randomly as transactions fly in. At first the app runs very
smoothly
and quickly. It pages in about 2 gig very quickly and runs for about a
half
hour very well. Once the 3 gig buffer region begins to be consumed (once
it's
about 80% used) the app slows to a crawl.

Using process explorer, and several of my own breadcrumbs, I find that I
am
"stuck" accessing a pointer most of the time (a pointer into the 3 gig
region). The 3 gig region, by the way, is mapped into the "highest"
address
space - I hope that makes sense. Process explorer shows this as my stack:

ntoskrnl.exe!ExAcquireSharedWaitForExclusive+0x115
ntoskrnl.exe!ExAcquireSharedWaitForExclusive+0x545
ntoskrnl.exe!MmProbeAndLockPages+0x5ce
ntoskrnl.exe!KeSynchronizeExecution+0x41c
ntoskrnl.exe!ExfReleasePushLock+0x1f
ntoskrnl.exe!ObReferenceObjectByPointer+0x16d
ntoskrnl.exe!LsaFreeReturnBuffer+0x2913
ntoskrnl.exe!IoFreeController+0x547
ntoskrnl.exe!PsReturnProcessNonPagedPoolQuota+0x82f
ntoskrnl.exe!NtAllocateVirtualMemory+0x4b7
timeseries.EXE+0x5c200

This looks to me to be memory management stuff. I'm mildly disturbed by
the
"allocateVirtualMemory" entry point because I'm NOT allocating virtual
memory
- I'm accessing a pointer of a mapped shared region.

Process explorer suffers mightily while trying to examine my process. It
frequently sits there for quite a long time before it actual does
anything.

I have tried using Virtuallock to hold the pages in my working set to
improve performance, but alas, this does not help. The virtuallock call
takes
MINUTES to complete and the performance is still swirling deeply in the
toilet.

I have a small service that creates the shared region and "holds it" in
memory as I stop and start this program and one other that map these
regions
to examine the current data and store new data.

Questions:

Since this is shared memory, SHOULD I be locking it in my working set? If
so, why does the performance still stink?

What can I do to improve the memory access of this very large shared
region?
Increase my working set more?

Process explorer only seems to suffer when my app is also suffering..
could
my system simply need some tuning? If so, what do I tweak to make this
machine better use it's honking 16 gig of RAM?

--- More info ---

It did NOT seem to help to have the original creator of the region (the
little service I mentioned before that "holds open" the shared memory)
lock
the pages down. On the contrary, this resulted in my processes totally
locking up. I couldn't kill it with taskmanager or process explorer. I had
to
reboot.

I NEED this to work...it SHOULD work. It just doesn't and I don't know
why.
Please help me to understand. I have programmed on VMS systems and this
shared memory stuff - the non-paged pool references etc. look very
VMS-like.
No surprise really since it was probably designed by the illustrius David
Cutler, the father of VMS. If there is some way to reserve this memory and
keep it "as my own" - that is, keep XP from paging it or charging my
process
for it or whatever, please tell me.

Thanks for your time.


.

Relevant Pages

  • Re: Windows XP Shared memory woes
    ... shared memory is a memory-mapped file. ... One of them is 3 gig in size. ... Using process explorer, and several of my own breadcrumbs, I find that I ...
    (microsoft.public.dotnet.framework.performance)
  • [HPADM] RE: SHMMAX modification
    ... The problem is that you are using a 32bit version of Oracle ... will allow access to as much as 1750 megs of shared memory. ... the kernel maintains a map relating the virtual ...
    (HP-UX-Admin)
  • Windows XP Shared memory woes
    ... latency-sensitive app that needs this memory. ... One of them is 3 gig in size. ... Using process explorer, and several of my own breadcrumbs, I find that I am ...
    (microsoft.public.dotnet.framework.performance)
  • Re: OpenOffice does not start when Firefox open
    ... openings and closings of OO so I don't think it's a memory leak) ... you're left with this small sliver dividing up the remaining space. ... able to allocate a total amount of shared memory that is more than the ...
    (comp.os.os2.apps)
  • Re: [RFC v11][PATCH 05/13] Dump memory address space
    ... in one or more chunk: each chunk begins with a header that specifies ... should not change in the interim anyways, the locks protects us from, e.g. ... Even scanning over a large amount of memory, you aren't going to see a performance difference for accessing pgarr from an argument vs off of field in ctx which is going to be cache-hot. ... It is strictly necessary when it comes to handling shared memory. ...
    (Linux-Kernel)