Cache coherency issues using AllocateCommonBuffer(..)
- From: smann <ssm11b@xxxxxxxxxxx>
- Date: Thu, 23 Mar 2006 10:36:29 -0800
Hi,
I am seeing a cache coherency issue with memory allocated through
AllocateCommonBuffer(..). This issue is occuring with a PCI Experss driver.
The application I am running is in a dual host enviornment, where each host
(PC) can directly wirte to each others main memory. This achieved through a
special feature in the PCIe switch. Through various debugging techniques I
have verifed that when (local) host correctly writes through the PCIe switch
to the physical memory of the other (remote) host. Once the data is written,
the remote host is signaled through the PCIe switch to read new data in its
buffer. I have also verifed that the data read by the remote application
from its shared memory buffer, is stale, i.e. cached data.
Through other debugging mechanism I can invalidate the cache on the remote
host. When I do this the behavior behavior of the remote host is corrected
and it read the contents of the buffer not its cache.
The driver is using AllocateCommonBuffer from the DMA_OPERATIONS structrue.
I tried other depricated MmAllocateContiguousMemorySpecifyCache() function
and the cache coherency issue is resolved.
So I am baffeled, I almost certian I am allocating the buffer correctly w/
AllocateCommonBuffer(...). Here is a snapshot of what I am doing. In the
drivet I allocate with the following call:
// Attempt to allocate buffer
pKernelVa =
pdx->pDmaAdapter->DmaOperations->AllocateCommonBuffer(
pdx->pDmaAdapter,
BufferSize,
&BufferLogicalAddress,
bCacheEnabled // Enable Caching for buffer?
);
Here the bCacheEnabled is set to FALSE. Then allocate the MDL with the
following call:
pBufferObj->pMdl =
IoAllocateMdl(
pKernelVa,
BufferSize,
FALSE, // Is this a secondary buffer?
FALSE, // Charge quota?
NULL // No IRP associated with MDL
);
MmBuildMdlForNonPagedPool(
pBufferObj->pMdl
);
Next I am getting the User Virtual Address for the buffer through the
followng call:
pUserVa =
MmMapLockedPagesSpecifyCache(
pBufferObj->pMdl ,
UserMode,
CacheMode, // CacheMode == MmNonCached
NULL,
FALSE,
NormalPagePriority
);
I return the virtual address and the physical addres to the user mode
application. The Physical address of the memory remote host is passed to
local host and is used for wirting to the remote memory. After the remote
memory is written, the remote host uses the virtual address to read data from
the shared buffer.
And as I mentioned the data it reads is the data in it cache even though the
buffer was allocated with the non-cachable attribute. When I ran multiple
iterations of writes, I could see the data read by the remote host was from
one of the previous writes.
If anyone has any idea on why the buffer being cached, I would appreciate
the response?
--
smann
.
- Follow-Ups:
- Re: Cache coherency issues using AllocateCommonBuffer(..)
- From: Calvin Guan
- Re: Cache coherency issues using AllocateCommonBuffer(..)
- Prev by Date: Re: Using a sledge hammer
- Next by Date: Re: Using a sledge hammer
- Previous by thread: Re: Using a sledge hammer
- Next by thread: Re: Cache coherency issues using AllocateCommonBuffer(..)
- Index(es):
Relevant Pages
|
Loading
