Re: PCI bus-master and large contiguous memory buffers

On 10 Feb 2006 01:20:12 -0800, v_mirgorodsky@xxxxxxxxx wrote:

Hello Brian,

Brian wrote:
You should read up on the MDL, (Memory Descriptor List) Windows already does
what you are proposing. You are really attempting to support scatter
gather, your design is correct, but you're really re-inventing a wheel.

Thanks, Brian, for pointing this out :) Although being familiar with
x86 CPU memory management structures I never tried to dig into Windows
implementation of those details. I always treat MDL as black-box and I
work with it only by means of provided kernel API. Sure, I can dig into
its structure and make my research all the way down to CPU structures,
but that code will not be compatible with upcoming Windows versions and
64 bits CPU's. Is there any elegant way to figure out physical
structure of the memory buffer?

What about buffer consistency in the long run? Is it possible to
guaranty that its physical structure remain unchanged for any arbitrary
long period of time while it is locked in memory?


You misunderstand. The MDL is one of the key data structures for
supporting DMA operations and direct memory transfers from device to
user buffers, but you do not poke around with the internal pieces of
the MDL, you use the defined APIs for manipulating MDL objects. MDLs
are just one piece of the DMA architecture. The next piece is the
DMA_ADAPTER object obtained via a call to IoGetDmaAdapter. From the
DMA_ADAPTER you obtain a set of bus specific DMA methods that you can
call. In particular you want to read up on GetScatterGatherList and
PutScatterGatherList, and once you figure out how these work, you also
might want to understand BuildScatterGatherList and friends.

The scattergather list methods provide a vastly simplified mechanism
for doing scatter gather dma using data buffers described by MDLs.
Almost all of the OS specific housekeeping operations are performed
for you if you use Get/PutScatterGather - all you have to do is setup
your device's DMA registers and hit the 'go' bit.

I am going to use PCI MEMORY_WRITE_INVALIDATE command.

You are?

Documentation
states that this command should automatically keep cache coherencies.

Cache coherency from the perspective of the host system is guaranteed
by the platform.

Do I need to call KeFlushIoBuffers() and/or FlushAdapterBuffers() at
the end of the transfer operation? Since the system does not know
anything about my device internal DMA call to FlushAdapterBuffers()
seems to be redundant. What about KeFlushIoBuffers()? System does not
know what command I am using to transfer data, so it can not decide if
it is necessary to flush the CPU caches.


If you use the scattergather methods you do not have to flush
anything. PutScatterGather does whatever is required to guarantee that
the data is available.

What is the worst case latency in calling my DPC for ISR?

Indeterminate.

Note that the version 2 DMA_OPERATIONS support mechanisms to build
your own scattergather lists objects (BuildScatterGatherList
CalculateScatterGatherList) and that you may be able to architect your
device/driver interface such that you can tolerate a fair amount of
latency by having queues of DMA operations ready to go, with your
device moving asynchronously onto the next operation while your driver
is waiting for its DPC routine to run.

I read that
CD-ROM driver may keep the system for seconds at elevated IRQL levels,
effectively preventing all other DPC's from executing.

Not sure about seconds, but there continue to be badly behaved drivers
out there. On an MP system one device behaving badly should not
prevent other devices from being serviced on other CPUs.

Sure enough, I
can discourage using CD-ROMs while my device bursts its data, but may
be there are other bad citizens like that in driver community?

Yes this is out of your control unless you can predetermine the
platform configuration. You have to be able to tolerate a modest
amount of latency. Probably your users should not be burning CDs while
collecting data from your device.

With best regards,
Vladimir S. Mirgorodsky

"v_mirgorodsky@xxxxxxxxx" wrote:

Hello, ALL!

I am developing PCI bus-master device, which generates a lot of data
during its life-time. On average device generates about 85-100MB of
data per second. Since the amount of data is huge, I need relatively
large 32MB contiguous memory buffer, which is a big problem in modern
OS. After getting these data from device driver needs to perform some
small processing, like adding headers, cross-references, etc. Driver
restricts its use by more than single application at a time, at least
on data interface.

I can not follow standard DMA-like procedure when talking to device,
since it requires extremely low latency in data handling. I can not
have several small contiguous memory buffers since the data comes out
in a big bursts and device may already overwrite the data in such
buffer while OS not yet managed to invoke my DPC for ISR.

I would like to adopt the following solution for this task. First of
all user application allocates any size of buffer it desires and
notifies the driver about its starting address. Driver rounds the
buffer start address to the beginning of the physical page size (may be
even run-time configuration parameter) and maps the entire buffer into
kernel memory space and protects it from being paged out by memory
paging system. Then driver allocates a small amount of contiguous
non-paged memory, like 64kB or so and fills it with 4-bytes long
addresses of physical pages of memory, from which the user buffer
consists. That is all. As soon device needs next portion of physical
memory to store the data it just reads its address from the catalog. At
the end of the buffer it wraps to its beginning. As soon as new portion
of data ready, device generates the interrupt to notify driver about
this. Buffer may be allocated once and may be used as long as it locked
in kernel memory space. In the end of session driver deconfigures
device, unmaps user buffer, etc.

Are there any serious pitfalls with this approach I am not aware of? Is
there documented way to get the list of pages user buffer consists of?
Will the OS try to exchange one page to another while it locked in
memory?

With best regards,
Vladimir S. Mirgorodsky




=====================
Mark Roddy DDK MVP
Windows Vista/2003/XP/2000 Consulting
Device and Filesystem Drivers
Hollis Technology Solutions 603-321-1032
www.hollistech.com
.

Relevant Pages

  • Re: PCI bus-master and large contiguous memory buffers
    ... As soon as device reaches the end of the buffer ... Sure, I am developing both PCI adapter and device driver, so, it is ... not afford reinitializing DMA on my device after every transfer. ... x86 CPU memory management structures I never tried to dig into Windows ...
    (microsoft.public.development.device.drivers)
  • Re: PCI bus-master and large contiguous memory buffers
    ... I built my scatter gather list in SRAM that was on my device, ... could have done it in system memory had I needed to. ... interrupt when a buffer was filled, the application would save the buffer to ... beginning of the recording I made a device IO control call to my driver. ...
    (microsoft.public.development.device.drivers)
  • Re: PCI bus-master and large contiguous memory buffers
    ... x86 CPU memory management structures I never tried to dig into Windows ... What about buffer consistency in the long run? ... CD-ROM driver may keep the system for seconds at elevated IRQL levels, ... addresses of physical pages of memory, from which the user buffer ...
    (microsoft.public.development.device.drivers)
  • PCI bus-master and large contiguous memory buffers
    ... large 32MB contiguous memory buffer, which is a big problem in modern ... After getting these data from device driver needs to perform some ... buffer while OS not yet managed to invoke my DPC for ISR. ... addresses of physical pages of memory, from which the user buffer ...
    (microsoft.public.development.device.drivers)
  • [PATCH] Integrating SEP Driver with RAR Driver
    ... RAR stands for Restricted Access Region; this is memory ... This is upstream revision 4 of the SEP driver. ... +This functions maps and allocates the shared area on the external ...
    (Linux-Kernel)
Loading