Re: PCI bus-master and large contiguous memory buffers

Hello Brian,

Brian wrote:
You should read up on the MDL, (Memory Descriptor List) Windows already does
what you are proposing. You are really attempting to support scatter
gather, your design is correct, but you're really re-inventing a wheel.

Thanks, Brian, for pointing this out :) Although being familiar with
x86 CPU memory management structures I never tried to dig into Windows
implementation of those details. I always treat MDL as black-box and I
work with it only by means of provided kernel API. Sure, I can dig into
its structure and make my research all the way down to CPU structures,
but that code will not be compatible with upcoming Windows versions and
64 bits CPU's. Is there any elegant way to figure out physical
structure of the memory buffer?

What about buffer consistency in the long run? Is it possible to
guaranty that its physical structure remain unchanged for any arbitrary
long period of time while it is locked in memory?

I am going to use PCI MEMORY_WRITE_INVALIDATE command. Documentation
states that this command should automatically keep cache coherencies.
Do I need to call KeFlushIoBuffers() and/or FlushAdapterBuffers() at
the end of the transfer operation? Since the system does not know
anything about my device internal DMA call to FlushAdapterBuffers()
seems to be redundant. What about KeFlushIoBuffers()? System does not
know what command I am using to transfer data, so it can not decide if
it is necessary to flush the CPU caches.

What is the worst case latency in calling my DPC for ISR? I read that
CD-ROM driver may keep the system for seconds at elevated IRQL levels,
effectively preventing all other DPC's from executing. Sure enough, I
can discourage using CD-ROMs while my device bursts its data, but may
be there are other bad citizens like that in driver community?

With best regards,
Vladimir S. Mirgorodsky

"v_mirgorodsky@xxxxxxxxx" wrote:

Hello, ALL!

I am developing PCI bus-master device, which generates a lot of data
during its life-time. On average device generates about 85-100MB of
data per second. Since the amount of data is huge, I need relatively
large 32MB contiguous memory buffer, which is a big problem in modern
OS. After getting these data from device driver needs to perform some
small processing, like adding headers, cross-references, etc. Driver
restricts its use by more than single application at a time, at least
on data interface.

I can not follow standard DMA-like procedure when talking to device,
since it requires extremely low latency in data handling. I can not
have several small contiguous memory buffers since the data comes out
in a big bursts and device may already overwrite the data in such
buffer while OS not yet managed to invoke my DPC for ISR.

I would like to adopt the following solution for this task. First of
all user application allocates any size of buffer it desires and
notifies the driver about its starting address. Driver rounds the
buffer start address to the beginning of the physical page size (may be
even run-time configuration parameter) and maps the entire buffer into
kernel memory space and protects it from being paged out by memory
paging system. Then driver allocates a small amount of contiguous
non-paged memory, like 64kB or so and fills it with 4-bytes long
addresses of physical pages of memory, from which the user buffer
consists. That is all. As soon device needs next portion of physical
memory to store the data it just reads its address from the catalog. At
the end of the buffer it wraps to its beginning. As soon as new portion
of data ready, device generates the interrupt to notify driver about
this. Buffer may be allocated once and may be used as long as it locked
in kernel memory space. In the end of session driver deconfigures
device, unmaps user buffer, etc.

Are there any serious pitfalls with this approach I am not aware of? Is
there documented way to get the list of pages user buffer consists of?
Will the OS try to exchange one page to another while it locked in
memory?

With best regards,
Vladimir S. Mirgorodsky



.

Relevant Pages

  • Re: PCI bus-master and large contiguous memory buffers
    ... x86 CPU memory management structures I never tried to dig into Windows ... What about buffer consistency in the long run? ... The scattergather list methods provide a vastly simplified mechanism ... CD-ROM driver may keep the system for seconds at elevated IRQL levels, ...
    (microsoft.public.development.device.drivers)
  • Re: Shared Memory: Kernel/User Addresses
    ... I have this implemented this mechanism in my driver. ... DMA through and IOCTL call to the driver. ... pointer to the user buffer, and normally I do dma directly to and from ... When the driver gets the call, it gets an MDL for the user buffer. ...
    (microsoft.public.development.device.drivers)
  • Re: Discovering variable types...
    ... >- but I suppose MS expect us to use wrappers ... memory allocations for your variables from disk as well. ... >They most certainly are of fixed size, changing the size of a String ... >>me to keep buffer size and current postion right in the memory block. ...
    (comp.lang.pascal.delphi.misc)
  • PCI bus-master and large contiguous memory buffers
    ... large 32MB contiguous memory buffer, which is a big problem in modern ... After getting these data from device driver needs to perform some ... buffer while OS not yet managed to invoke my DPC for ISR. ... addresses of physical pages of memory, from which the user buffer ...
    (microsoft.public.development.device.drivers)
  • Re: Discovering variable types...
    ... >memory it points to is on the heap. ... sequentially reading data, if one is randomly reading records, then a ... >project is what's prompting me to improve disk access. ... from a memory buffer I can do it in about a second. ...
    (comp.lang.pascal.delphi.misc)