Hello,

We are developing a spin-off of one of our PCI-X devices which runs on custom hardware under VxWorks. Our new device will be a PCIe device that will plug into a Dell server (1950 and the likes) which runs a 2.6 Linux Kernel flavor (Possibly, Red hat enterprise 5.0). The main job of this PCIe device is to push externally gathered data at a high-rate into memory for an application to consume. The rates reach over 2 Gbps, so avoiding memory copies is desired. Our PCIe device is a bus master which DMAs data into a 256 MB circular buffer. When this device receives external data, it pushes this data onto the circular buffer and then updates a write pointer to indicate data is ready to read.

I have read Linux Device Drivers, third edition, but I’m stuck on one sticky point. How can I map a huge user space buffer (256Mb, or higher) into kernel space such that a PCIe device can freely write to it with out the need for constant mitigation by a device driver (marking memory as Dirty, etc). Do I even need to map this user space buffer into kernel space, or can I just pin this buffer in memory, translate the virtual address to a physical addresses and tell our PCIe device this physical address it can write to.

I was thinking of developing a char driver, which initializes our PCIe device and implements mmap(). We want to write a user space program which controls and configures this device. Part of the configuration is to write into its registers the base address of the 256MB buffer and the address of the write pointer. We would like to allocate this 256MB in user space, lock it into memory such that is can not get swapped, translate these addresses into physical addresses that our PCIe device can write freely to them. We want to lock this buffer into physical memory for the lifetime of the driver/application; its unlikely the application/driver will ever get unloaded, the server will have at least 4GB of RAM. We also want to use cache, but are unsure if the hardware supports cache coherency or if there is a way to tell the kernel to uphold cache coherecny for this buffer.

Our kernel space driver will basically map our PCIe device registers into user space, where a user space driver/application then configures this device to transfer all external data received into a user space buffer. Our user space application will then consume and work on the data in the buffer.

Any help is appreciated, I’m quite new to Linux kernel hacking.

Thank you!

-Will

I hate to tell you this, but what you're asking is probably way beyond what 99% of the people on this web site know, including myself! Try http://www.kernelnewbies.org instead as that site is specifically about kernel development, hacking, etc.

Mike

I tracked down a linux expert, here is the answer:

Thanks for posting back with the followup wkorbe, and Welcome to LQ!

I'll ditto J.W., thanks for the feedback! Now we all get to learn something!

Mike