Help answer threads with 0 replies.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 08-03-2008, 09:41 PM   #1
LQ Newbie
Registered: Aug 2008
Posts: 2

Rep: Reputation: 0
Question PCIe Driver - need high performance


We are developing a spin-off of one of our PCI-X devices which runs on custom hardware under VxWorks. Our new device will be a PCIe device that will plug into a Dell server (1950 and the likes) which runs a 2.6 Linux Kernel flavor (Possibly, Red hat enterprise 5.0). The main job of this PCIe device is to push externally gathered data at a high-rate into memory for an application to consume. The rates reach over 2 Gbps, so avoiding memory copies is desired. Our PCIe device is a bus master which DMAs data into a 256 MB circular buffer. When this device receives external data, it pushes this data onto the circular buffer and then updates a write pointer to indicate data is ready to read.

I have read Linux Device Drivers, third edition, but Im stuck on one sticky point. How can I map a huge user space buffer (256Mb, or higher) into kernel space such that a PCIe device can freely write to it with out the need for constant mitigation by a device driver (marking memory as Dirty, etc). Do I even need to map this user space buffer into kernel space, or can I just pin this buffer in memory, translate the virtual address to a physical addresses and tell our PCIe device this physical address it can write to.

I was thinking of developing a char driver, which initializes our PCIe device and implements mmap(). We want to write a user space program which controls and configures this device. Part of the configuration is to write into its registers the base address of the 256MB buffer and the address of the write pointer. We would like to allocate this 256MB in user space, lock it into memory such that is can not get swapped, translate these addresses into physical addresses that our PCIe device can write freely to them. We want to lock this buffer into physical memory for the lifetime of the driver/application; its unlikely the application/driver will ever get unloaded, the server will have at least 4GB of RAM. We also want to use cache, but are unsure if the hardware supports cache coherency or if there is a way to tell the kernel to uphold cache coherecny for this buffer.

Our kernel space driver will basically map our PCIe device registers into user space, where a user space driver/application then configures this device to transfer all external data received into a user space buffer. Our user space application will then consume and work on the data in the buffer.

Any help is appreciated, Im quite new to Linux kernel hacking.

Thank you!

Old 08-04-2008, 03:02 AM   #2
Registered: Jun 2008
Location: Phoenix, Arizona, USA
Distribution: Slackware
Posts: 326

Rep: Reputation: 49
I hate to tell you this, but what you're asking is probably way beyond what 99% of the people on this web site know, including myself! Try instead as that site is specifically about kernel development, hacking, etc.

Old 08-11-2008, 02:03 PM   #3
LQ Newbie
Registered: Aug 2008
Posts: 2

Original Poster
Rep: Reputation: 0
I tracked down a linux expert, here is the answer:

Normally, you would allocate memory with something like
pci_alloc_consistent() or dma_alloc_coherent() when you want a PCI
device to be able to DMA into it. You are correct that using these (or
even kmalloc()) for any largish size will likely fail (due to inevitable
memory fragmentation).

Allocating the buffer from userspace won't really work, because you
won't have any way to get a chunk of _physically contiguous_ memory from
there. It would be the same as using something like vmalloc() in the
kernel -- virtually contiguous but physically fragmented.

Unfortunately there is no clean solution to this in the standard kernel
that I know of. I think this type of situation is fairly unique --
usually drivers/devices will have smaller DMA regions or utilize a
scatter/gather streaming DMA method.

In order to get a large chunk of physically contiguous memory, you'll
have to reserve it during boot time. One solution is to use the
"bigphysarea" patch that's been floating around, which is intended for
exactly this situation. You can find fairly recent forward-ports of
this patch here:

Old 08-11-2008, 08:13 PM   #4
LQ Veteran
Registered: Mar 2003
Location: Boise, ID
Distribution: Mint
Posts: 6,642

Rep: Reputation: 85
Thanks for posting back with the followup wkorbe, and Welcome to LQ!
Old 08-12-2008, 03:03 AM   #5
Registered: Jun 2008
Location: Phoenix, Arizona, USA
Distribution: Slackware
Posts: 326

Rep: Reputation: 49
I'll ditto J.W., thanks for the feedback! Now we all get to learn something!



Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the Linux difference between PCIE and PCI device driver? jbreaka4lyfe Linux - General 0 06-04-2008 02:54 PM
LXer: Performance Technologies Announces Availability of AMC121 High-Performance Comp LXer Syndicated Linux News 0 09-18-2007 10:30 AM
ATI-Driver problems w/X1600 Pro PCIE Varean Linux - Hardware 15 05-24-2007 07:21 AM
Looking for ABIT Radeon XT600 PCIe Driver for Linux 9.3 or 10.0 haeusermannanna Linux - Newbie 4 01-01-2006 06:09 PM
high-performance laptop memonvil Linux - Laptop and Netbook 7 03-02-2005 07:41 AM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 08:00 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration