LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 08-03-2008, 10:41 PM   #1
wkorbe
LQ Newbie
 
Registered: Aug 2008
Posts: 2

Rep: Reputation: 0
Question PCIe Driver - need high performance


Hello,

We are developing a spin-off of one of our PCI-X devices which runs on custom hardware under VxWorks. Our new device will be a PCIe device that will plug into a Dell server (1950 and the likes) which runs a 2.6 Linux Kernel flavor (Possibly, Red hat enterprise 5.0). The main job of this PCIe device is to push externally gathered data at a high-rate into memory for an application to consume. The rates reach over 2 Gbps, so avoiding memory copies is desired. Our PCIe device is a bus master which DMAs data into a 256 MB circular buffer. When this device receives external data, it pushes this data onto the circular buffer and then updates a write pointer to indicate data is ready to read.

I have read Linux Device Drivers, third edition, but Iím stuck on one sticky point. How can I map a huge user space buffer (256Mb, or higher) into kernel space such that a PCIe device can freely write to it with out the need for constant mitigation by a device driver (marking memory as Dirty, etc). Do I even need to map this user space buffer into kernel space, or can I just pin this buffer in memory, translate the virtual address to a physical addresses and tell our PCIe device this physical address it can write to.

I was thinking of developing a char driver, which initializes our PCIe device and implements mmap(). We want to write a user space program which controls and configures this device. Part of the configuration is to write into its registers the base address of the 256MB buffer and the address of the write pointer. We would like to allocate this 256MB in user space, lock it into memory such that is can not get swapped, translate these addresses into physical addresses that our PCIe device can write freely to them. We want to lock this buffer into physical memory for the lifetime of the driver/application; its unlikely the application/driver will ever get unloaded, the server will have at least 4GB of RAM. We also want to use cache, but are unsure if the hardware supports cache coherency or if there is a way to tell the kernel to uphold cache coherecny for this buffer.

Our kernel space driver will basically map our PCIe device registers into user space, where a user space driver/application then configures this device to transfer all external data received into a user space buffer. Our user space application will then consume and work on the data in the buffer.

Any help is appreciated, Iím quite new to Linux kernel hacking.

Thank you!

-Will
 
Old 08-04-2008, 04:02 AM   #2
storkus
Member
 
Registered: Jun 2008
Posts: 310

Rep: Reputation: 45
I hate to tell you this, but what you're asking is probably way beyond what 99% of the people on this web site know, including myself! Try http://www.kernelnewbies.org instead as that site is specifically about kernel development, hacking, etc.

Mike
 
Old 08-11-2008, 03:03 PM   #3
wkorbe
LQ Newbie
 
Registered: Aug 2008
Posts: 2

Original Poster
Rep: Reputation: 0
I tracked down a linux expert, here is the answer:

Quote:
Normally, you would allocate memory with something like
pci_alloc_consistent() or dma_alloc_coherent() when you want a PCI
device to be able to DMA into it. You are correct that using these (or
even kmalloc()) for any largish size will likely fail (due to inevitable
memory fragmentation).

Allocating the buffer from userspace won't really work, because you
won't have any way to get a chunk of _physically contiguous_ memory from
there. It would be the same as using something like vmalloc() in the
kernel -- virtually contiguous but physically fragmented.

Unfortunately there is no clean solution to this in the standard kernel
that I know of. I think this type of situation is fairly unique --
usually drivers/devices will have smaller DMA regions or utilize a
scatter/gather streaming DMA method.

In order to get a large chunk of physically contiguous memory, you'll
have to reserve it during boot time. One solution is to use the
"bigphysarea" patch that's been floating around, which is intended for
exactly this situation. You can find fairly recent forward-ports of
this patch here:

[www.feise.com]
 
Old 08-11-2008, 09:13 PM   #4
J.W.
LQ Veteran
 
Registered: Mar 2003
Location: Milwaukee, WI
Distribution: Mint
Posts: 6,642

Rep: Reputation: 69
Thanks for posting back with the followup wkorbe, and Welcome to LQ!
 
Old 08-12-2008, 04:03 AM   #5
storkus
Member
 
Registered: Jun 2008
Posts: 310

Rep: Reputation: 45
I'll ditto J.W., thanks for the feedback! Now we all get to learn something!

Mike
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the Linux difference between PCIE and PCI device driver? jbreaka4lyfe Linux - General 0 06-04-2008 03:54 PM
LXer: Performance Technologies Announces Availability of AMC121 High-Performance Comp LXer Syndicated Linux News 0 09-18-2007 11:30 AM
ATI-Driver problems w/X1600 Pro PCIE Varean Linux - Hardware 15 05-24-2007 08:21 AM
Looking for ABIT Radeon XT600 PCIe Driver for Linux 9.3 or 10.0 haeusermannanna Linux - Newbie 4 01-01-2006 07:09 PM
high-performance laptop memonvil Linux - Laptop and Netbook 7 03-02-2005 08:41 AM


All times are GMT -5. The time now is 01:54 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration