LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices



Reply
 
Search this Thread
Old 07-08-2009, 11:45 AM   #1
robvoo
LQ Newbie
 
Registered: Jul 2009
Distribution: debian, ubuntu
Posts: 22

Rep: Reputation: 2
PCIe dma-able memory location in physical memory for Intel Nehalem / Tylersburg arch


Hello,

We have the following question regarding the (re-)allocation of dma-able memory for a PCIe device. We want to know on which CPU / Memory controller the dma-able memory will be allocated. In addition we would like to know how we can control this allocation process, or how to reroute it to another CPU / Memory controller.

The system we use contains two Nehalem processors connected via intel QPI bus. Both processors have 3GB of DDR3 RAM in their memory banks. The PCIe bus is controlled by a tylersburg io hub which is also connected to the intel QPI bus

The system runs a 64bit linux kernel. Furthermore a PCIe device which can only use 32 bit addressing is connected to the PCIe bus. We want to allocate dma-able memory for this device using a driver. For maximal performance we would like to control the allocation of dma-able memory, in particular the allocation to a certain CPU / memory controller.

So our questions are:
-1- How can we know were our dma-able memory is allocated in physical memory?
-2- How can we control / redirect this allocation to a certain memory controller / CPU?

In this context CPU is the physical CPU not a logical one.

Best regards,

robvoo
 
Old 07-14-2009, 10:14 AM   #2
cladisch
Member
 
Registered: Oct 2008
Location: Earth
Distribution: Slackware
Posts: 227

Rep: Reputation: 54
Quote:
How can we control / redirect this allocation to a certain memory controller / CPU?
Use alloc_pages_node(), kmalloc_node(), or vmalloc_node().
dma_alloc_coherent() will allocate on the device's node.
 
Old 07-15-2009, 10:45 AM   #3
robvoo
LQ Newbie
 
Registered: Jul 2009
Distribution: debian, ubuntu
Posts: 22

Original Poster
Rep: Reputation: 2
Device Node

Quote:
Originally Posted by cladisch View Post
dma_alloc_coherent() will allocate on the device's node.
But what is the node of a device? Remember the device only supports 32 bit adressing or doesn't this have any influence on the allocation?
 
Old 07-15-2009, 11:05 AM   #4
cladisch
Member
 
Registered: Oct 2008
Location: Earth
Distribution: Slackware
Posts: 227

Rep: Reputation: 54
Quote:
But what is the node of a device?
device.h says:
Code:
static inline int dev_to_node(struct device *dev)
{
        return dev->numa_node;
}
I don't know where this value is set; probably you are supposed to call set_dev_node().

Quote:
Remember the device only supports 32 bit adressing or doesn't this have any influence on the allocation?
This isn't related to NUMA as far as I can see ...
 
Old 07-31-2009, 07:21 AM   #5
robvoo
LQ Newbie
 
Registered: Jul 2009
Distribution: debian, ubuntu
Posts: 22

Original Poster
Rep: Reputation: 2
We got a bit further with the DMA allocation, we know allocate a number of blocks of 4MB to transfer the data from the device.
The device allocates the dma memory on NUMA node 0 were zone DMA and zone DMA32 (x86_64) are located.
We use pci_alloc_consistent() to allocate a number of blocks of 4MB.

Our /proc/buddyinfo displays we don't have a zone NORMAl on node0.

Node 0, zone DMA 6 4 5 5 5 4 3 3 2 1 1
Node 0, zone DMA32 272 170 115 58 16 5 2 0 0 3 614
Node 1, zone Normal 388 281 157 73 12 3 0 5 2 4 656

We have a number of processes running on the other node (node 1) which does need to acces the dma-able memory
- QUESTION: how may we shrink zone DMA32 or influence how zones are mapped, we would like to have a part of zone NORMAL at node 0.

The process that runs on the other node needs to make use of the cache to get a reasonable performance
- QUESTION: how does pci_alloc_consistent set the cache settings of the memory, will this not be cached, or does it guarantee that the cache is synced?
- QUESTION: the device only transfers data from the device to the kernel, is pci_alloc_consisten the way to go or could we better use pci_map_single to set the
direction and then use pci_dma_sync_single() to sync the cache manually.

Hope my questions are clear,

Thanks in advance,
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
DMA Transfer from Network Interface to Memory? Intel e1000 nematoad Linux - Hardware 3 12-27-2008 05:15 PM
Physical Memory and Virtual Memory Murdock1979 Linux - General 1 01-28-2008 08:35 AM
Less physical memory? ariek Debian 5 07-21-2005 03:23 PM
get Physical memory in C/C++ Dreampainter Programming 1 10-30-2004 09:54 PM
How Much Physical Memory ? tied2 Linux - Newbie 3 06-17-2002 02:30 AM


All times are GMT -5. The time now is 09:47 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration