PCIe dma-able memory location in physical memory for Intel Nehalem / Tylersburg arch

robvoo · 07-08-2009, 10:45 AM

Hello,

We have the following question regarding the (re-)allocation of dma-able memory for a PCIe device. We want to know on which CPU / Memory controller the dma-able memory will be allocated. In addition we would like to know how we can control this allocation process, or how to reroute it to another CPU / Memory controller.

The system we use contains two Nehalem processors connected via intel QPI bus. Both processors have 3GB of DDR3 RAM in their memory banks. The PCIe bus is controlled by a tylersburg io hub which is also connected to the intel QPI bus

The system runs a 64bit linux kernel. Furthermore a PCIe device which can only use 32 bit addressing is connected to the PCIe bus. We want to allocate dma-able memory for this device using a driver. For maximal performance we would like to control the allocation of dma-able memory, in particular the allocation to a certain CPU / memory controller.

So our questions are:
-1- How can we know were our dma-able memory is allocated in physical memory?
-2- How can we control / redirect this allocation to a certain memory controller / CPU?

In this context CPU is the physical CPU not a logical one.

Best regards,

robvoo

cladisch · 07-14-2009, 09:14 AM

Quote:

How can we control / redirect this allocation to a certain memory controller / CPU?

Use alloc_pages_node(), kmalloc_node(), or vmalloc_node().
dma_alloc_coherent() will allocate on the device's node.

robvoo · 07-15-2009, 09:45 AM

Quote:

Originally Posted by cladisch

dma_alloc_coherent() will allocate on the device's node.

But what is the node of a device? Remember the device only supports 32 bit adressing or doesn't this have any influence on the allocation?

cladisch · 07-15-2009, 10:05 AM

Quote:

But what is the node of a device?

device.h says:

Code:

static inline int dev_to_node(struct device *dev)
{
        return dev->numa_node;
}

I don't know where this value is set; probably you are supposed to call set_dev_node().

Quote:

Remember the device only supports 32 bit adressing or doesn't this have any influence on the allocation?

This isn't related to NUMA as far as I can see ...

robvoo · 07-31-2009, 06:21 AM

We got a bit further with the DMA allocation, we know allocate a number of blocks of 4MB to transfer the data from the device.
The device allocates the dma memory on NUMA node 0 were zone DMA and zone DMA32 (x86_64) are located.
We use pci_alloc_consistent() to allocate a number of blocks of 4MB.

Our /proc/buddyinfo displays we don't have a zone NORMAl on node0.

Node 0, zone DMA 6 4 5 5 5 4 3 3 2 1 1
Node 0, zone DMA32 272 170 115 58 16 5 2 0 0 3 614
Node 1, zone Normal 388 281 157 73 12 3 0 5 2 4 656

We have a number of processes running on the other node (node 1) which does need to acces the dma-able memory
- QUESTION: how may we shrink zone DMA32 or influence how zones are mapped, we would like to have a part of zone NORMAL at node 0.

The process that runs on the other node needs to make use of the cache to get a reasonable performance
- QUESTION: how does pci_alloc_consistent set the cache settings of the memory, will this not be cached, or does it guarantee that the cache is synced?
- QUESTION: the device only transfers data from the device to the kernel, is pci_alloc_consisten the way to go or could we better use pci_map_single to set the
direction and then use pci_dma_sync_single() to sync the cache manually.

Hope my questions are clear,

Thanks in advance,