Hi,

I am working with AMD EPYC 7643 server machine. I have connected 2 AMD machines via PCIe SFF 8644 cable. I have enabled NTB on one of the machines. I would like to read the configuration space of the device(any PCIe endpoint), from the other machine. I was able to create a shared memory window between the systems using ntb_tool. Now, I want to map a device to this shared memory window, so that I can access the device from the other machine.

I was trying to figure out how to achieve this, but I could not find any solution. It's been quite a long time since I am trying this. Could someone help me out here? It would be of great help to me, if someone could point out some resources or forums where I could get some help.

Thanks in advance.

Regards,
Sai Kiran.

I didn't get very far with NTB when I tried working on it (10 years ago). We wanted to use it for a dual-machine failover application.

It sounds like instead of allocating a contiguous physical RAM region and mapping that on the peer, you want to map to memory-mapped PCI device configuration space. PCI configuration space is part of the physical address map of the CPU, but it is not at all clear to me that the same address coming from a PCI device (your non-transparent bridge) can access PCI config space on a different device.

Hi,

Thanks for your reply. I am not sure how to map NTB to memory mapped PCIe configuration space. I tried to create NTB shared memory window at the device location. I thought that it's close to mapping device to NTB. However, when I try to access the memory window, from the other machine, I get, IO_PAGE_FAULT error in the dmesg.

Computers aren't like old PC's with everything on one bus and one address space. I doubt a device address means anything to another device. Maybe an IOMMU can be configured to make access possible; I don't know.

I was looking at the source of ntb_tool. I found that the memory window is created using the API dma_alloc_coherent(). I tried to create memory window using kzalloc(). But once I do this, I get IO_PAGE_FAULT in the dmesg, when I am accessing the memory window. So, does this mean that only DMAable memory can be used for NTB memory window? Looking deeper into the NTB source code for IO_PAGE_FAULT error, I see that there is no issue with DMA, if this error is printed on screen. I am not sure what it means. Could it be some permission issue with the kernel pages allocated or is it DMA related?

Access by NTB is DMA since it doesn't go through the CPU, so I suppose it must be a DMAable address.

Okay. The NTB memory window is DMAable. I want to map a device's memory, to this memory window. I mean, when data is written to the memory window, it should actually be written to the device's memory. Any inputs on how to do this?