With the 4.20 kernel, PCI-E peer to peer support has been added. I have a device that I'm using as a p2p memory buffer. I have complete visibility into this device in case that's relevant. I'm able to create p2p memory areas and everything looks good. Unfortunately I'm not very knowledgeable of many of the lower lever details in the kernel and kernel modules. What I want to do is take my p2p memory regions and use the NVME target driver to copy data from the p2p memory onto disk. I don't really have an opinion on the interface to accomplish this, just that the peer to peer path is exercised. I've looked for examples, but haven't really found anything. I tried a couple of things myself including:

1. Using kernel file IO (ugly I know) and opening a file directly to the nvme disk itself (/dev/nvme0n1) and passing in the p2pmem buffer this way. Unfortunately this just routes everything through the page cache and the CPU copies one word at a time into the page cache and then DMAs to the disk. I'm trying to bypass system memory.

2. Same as #1 above, but Tried using O_DIRECT. Unfortunately O_DIRECT really only works on userspace memory buffers near as I can tell so no luck there.

3. Tried the NVME userspace ioctls and passing mmaped memory from the PCI-E bar. Perhaps there's a way to mmap the p2pmem buffer itself? Not sure if that question even makes sense.

Any thoughts/articles/examples would be greatly appreciated.

Thank you!

Until a standard kernel API for p2pmem is added you should take a look at SPDK.

https://www.flashmemorysummit.com/En...01-1_Bates.pdf

I think you can do it in both Linux and SPDK...

Your P2Pmem is in a NVMe drive or it is another type of device? Do you already have a P2PMem part of the driver for it?