NVIDIA Expands DMA-BUF Support for VFIO PCI Devices in Linux 6.19
Beyond improving peer-to-peer (P2P) DMA for block devices in Linux 6.19, NVIDIA also spearheaded a change that enables DMA-BUF support for VFIO PCI devices. This opens up several new scenarios as the VFIO work lands in the Linux 6.19 patch set.
The DMA-BUF integration for VFIO PCI devices comes from the collaboration of NVIDIA engineers Leon Romanovsky and Jason Gunthorpe, alongside Intel’s Vivek Kasireddy. The VFIO patch notes describe the new capability as follows:
"Introduce dma-buf support for vfio-pci devices, allowing MMIO regions to be exposed through dma-buf objects with lifecycle managed via move operations. This enables low-level interactions such as a vfio-pci based SPDK driver communicating directly with dma-buf capable RDMA devices to enable peer-to-peer operations. IOMMUFD can now build on this support to address a long-standing gap versus the legacy vfio type1 IOMMU backend by implementing P2P support for VM use cases that better manage the lifecycle of the P2P mappings."
The patch series (detailed in the linked discussion) further clarifies:
"This series extends the VFIO PCI subsystem to export MMIO regions from PCI device BARs as dma-buf objects, enabling safe sharing of non-struct memory with controlled lifetimes. This permits RDMA and other subsystems to import dma-buf FDs and incorporate them into memory regions for PCI P2P operations.
A concrete SPDK use-case is supported, where an NVMe device can be owned by SPDK via VFIO but interact with a RDMA device. The RDMA device may directly access the NVMe CMB or manipulate the NVMe doorbell through PCI P2P.
As a general mechanism, it covers many VFIO scenarios. This dma-buf approach can also be leveraged by iommufd for safe P2P mappings.
Additionally, the patch set enables sharing of buffers located in device memory (for example VRAM) between two dGPU devices or instances, provided one is bound to VFIO PCI and both support P2P DMA."
A substantial refactor accompanies this work, reorganizing the PCI P2PDMA subsystem to separate core P2P functionality from memory allocation logic. The result is a more modular design better suited for VFIO scenarios that do not depend on struct page support.
Linus Torvalds merged these VFIO changes into the Linux 6.19 cycle on a Friday.
If you’d like, this summary can be expanded with practical implications, potential deployment steps, or a quick comparison of how DMA-BUF P2P works versus previous VFIO limitations.
