GPU Passthrough with VFIO

Introduction¶

VFIO GPU passthrough gives a VM direct, exclusive access to a physical GPU — near-native performance for gaming, ML, or CAD in a VM. It's fiddly: the hardware, firmware, and kernel all have to cooperate.

Prerequisites¶

• CPU + motherboard IOMMU support: Intel VT-d / AMD-Vi, enabled in BIOS.
• Enable in the kernel cmdline:

  intel_iommu=on iommu=pt        # Intel
  amd_iommu=on  iommu=pt          # AMD
  

• A second GPU (or integrated graphics) for the host, unless you're doing a headless host / single-GPU passthrough (advanced, involves unbinding the console).

Check IOMMU groups¶

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/}; n=${n%%/*}
  printf 'Group %s: ' "$n"
  lspci -nns "${d##*/}"
done | sort -V

The GPU and its audio function should be alone in their IOMMU group (or with only their own bridge). If unrelated devices share the group, you must pass them all through too, or use the ACS override patch (breaks isolation guarantees — know the risk) or move the card to a different slot.

Bind to vfio-pci¶

Get the device IDs (lspci -nn → 10de:2482, 10de:228b for the HDMI audio):

# /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2482,10de:228b
softdep drm pre: vfio-pci

# ensure vfio loads early
# /etc/modules-load.d/vfio.conf
vfio_pci
vfio
vfio_iommu_type1

Rebuild initramfs (update-initramfs -u / dracut -f) and reboot. Verify:

lspci -nnk -d 10de:2482
#   Kernel driver in use: vfio-pci        <- success

If the host GPU driver (nvidia, amdgpu, nouveau) grabbed it first, blacklist it or ensure vfio-pci binds earlier (softdep).

The libvirt VM¶

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source><address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/></source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source><address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/></source>   <!-- HDMI audio -->
</hostdev>

• <vcpu> pinning + <cputune> to dedicate physical cores; keep the VM's cores off the host scheduler (isolcpus / cpuset).
• Hugepages for the VM's RAM (<memoryBacking><hugepages/>) reduces overhead.
• Machine type q35 (PCIe), OVMF/UEFI firmware (<loader>), not SeaBIOS for modern GPUs.
• NVIDIA + older drivers: hide the hypervisor (<kvm><hidden state='on'/></kvm>, <vendor_id state='on' value='randomid'/>) to avoid Code 43. Recent drivers (2021+) don't need this.

vBIOS¶

Some GPUs (especially if it was the boot GPU) need their video BIOS ROM supplied to the VM:

# dump it (best done with the card NOT initialised — before the host uses it)
echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom
cat /sys/bus/pci/devices/0000:01:00.0/rom > /usr/share/vgabios/gpu.rom
echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom

<hostdev ...>
  <rom file='/usr/share/vgabios/gpu.rom'/>
</hostdev>

Patched vBIOS files from the internet are a risk; prefer dumping your own or using a known-good one for your exact card.

The reset bug¶

Many AMD GPUs (Vega, Navi 10) don't reset properly between VM shutdowns — the card hangs and the host needs a reboot to use it again. Mitigations: vendor-reset kernel module, keeping the VM running, or a card known not to have the bug. NVIDIA cards generally reset fine.

Verification and troubleshooting¶

dmesg | grep -iE 'vfio|iommu|DMAR'
lspci -nnk -d 10de:2482
virsh nodedev-list --tree | grep -i pci

• Black screen / no output on the passed-through monitor — no vBIOS (add the <rom>), wrong monitor input, the card needs UEFI (OVMF) not SeaBIOS, or the guest driver isn't installed yet (you get nothing until Windows/Linux loads the GPU driver — connect via a virtual display / RDP / SSH first to install it).
• NVIDIA Code 43 — old driver detecting the hypervisor; hide it (kvm hidden
• random vendor_id), or update the driver.
• vfio-pci: probe of 0000:01:00.0 failed with error -22 / device in use — the host driver claimed it. Blacklist it, fix softdep, or the IOMMU group isn't isolated.
• Failed to set iommu for container: Operation not permitted — IOMMU not enabled in BIOS or the kernel cmdline didn't take (cat /proc/cmdline).
• Group has extra devices — pass them all, relocate the card, or ACS override (pcie_acs_override=downstream,multifunction — reduces isolation security).
• AMD card won't work after the first VM reboot — the reset bug; vendor-reset or don't reboot the VM.
• Stutter / poor performance — CPU cores not pinned/isolated, no hugepages, the host stealing the VM's cores, or cache=none missing on the VM disk.

Related tools and reading¶

• On-site: Linux commands reference.
• Related posts: Nested virtualization, Live migration requirements.