Nested Virtualization

Introduction¶

Nested virtualization means an L1 guest (a VM) can itself run L2 guests (VMs inside that VM). Uses: CI runners that need KVM, building/testing hypervisor configs, running WSL2 / Docker Desktop / Android emulators inside a Windows VM, training labs.

Enabling on the host (L0)¶

# Check current state
cat /sys/module/kvm_intel/parameters/nested     # Y or 1 = enabled
cat /sys/module/kvm_amd/parameters/nested

# Enable persistently
# /etc/modprobe.d/kvm-nested.conf
options kvm_intel nested=1        # Intel
options kvm_amd   nested=1         # AMD

# reload (no running VMs) or reboot
modprobe -r kvm_intel && modprobe kvm_intel

Exposing to the L1 guest¶

The L1 VM must see the virtualization CPU flags. In libvirt:

<cpu mode='host-passthrough' check='none' migratable='on'/>

host-passthrough exposes vmx (Intel) / svm (AMD) to the guest. host-model often works too on recent libvirt. A fixed named CPU model usually does not include the virtualization flags — check.

Verify inside the L1 guest:

grep -E -o 'vmx|svm' /proc/cpuinfo | sort -u
lscpu | grep -i virtual
kvm-ok                    # Ubuntu: "KVM acceleration can be used"
ls -l /dev/kvm            # must exist in the L1 guest

Then install libvirt/qemu in the L1 guest and create L2 VMs normally.

Performance¶

• L2 CPU — close to L1 (the hardware does most nested work via VMCS shadowing / nested page tables). Expect maybe 5–15% overhead vs L1.
• L2 I/O and interrupts — the bigger hit; every L2 exit potentially causes an L1 exit. Use virtio everywhere in the L2, and vhost where possible.
• Not for production workloads. Fine for CI, dev, and functional testing; don't run a customer-facing database three levels deep.

Migration: live-migrating an L1 VM that's running L2 guests is generally not supported / unreliable. Plan L1 hosts running nested workloads as non-migratable.

Cloud support¶

• AWS — only bare-metal instance types (*.metal) can do KVM; regular EC2 instances cannot nest (no /dev/kvm).
• GCP — supported on most Intel instances if you set the enable-nested-virtualization licence/flag on the image; not on all machine types.
• Azure — supported on Dv3/Ev3 and later families (v3+), not v2.
• Nested containers (Docker-in-Docker) are a different, easier thing — no hardware virtualization needed, just cgroups/namespaces and often --privileged or sysbox.

Windows / Hyper-V specifics¶

• Running Hyper-V, WSL2, or Docker Desktop (WSL2 backend) inside a Windows VM needs nested virt exposed to that VM.
• Once Hyper-V is enabled in the Windows guest, Windows runs itself under its own hypervisor — so anything else wanting vmx/svm in that guest (VirtualBox, VMware Workstation) must use the Hyper-V/WHP platform APIs.

Verification and troubleshooting¶

# L0
cat /sys/module/kvm_intel/parameters/nested
# L1 guest
egrep -c '(vmx|svm)' /proc/cpuinfo        # >0
dmesg | grep -i kvm

• L1 guest has no vmx/svm — nested not enabled on L0, CPU mode isn't host-passthrough/host-model, or a named CPU model without the flag. Also: the L1 guest was created before you enabled nesting — recreate/redefine.
• /dev/kvm missing in the L1 guest — flags present but the kvm / kvm_intel module didn't load in the guest (blacklisted, or the guest kernel lacks it), or permissions (kvm group).
• L2 VM extremely slow / falls back to TCG emulation — QEMU in L1 isn't using KVM (no /dev/kvm, or -accel tcg). virsh domcapabilities in L1; -accel kvm.
• L1 crashes / L2 hangs on start — a known incompatibility between L0 and L1 kernel versions for nested features; update both, or match hypervisor versions.
• Cloud instance: KVM: disabled by BIOS — the instance type doesn't support nesting (most don't). Use a metal instance or the cloud's flag.
• Migration of the L1 broke the L2s — expected; don't migrate nested hosts.

Related tools and reading¶

• On-site: Linux commands reference.
• Related posts: GPU passthrough with VFIO, KVM and libvirt: your first VM.