PCI passthrough via OVMF

Tango-view-fullscreen.png

Tango-view-fullscreen.png

This article or section needs expansion.

Reason: Missing introduction and descriptions of why the steps are necessary (it may be explained in the external sources, but still...) (Discuss in Talk:PCI passthrough via OVMF#)

This is a guide on how to do PCI VGA Passthrough on QEMU. Since kernel 3.19 and a change in QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful when doing graphic-intensive tasks such as gaming. To do this, you need two graphics cards, one for the host and one for the VM; it is possible to use integrated graphics for the host. Your processor and motherboard also need to support AMD-VI/VT-D.

Installation

Install qemu and rpmextract.

Install edk2.git-ovmf-x64 from Gerd Hoffman's repository.

Extract that archive to /usr:

# rpmextract.sh edk2.git-ovmf-x64-0-20150223.b877.ga8577b3.noarch.rpm
# cp -R ./usr/share/* /usr/share

Ensure /usr/share/edk2.git/ovmf-x64 contains these files:

$ ls /usr/share/edk2.git/ovmf-x64/*pure*.fd
/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd
/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd
/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd

This will be the BIOS that the VM will use. Non-UEFI users may need to use i440fx without OVMF, and the i915 vga arbiter patch for Intel graphics as host, see this forum thread. For users that do have a UEFI compatible motherboard but a UEFI incompatible graphics card, look at this post.

Setting up libvirt

See Polkit#Bypass password prompt to bypass the password prompt, then enable and start libvirtd.service.

Start virt-manager and configure the hypervisor. Virtmanager should connect to the qemu session.

PCI-STUB & IOMMU

If you are on Intel, add intel_iommu=on to your bootloader kernel options.

You may also need to add pci-stub to /etc/mkinitcpio.conf

/etc/mkinitcpio.conf
MODULES="... pci-stub ..."

if it doesn't work, you may try add it to /etc/modules-load.d/:

echo "pci-stub" > /etc/modules-load.d/libvirt.conf

Check dmesg to confirm after reboot:

dmesg|grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.028950] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    0.536212] DMAR: No ATSR found
[    0.536229] IOMMU 0 0xfed90000: using Queued invalidation
[    0.536230] IOMMU 1 0xfed91000: using Queued invalidation
[    0.536231] IOMMU: Setting RMRR:
[    0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]
[    0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    2.182790] [drm] DMAR active, disabling use of stolen memory

Bind the GPU

Find your target card's PCI location:

lspci -nn|grep -iP "NVIDIA|Radeon"
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] [1002:6719]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80]
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation G73 [GeForce 7600 GS] [10de:0392] (rev a1)

Add the PCI location to the kernel command line (e.g., in file /etc/default/grub, parameter "GRUB_CMDLINE_LINUX_DEFAULT"):

pci-stub.ids=10de:0392

If your graphic card have audio as separated PCI device it's must be added as well:

pci-stub.ids=1002:6719,1002:aa80

Check dmesg output for successful assigning of the device to a pci-stub:

dmesg | grep pci-stub
...
[    2.390128] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.390143] pci-stub 0000:01:00.0: claimed by stub
[    2.390150] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.390159] pci-stub 0000:01:00.1: claimed by stub
...

There are many methods to bind the card to vfio, here are two examples:

Blacklist modules

Blacklist radeon or nouveau or nvidia or fglrx in /etc/modprobe.d/blacklist.conf.

Example, blacklisting the opensource radeon module:

/etc/modprobe.d/modprobe.conf
blacklist radeon

QEMU permissions

Give QEMU access to hardware (there may be safer ways of doing this):

/etc/libvirt/qemu.conf
...
user = "root"
group = "root"
clear_emulator_capabilities = 0

QEMU also needs acces to VFIO files. Include every numbered file in /dev/vfio:

ls -1 /dev/vfio
/etc/libvirt/qemu.conf
...
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
    "/dev/vfio/1"
]
...

Referenced from firewing1's webpage

QEMU commands

This is the command to run QEMU with VGA Passthrough:

cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd
qemu-system-x86_64 \
  -enable-kvm \
  -m 2048 \
  -cpu host,kvm=off \
  -vga none \
  -device vfio-pci,host=01:00.0 \
  -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
  -drive if=pflash,format=raw,file=/tmp/my_vars.fd

-enable KVM - enables KVM for your system, using AMD-VI/VT-D for hardware virtualisation.

-m [number] - sets the amount of memory the VM should have.

-cpu host, kvm=off - emulate the host's exact CPU. kvm=off is used for NVIDIA cards to stop it detecting a hypervisor and therefore exiting with an error.

-vga none - disables the built in graphics card emulation.

-device vfio-pci,host=01:00.0 \ - graphics card(s) you are using for VGA passthrough.

-drive if=flash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd BIOS location.

For more commands see QEMU.

Create and configure VM for OVMF

Alex Williamson's blog

Use virsh to edit the VM with these changes:

<domain type='kvm'>
<os>
 <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
 <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/>
</os>
<hyperv>
 <relaxed state='off'/>
 <vapic state='off'/>
 <spinlocks state='off'/>
</hyperv>
<features>
 <kvm>
  <hidden state='on'/>
 </kvm>
</features>
<clock>
 <timer name='hypervclock' present='no'/>
</clock>

Operating system

Depending on your operating system, you may find that it may refuse to boot after a certain point. To work around this, simply replace -vga none to -vga qxl, install your operating system, check Device Manager and see if your graphics card has PCI device id equal to your actual GPU and install the graphics card driver, and then change it back to -vga none.

See also