PCI passthrough via OVMF/Examples

From ArchWiki
Jump to: navigation, search

As PCI passthrough is quite tricky to get right (both on the hardware and software configuration sides), this page presents working, complete VFIO setups. Feel free to look up users' scripts, BIOS/UEFI configuration, configuration files and specific hardware. If you have a problem, it might have been stumbled upon by other VFIO users and fixed in the examples below.

Note: If you have got VFIO working properly, please post your own setup according to the template on the bottom.

Users' setups

mstrthealias: Intel 7800X / X299, GTX 1070

Hardware:

  • CPU: Intel(R) Core(TM) i7-7800X CPU
  • Motherboard: ASRock X299 Taichi (Revision: A, BIOS/UEFI Version: 1.60A)
  • GPU: Asus STRIX GTX 1070
  • RAM: 32GB DDR4

Configuration:

  • Kernel: Kernel version 4.14.8-1-skx (patched crystal_khz=24000).
    • Custom patches:
      • skylakex-crystal_khz-24000.patch (see below)
    • Patches used from linux-ck:
      • enable_additional_cpu_optimizations_for_gcc_v4.9+_kernel_v4.13+.patch
      • 0001-add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by.patch
      • 0001-e1000e-Fix-e1000_check_for_copper_link_ich8lan-retur.patch
      • 0002-dccp-CVE-2017-8824-use-after-free-in-DCCP-code.patch
    • Config:
      • PREEMPT, NO_HZ_IDLE, 300HZ, MSKYLAKE
  • GitHub: Link TBD
  • Benchmarks: https://imgur.com/a/hIfQD
  • Using libvirt/QEMU: libvirt 3.10.0 / QEMU 2.11.0
  • Issues you have encountered, special steps taken to make something work a bit better, etc.
    • Skylake-X default clock incorrect in 4.14.8 (https://bugzilla.kernel.org/show_bug.cgi?id=197299)
      • Was unable to resolve timing issue using adjtimex
      • Patching kernel source to crystal_khz = 24000 resolved timing/performance issues
    • Enable 'Intel SpeedShift' in BIOS, installed cpupower', set governor='performance'
      • Verify: dmesg|grep HWP
        • intel_pstate: HWP enabled
    • Enable HT in BIOS
    • Enable 'deadline' IO sceduler:
      • echo 'ACTION=="add|change", KERNEL=="sd*[!0-9]|sr*", ATTR{queue/scheduler}="deadline"' >> /etc/udev/rules.d/60-schedulers.rules
    • Bypass x2apic opt-out:
      • GRUB_CMDLINE_LINUX="... intremap=no_x2apic_optout ..."
    • Isolate cores for Windows VM:
      • GRUB_CMDLINE_LINUX="... isolcpus=2-5,8-11 nohz_full=2-5,8-11 rcu_nocbs=2-5,8-11 ..."
    • Use hugepages (2MB) for all VM memory allocation
    • memoryBacking: <hugepages/><nosharepages/><locked/><access mode='private'/><allocation mode='immediate'/>
    • Extracted rom from GPU; used for <rom file=../> config
    • Using MSI for GPU and GPU Audio (configured in Windows registry; FPS seems same as using line-based interrupts)
  • Hardware setup
    • PCIE1: NVIDIA GeForce GT 710B (for host)
    • Onboard: ASRock XHCI 3.1 USB (for host)
    • Onboard: Intel I219 NIC (bridged)
    • PCIE3: Asus Xonar STX (passthrough to Win10)
    • PCIE5: NVIDIA GeForce GTX 1070 (passthrough to Win10)
    • M2_1: Samsung 960 EVO 500GB (passthrough to Win10)
    • Onboard: Intel XHCI USB 3.0 (passthrough to Win10)
    • Onboard: Intel HDA (passthrough to Win10)
    • Onboard: Intel I211 NIC (passthrough to Win10)
    • Onboard: ASRock AHCI SATA A1/A2 (passthrough to Linux)


DragoonAethis: 6700K, GA-Z170X-UD3, GTX 1070

Hardware:

  • CPU: Intel Core i7-6700K (using iGPU as the host GPU)
  • Motherboard: Gigabyte GA-Z170X-UD3 (Revision 1.0, BIOS/UEFI Version: F22)
  • GPU: MSI GeForce 1070 Gaming X (10Gbps)
  • RAM: 16GB DDR4 2400MHz

Configuration:

  • Kernel: Linux-ck (no ACS patch needed).
  • Using libvirt: XML domain, helper scripts, IOMMU groups, etc available in my VFIO repository.
  • Guest OS: Windows 8.1 Pro.
  • The entire HDD is passed to the VM as a raw device (formatted as a single NTFS partition).
  • USB keyboard and mouse are passed to the guest VM and shared with the host with Synergy.
  • Virtualized audio is working: PulseAudio on the host is configured to accept TCP connections, and the envvars required for QEMU to use PA are pointed at the PA server running on 127.0.0.1. This way it's not required to change the QEMU user, everything works flawlessly. (Exact details in the repo.)
  • Bridged networking (with NetworkManager's and this tutorial's help) is used. bridge0 is created, eth0 interface is bound to it. STP disabled, VirtIO NIC is configured in the VM and that VM is seen in the network just as any other computer (and is being assigned an IP address from the router itself, can communicate freely with other computers).

Manbearpig3130's Virtual Gaming Machine

Hardware:

  • CPU: Intel Core i7-6850K 3.6GHz
  • Motherboard: Gigabyte x99-Ultra Gaming (Revision 1.0, BIOS/UEFI Version: F4)
  • Host GPU: AMD Radeon HD6950 1GB
  • Guest GPU: AMD R9 390 8GB
  • RAM: 32GB G-Skill Ripjaws DDR4 runing at 3200MHz

Configuration:

  • Host Kernel: Kernel version Linux 4.7.2-1 (No ACS patch).
  • Using libvirt QEMU/KVM with OVMF: link to domain XMLs/scripts/notes: https://github.com/manbearpig3130/MBP-VT-d-gaming-machine
  • Host OS: Arch Linux
  • Guest OS: Windows 10 Pro
  • 2x 480GB SSDs set up in LVM striped mode (with mdadm) formatted to ext4 are mounted in linux which contains the guest's qcow2 virtual VirtIO disk file.
  • USB Host controller is passed through, giving most USB ports to the VM, leaving my USB 3.1 controller with attached USB hub for the host.
  • Motherboard has two NICs, one is passed into VM (Works perfectly after installing Killer NIC Driver).
  • VM gets dedicated 16GB RAM via static hugepages.
  • CPU pinning increased performance considerably.
  • Windows boots straight into Steam big picture mode on primary display (43" Sony Bravia). Overall an awesome gaming machine that meets my gaming needs and lust for GNU/Linux at the same time.
  • Quirks:
  • I sometimes have to reinstall the AMD drivers in Windows to get HDMI audio working properly, or roll back to Windows HDMI driver.
  • I find that if I allow Windows to go through its shutdown procedure it has trouble booting again. To get around this I force off the machine in virtual machine manager gui.

Bretos' Virtual Gaming Setup

Hardware:

  • CPU: Intel Core i7-7700k
  • Motherboard: Z270 GAMING M3 (MS-7A62)
  • GPU: ASUS GeForce GTX960
  • RAM: Kingston HyperX 3x8GB DDR4 2.4GHz
  • Storage: 2x Corsair MP500 m.2 240G SSDs in mdadm RAID0, 1x WD Black 1TB for storage. 100GB LVM volume as writeback cache for HDD

Configuration:

  • Kernel: vanilla
  • Host OS: Arch Linux
  • Guest OS: Windows 10 Pro
  • Using libvirt/QEMU: GitHub config repository: [1]
  • Issues you have encountered: AUDIO. Had to get USB audio adapter and pass it through.
  • No issues other than audio. Works like a charm.

Skeen's Virtual Gaming Rack Machine

Still work in progress.

Hardware:

  • CPU: AMD FX(tm)-8350
  • Motherboard: MSI 970A SLI Krait Edition (MS-7693) (Revision 5.0, BIOS/UEFI Version: 25.4)
  • Host GPU: ASUS GeForce GTX 480 1536MB
  • Guest GPU: ASUS GeForce GTX 480 1536MB
  • RAM: 2x8GB Kingston HyperX Fury White DDR3 1866MHz
  • Storage: 2x250GB Samsung EVO (MZ-75E250) set up in LVM striped mode (with mdadm), 2x1TB WD Blue (WDC_WD10SPCX) for storage. 250GB LVM volume as writeback cache for HDD.

Configuration:

  • Host Kernel: Linux 4.9.0-3 (No ACS)
  • Host OS: Debian Stretch
  • Guest OS: Windows 10 Home (10_1703_N, International Edition)
  • Using libvirt QEMU/KVM with OVMF: See Github

Issues you have encountered:

  • Identifical GPUs; solved using this section on the wiki, but with the script from the corresponding discussion page. Several adaptations for Debian were required too, but are not applicable for this forum.
  • "Error 43: Driver failed to load";
    • Spoofing vendor_id caused Windows to crash during boot-up.
    • Linux VMs complain unable to find GPU from Grub2, and booted into blind-mode, but would still pick up the graphics card during the boot process, and would remain functional until VM reboot.
    • Vendor_id spoofing turned out to work after solving the real problem (Missing UEFI compatability in VBIOS).
  • Missing UEFI (OVMF) compatability in VBIOS;
    • Requested a GOP/UEFI compatible VBIOS upgrade from ASUS, but ASUS could neither understand the request, or provide the upgrade (The only thing supplied was standard support answers).
    • No compatible VBIOS was found at TechPowerUp.
    • Finally solved by manually hacking GOP/UEFI support into the ROM, using GOPupd. Current rom was dumped within a Windows 10 VM using GPU-Z, then modified using GOPupd, pulled to Linux, and provided using the rom file parameter in the VM XML file.
  • VM only uses one core (even with mode=host-passthrough): solved using this section on the wiki.

Quirks:

  • The GPU that is being passed through, does not support resetting, and thus doing a hard-reboot / shutdown of the VM locks the GPU.
    • The VM cannot be started again unless the Host machine is rebooted.
      • When doing a clean reboot / shutdown, allows the VM to start up as expected without reboot..
    • Removing and rescanning the PCI device, does not change anything.
    • No further attempts at powercycling the GPU from the host has been done (Yet).
  • Passing VM audio to host via PulseAudio results in heavy crackling.

droserasprout poor man's setup

Hardware:

  • CPU: Intel Core i3-6100
  • Motherboard: ASRock H110M2 D3 (BIOS version 0603)
  • Host GPU: Intel HD 530
  • Guest GPU: Sapphire Radeon R7 360
  • RAM: Apacer 8Gb 75.C93DE.G040C, Kingston 4Gb 99U5401-011.A00LF

Configuration:

  • Kernel: linux-lts 4.9.67-1 (vanilla)
  • Host OS: Arch Linux
  • Guest OS: Windows 10 Pro 1709 (build 16299.98)
  • Using libvirt/QEMU: See my configs and IOMMU groups on Github
  • HDD partition is passed to the VM as a raw virtio device.
  • HD Audio is passed too. Works fine with both playing and recording, no latency issues or glitches. After VM is powered off host audio works fine too.
  • Guest's latency is slightly better when CPU cores are isolated for VM.
  • i2c-dev module added to bypass 'EDID signature' error when switching HDMI. Without it I had to switch video output before starting VM for some reason.
  • intremap=no_x2apic_optout kernel option added to bypass motherboard firmware falsely reporting x2APIC method is not supported. Seems to have a strong influence on the guest's latency.
  • Overall performance is pretty close to the native OS setup.

Adding your own setup

Add a new section with your nickname, CPU, motherboard and GPU models, then copy and paste this template to your section:

Hardware:

* '''CPU''': 
* '''Motherboard''': (Revision , BIOS/UEFI Version: )
* '''GPU''': 
* '''RAM''': 

Configuration:

* '''Kernel''': Kernel version (vanilla/CK/Zen/ACS-patched or not).
* Using '''libvirt/QEMU''': link to domain XMLs/scripts/notes (Git repo preferred).
* Issues you have encountered, special steps taken to make something work a bit better, etc.
* Describe your setup loosely here, so that when other wiki users are looking for something, they can easily skim through available setups.

Replace proper sections with your own data. Make sure to provide the exact motherboard model, revision (if possible - should be on both the motherboard itself and the box it came in) and BIOS/UEFI version you are using. Describe your exact software setup and add a link to your configuration files. (GitHub, GitLab, BitBucket, etc can host a public repository which you may update once in a while, but uploading them to pastebins is fine, too. Do not post the entire config file contents here.)