This document catalogs my working setup for configuring an Arch Linux host OS to run a Windows 10 VM while passing through a virtualized GPU.
The resources I used during my own install. This document is a compilation of the instructions and files provided by these people.
- vGPU Wiki
- PCI passthrough via OVMF
- Proxmox 7 vGPU by wvthoog
- vGPU YouTube guide by Craft Computing
- Looking Glass
- Looking Glass YouTube guide by Pavol Elsig
- vGPU unlock scripts by DualCoder
- vGPU_LicenseBypass by KrutavShah
- vGPU unlock patch for kernel 5.12 by rupansh
The what and how. Mostly described in suggested installation order unless otherwise stated.
Configuration of the host OS for virtualization using KVM and device passthrough using OVMF, IOMMU, and VFIO.
- vfio
- vfio_iommu_type1
- vfio_pci
- vfio_virqfd
- Edit /etc/mkinitcpio.conf
- Add
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)
to the MODULES line. - Add
HOOKS=(... modconf ...)
to the HOOKS line. - Update
update-initramfs -P
(usemkinitpico -P
for manjaro)
- amd_iommmu=on
- iommu=pt
- vfio_iommu_type1.allow_unsafe_interrupts=1
- kvm.ignore_msrs=1
- blacklist=nouveau
- edit /etc/default/grub
- Add to GRUB_CMDLINE_LINUX_DEFAULT:
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 blacklist=nouveau ..."
- Update GRUB
update-grub
The physical devices present on my system.
- CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz
- GPU: NVIDIA GeForce GTX 1060 6GB
- Memory: 16MiB
- 1440p Monitor
What needs to be installed.
Packages installed usingpacman
.
The OS kernel.
Version: 5.12.15How to installFirst option- Install
downgrade
from AUR (usingyay
or other AUR helper) downgrade linux
- Select 5.12.15
- Download package
curl -O https://archive.archlinux.org/packages/l/linux/linux-5.12.15.arch1-1-x86_64.pkg.tar.zst
- Install with
pacman -U linux-5.12.15.arch1-1-x86_64.pkg.tar.zst
Needed to build kernel modules, particularly the Nvidia driver.
Version: 5.12.15How to installFirst option- Install
downgrade
from AUR (usingyay
or other AUR helper) downgrade linux-headers
- Select 5.12.15
- Download package
curl -O https://archive.archlinux.org/packages/l/linux-headers/linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst
- Install with
pacman -U linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst
Used to create and run virtual machines.
(Kernel-based Virtual Machine). A hypervisor. Used for virtualization.
Used to view VM in graphical display.
For networking with the VM.
Used for VM networking.
Used to create vGPU devices
Used to read audio stream from network.
How to setup- Install using package manager.
- Create systemd-file
[Unit] Description=Scream network audio client [Unit] Description=Scream Receiver After=pulseaudio.service network-online.target Wants=pulseaudio.service [Service] Type=simple ExecStartPre=/bin/sleep 3 ExecStart=scream -i virbr0 Restart=always RestartSec=1 [Install] WantedBy=default.target
pip
.
How to installIf using conda
- Install
python-pip
using pacman. - Run
/bin/pip install frida
pip install frida
curl -O the_link
Make the driver executable. chmod +x NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5.run
Extract it. ./NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5.run -x
cd NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5
Apply the 5.12 patch patch -p0 < ../twelve.patch
Disable graphics sudo systemctl isolate multi-user.target
Re-login as root and navigate back to the driver directory
cd /home/user/Downloads/NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5
./nvidia-installer --dkms
vgpu_unlock
to service files.
nano /lib/systemd/system/nvidia-vgpud.service
Replace ExecStart with:
ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpud
nano /lib/systemd/system/nvidia-vgpu-mgr.service
Replace ExecStart with:
ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpu-mgr
systemctl daemon-reload
vgpu_unlock
to source files.
nano /usr/src/nvidia-460.73.01/nvidia/os-interface.c
Under #include “nv-time.h” insert this line
#include "/root/vgpu_unlock/vgpu_unlock_hooks.c"
nano /usr/src/nvidia-460.73.01/nvidia/nvidia.Kbuild
Add this to the bottom of the file.
ldflags-y += -T /root/vgpu_unlock/kern.ldRemove and reinstall the Nvidia dkms module
dkms remove -m nvidia -v 460.73.01 --all
dkms install -m nvidia -v 460.73.01
chmod -R +x vgpu_unlock
- nvidia-48 GRID P40-3Q
mdevctl types
I chose nvidia-47 (GRID P40-3Q) because it has 3Q at the end of its name. This means I can have two 3GB vGPU’s running.
Determine GPU PCI addresslspci | grep VGA
PCI_ADDRESS=$(lspci | grep VGA | grep -Po '\d\w:\d\d.\d')
echo $PCI_ADDRESS
UUID1=$(uuidgen)
UUID2=$(uuidgen)
echo $UUID1 $UUID2
mdevctl start -u $UUID1 -p 0000:$PCI_ADDRESS -t nvidia-48
mdevctl start -u $UUID2 -p 0000:$PCI_ADDRESS -t nvidia-48
mdevctl define -a -u $UUID1
mdevctl define -a -u $UUID2
The virtual machine configuration.
- Chipset: Q35
- Firmware: UEFI x86_64: /usr/share/edk2-ovmf/x64/OVMF_CODE.fd
- Hypervisor KVM
- Spice Server
- Video QXL
- MDEV
- shmem looking-glass
The size is determined like this.
<shmem name='looking-glass'>
<model type='ivshmem-plain'/>
<size unit='M'>64</size>
</shmem>
Make a file /etc/tmpfiles.d/10-looking-glass.conf
#Type Path Mode UID GID Age Argument
f /dev/shm/looking-glass 0660 user kvm -
Where $UUID1
is the UUID of one of the mdev vGPU’s created earlier.
<hostdev mode="subsystem" type="mdev" managed="no" model="vfio-pci" display="on">
<source>
<address uuid="$UUID1"/>
</source>
</hostdev>
- Start VM. (May need to click run several times if error related to “vfio-<bus>” occurs.
- Run looking-glass client
export SDL_VIDEO_X11_VISUALID= looking-glass-client
Inside the VM.
- Windows 10
- virtio guest tools
- spice guest tools
- looking glass host
- virtio PCI RAM controller driver
- Nvidia GRID driver
- vGPU_LicenseBypass
- Scream audio driver
- Download components.
- Update driver in
Device Manager->System Devicess->PCI standard RAM Controller
to virtio PCI RAM controller driver. - Install virtio guest tools and spice guest tools.
- Install looking glass host by running
looking-glass-host-setup.exe
as administrator. - VM.
- Run Nvidia GRID driver installer.
- Reboot.
- In
Nvidia Console
set resolution. My monitor used 2560x1440. - Run vGPU_LicenseBypass.
- Install driver.
- Change sound source to Scream (WDM).
Since there is a lot of messing with the kernel and graphics drivers, it’s important to have a plan for recovering your host OS.
- Use timeshift or other auto-snapshotting of the host.
- Have a recovery drive available.