Howto gpu passthrough

bios settings

make sure you have virtualization support on in your bios:

for intel enable:

intel virtualization technology
vt-d 

for amd enable:

virtualization
iommu

configure grub

add the following to /etc/default/grub:

for intel:

GRUB_CMDLINE_LINUX_DEFAULT="quiet ... intel_iommu=on"

for amd:

GRUB_CMDLINE_LINUX_DEFAULT="quiet ... amd_iommu=on"

• optional

If intel_iommu=on or amd_iommu=on works, you can try replacing them with intel_iommu=pt or amd_iommu=pt. The pt option only enables IOMMU for devices used in passthrough and will provide better host performance. However, the option may not be supported on all hardware. Revert to previous option if the pt option doesn't work for your host.

re-configure your grub:

grub-mkconfig -o /boot/grub/grub.cfg

isolating gpu

find iommu groups:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

find the device id of the gpu:

lspci -nn |grep -E NVIDIA

command output:

1:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070] [10de:2484] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)

edit /etc/modprobe.d/vfio.conf file and adding the following line with your ids:

options vfio-pci ids=10de:2484,10de:228b

if using the nvidia drivers add the following to /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="vfio-pci.ids=10de:2484,10de:228b" 

re-configure your grub:

grub-mkconfig -o /boot/grub/grub.cfg

edit /etc/mkinitcpio.conf and add the following:

MODULES="vfio vfio_iommu_type1 vfio_pci"

In the same file, also add modconf to the HOOKS line:

HOOKS="modconf"

backup your existing initramsfs image:

cp /boot/initramfs-linux.img /boot/default-initramfs-linux.img

rebuild initramfs:

mkinitcpio -g /boot/initramfs-linux.img -c /etc/mkinitcpio.conf

reboot your system:

reboot now

find your gpu and ensure that under “Kernel driver in use:” vfio-pci is displayed:

lspci -nnk

create bridge interface

create the bridge interface:

nmcli connection add type bridge autoconnect yes con-name br0 ifname br0

add device to bridge:

nmcli connection add type bridge-slave autoconnect yes con-name br0-eth0 ifname eth0 master br0

if you use static ip use the following command:

nmcli connection modify br0 ipv4.address 10.44.1.1/24 ipv4.method manual

if you use dhcp use the following command:

nmcli connection modify br0 ipv4.method auto

active interfaces:

nmcli connection up br0
nmcli connection up br0-eth0

share internet from wifi to windows

create script with iptables rules:

cat > /usr/local/bin/sharenetwlan << EOF
#!/bin/bash
echo "1" > /proc/sys/net/ipv4/ip_forward
iptables -I INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD  -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -I POSTROUTING -o wlan0 -j MASQUERADE
EOF

fix permissions:

chmod +x /usr/local/bin/sharenetwlan

add this script as systemd service:

cat > /etc/systemd/system/sharenetwlan.service << EOF
[Unit]
Description=Scream network pulse reciever
After=NetworkManager.service
Wants=NetworkManager.service

[Service]
Type=simple
ExecStart=/usr/local/bin/sharenetwlan

[Install]
WantedBy=default.target
EOF

reload systemd:

systemctl daemon-reload

start and enable service with systemd:

systemctl enable sharenetwlan.service
systemctl start sharenetwlan.service

configure libvirtd

install virt-manager and ovmf:

pacman -S libvirt virt-manager ovmf qemu qemu-audio-pa

edit /etc/libvirt/qemu.conf and add the following:

nvram = [
	"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"
]

add your username and libvirt group to /etc/libvirt/qemu.conf:

user = "linux"
group = "libvirt"

start and enable libvirtd service:

systemctl start libvirtd.service
systemctl enable libvirtd.service

start and enable virtlogd service:

systemctl start virtlogd.socket
systemctl enable virtlogd.socket

add yourself to the libvirt group:

gpasswd -a username libvirt

install windows

install virtio windows drivers:

yay -S virtio-win

looking glass

on windows download and install the looking glass host:

https://looking-glass.io/artifact/stable/host

add the following to your libvirt machine configuration inside the ‘devices’ section by running:

<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>32</size>
</shmem>

add virtio mouse:

<input type='mouse' bus='virtio'/>

add virtio keyboard:

<input type='keyboard' bus='virtio'/>

install looking glass client:

yay -S looking-glass

to lauch looking-glass client on linux host:

looking-glass -F 

you can install spice-tools to get better mouse performance on windows:

https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe

additional xml configurations

for better performance, enable the hyper-v enlightenments:

<hyperv>
  <relaxed state="on"/>
  <vapic state="on"/>
  <spinlocks state="on" retries="8191"/>
  <vendor_id state="on" value="0123456789ab"/>
  <vpindex state='on'/>
  <runtime state="on"/>        
  <synic state='on'/>
  <stimer state="on">
  <direct state="on"/>
  </stimer>
  <reset state="on"/>
  <frequencies state="on"/>
  <reenlightenment state="on"/>
  <tlbflush state="on"/>
  <ipi state="on"/>
  <evmcs state="off"/>
</hyperv>

disable memory ballooning:

<memballoon model="none"/>

cpu passthrough, cores and threads depends on your how many your cpu have:

<vcpu placement='static'>4</vcpu>

 <cpu mode='host-passthrough' check='none' migratable='on'>
   <topology sockets='1' dies='1' cores='2' threads='2'/>
   <cache mode='passthrough'/>
   <feature policy='require' name='topoext'/>
 </cpu>

modify clock configuration:

<clock offset="localtime">
   <timer name="rtc" tickpolicy="catchup"/>
   <timer name="pit" tickpolicy="delay"/>
   <timer name="hpet" present="no"/>
   <timer name="hypervclock" present="yes"/>
   <timer name="tsc" present="yes" mode="native"/>
 </clock>

for the above configuration your linux host must use tsc, check that using the following command:

cat /sys/devices/system/clocksource/*/current_clocksource

you should have the following result:

tsc

sound

use pulse audio to get sound, edit your vm and add the following:

|| make sure to change 1001 with your uid ||

<sound model='ich9'>
  <codec type='micro'/>
  <audio id='1'/>  
</sound>
<audio id='1' type='pulseaudio' serverName='/run/user/1001/pulse/native'>
  <input mixingEngine='no'/>
  <output mixingEngine='no'/>
</audio>

as root user copy your normal user pulse cookie to root profile:

mkdir -p  ~/.config/pulse
cp /home/linux/.config/pulse/cookie ~/.config/pulse/cookie

as root create the pulse audio client.conf:

cat > ~/.config/pulse/client.conf << EOF
default-server = unix:/run/user/1001/pulse/native
EOF

references

• https://github.com/vanities/GPU-Passthrough-Arch-Linux-to-Windows10
• https://www.heiko-sieger.info/creating-a-windows-10-vm-on-the-amd-ryzen-9-3900x-using-qemu-4-0-and-vga-passthrough
• https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Passing_VM_audio_to_host_via_PulseAudio