The ThinkPad T430 is not a remarkable laptop. It’s thick, bulky and built like a tank. I got mine in 2016 when the first university scholarship money dropped, and it’s still my backup laptop of choice.

Around 2017 I did something every reasonable poor computer science student would do: I got an eGPU adapter for it to play some games. I never ended up playing many games, but I loved tinkering with and testing this setup a lot.

These are my notes on the setup that I used to have. The notes used to be on Reddit, but after yet another user-hostile change¹ I deleted my account and all the content associated with it.

The specs

The ThinkPad T430 that I used had a CPU swap at one point, running an Intel i7-3820QM. This upgrade doubled the CPU performance on the laptop compared to the dual-core CPU that was in it before. This change was critical because it opened the door to running lots of modern games at reasonable performance.

The eGPU adapter I ran with was the EXP GDC Beast v8.4, using the ExpressCard34 connector. This adapter turned the ExpressCard34 slot into a PCIe 2.0 x1 slot. It’s not really hotpluggable, but using this connector instead of something like the internal Wi-Fi adapter slot made this setup much more portable. Done gaming? Shut down, disconnect the ExpressCard cable and you’re good to go!

This adapter is also a good example of connector reuse. The other end of the ExpressCard plug is an HDMI port. Video signals require good quality cables, and so does PCI Express.

The eGPU was once powered by a simple ATX PSU with the jumper wire set on the 24-pin connector, but eventually I bought a Dell DA-2 power supply. The adapter has a power connector designed specifically for that pinout and it made the setup a lot cleaner.

At one point I got a fully metal case for the eGPU. It made the setup a lot cleaner and the risk of the GPU tipping over while turned on was also substantially reduced.

I tested a few GPU-s:

• AMD Radeon HD 6870
• nVidia GTX 950
• nVidia GTX 1080
• AMD Radeon RX 480

The HD 6870 was an old card back in 2017, and it did not run great at all with this eGPU adapter. It was buggy and the performance sucked.

Loaner GPU-s, such as the RX 480 and GTX 1080 were much better experiences, as long as you output the video signal to an external display right from the GPU.

The nVidia GTX 950 was the card I ran with for the longest time in this setup. It worked well with this setup and had decent gaming performance.

eGPU tech tips

You will want to use an eGPU adapter with an external monitor for the best performance. The PCIe bandwidth is very limited in a setup like this. Using the internal display of the laptop will mean that the final rendered image will have to be sent back from the eGPU to the CPU. 1x PCIe 2.0 lane is about 500 MB/s of bandwidth. Sending 60+ frames in the native resolution (1600x900) will eat it all up. The higher the resolution on the internal display, the fewer frames you’ll get.

I tested out these two scenarios in CSGO. The internal display yielded about 61 FPS while an external display would result in 100-180+ FPS.

The PCIe link bandwidth was not actually that noticeable in a lot of games, such as GTA V, Rocket League and Dirt 3.

In some games, such as Dirt Rally (2015), it did not matter what settings you went with, something about the game made it run poorly no matter what graphics settings you went with. In cases like that you might as well cap the framerate to something tolerable like 30 FPS and crank the graphics settings up.

VFIO on a laptop???

This setup was also around the time when I first learned about the magical world of VFIO and GPU passthrough.

The logical continuation of that discovery was to try it out on my laptop, and to my surprise it actually worked. I could pass the eGPU to a VM and play GTA V on it!

This setup ran on Antergos OS, a flavor of Arch Linux. It’s been so long since then that the flavor of Arch Linux is now discontinued.²

The laptop was ideal for this purpose as well. You have the integrated GPU, display, keyboard and mouse for the host system, and you can pass USB devices and the eGPU to the VM.

There were the usual VFIO tricks that I had to try out to avoid the dreaded error code 43 issue, but after that it was smooth sailing.

The performance was very similar to what I saw on the host system.

Smartphone cameras have come a long way since then.

ExpressCard eGPU-s in 2024?

What got me started with writing down my notes were some YouTube videos that were made on this topic recently.

A pimped-out ThinkPad T430 can still be a perfectly serviceable computer in 2024, and an eGPU will completely sidestep the problem of the weak integrated GPU.

The T430 is not powerful enough to natively run a 3440x1440 ultrawide monitor at 60 Hz. It is even struggling to do it at 30 Hz. An eGPU adapter with a modern low-power GPU, such as the AMD Radeon RX 6400, could solve this issue nicely and give the ThinkPad T430 a new lease of life.

I’d be lying if I wasn’t considering remaking this setup with everything I’ve learned 6 years later. GPU pricing sucks, though.³ They still sell these eGPU kits in various forms, however the price has doubled or even tripled over the years.

If you’re interested in building a similar setup, then do check out egpu.io. With an M.2 or Thunderbolt connector you can build a much more capable eGPU setup.

If you can get a PCI Express signal out of a computer, then you can probably rig a GPU up to it. Unless you’re Jeff Geerling, in which case you might need to recompile the kernel a few times.

Not too long ago, I had the goal of consolidating all my computing needs onto a single desktop PC. That meant moving from an mITX-based build to something that had a bit more expandability. Since I already had a Fractal Meshify C Mini, I decided to go for an mATX motherboard. And that’s how I ended up with an ASRock X570M Pro4.

mATX: a dying breed?

One of the requirements for the build was that it had to support IOMMU well so that a VFIO setup would be possible. While AMD B-series motherboards may work just fine in such scenarios, the X-series boards allegedly have better IOMMU groups and thus better compatibility with VFIO.

Once I started the search for the new motherboard, I was surprised to see that locally there was only one mainstream option available. There were plenty of B450 and B550 boards available, but only a single X570 board in this form factor.

I checked the board out, saw that it had plenty of NVMe and SATA storage options, and went for it.

PCIe connectivity

When you have a bunch of PCIe devices to connect, the way those lanes are connected start to matter.

Based on my own testing, I’ve found the following:

• the first M.2 slot (under the heatsink) is connected to the CPU
• the first PCIe x16 slot is also connected to the CPU
• the rest of PCIe connectivity goes through the chipset
• most (if not all) SATA ports seem to go through the chipset as well

This matters, because anything connected to the CPU via the chipset shares the bandwidth of a limited x4 link. This was apparent when running two NVMe drives and noticing that one of them was much slower when I also put load on SATA SSD-s.

GPU-s can still work fine when connected via the chipset, but there’s definitely a performance penalty involved. While I don’t have the exact numbers available at this time, I do remember a CSGO benchmarking map showing at least a double-digit framerate difference compared to the full x16 slot.

I also gave PCIe bifurcation a go. I borrowed an ASUS PCIe adapter card that can support up to 4 NVMe SSD-s, hoping that I could set up a crazy storage setup. The bifurcation works, but for the CPU I used (AMD Ryzen 7 5700G) I was only provided the option of 2x4 split, which meant that at most I could run two SSD-s in the riser card at a time. A non-APU Ryzen CPU is likely needed to take full advantage of the bifurcation support.

For VFIO, this board is fine. Anything in PCIe slots, including the chipset ones, I was able to pass through to the VM. USB and SATA ports were a different story, I wasn’t able to pass through a single controller, the VM would not start up properly.

If you have an use case where you need reliable and fast PCIe connectivity, then I’d recommend another platform that has lots of real PCIe connectivity, such as the AMD Threadripper/EPYC platform. This board exhibited some odd behaviour when trying to utilize all the PCIe slots, or when trying to use the ASUS Hyper M.2 card in bifurcation mode while also having the GPU connected at the bottom PCIe x16 slot.

Chipset fan

One aspect of X570 boards that stuck out to most people on release was the inclusion of a chipset fan. Not too long after, people started to report that those fans are very loud and low quality, resulting in them failing easily.

For this particular board, ASRock provides an option in UEFI settings that allows you to run this chipset fan semi-passively. It will spin up once on boot and sit idle the rest of the time, unless the temperatures are high enough. I was not able to get that fan to spin even with a full GPU load and all SSD-s running heavy read operations at the same time.

If you have PWM fans, it’s also possible to manually control these in Linux with the use of fancontrol. For me, it just took a bit of trial and error to figure out the mapping between the software fan controls and the actual fan in the PC, after that I could fine-tune them to run as quietly as possible. Yes, even the chipset fan was controllable using the same method.

AM4 socket long term support

AMD introduced the AM4 socket back in 2016 and promised to support it until 2020. I applaud that effort and am happy that I was able to run the latest AMD Ryzen 5000 series APU on a B450 board. If you happen to have a newer CPU that requires a UEFI upgrade for it to be supported, then you will be in a world of hurt.

I found myself in a situation where I had a motherboard that didn’t have the newest UEFI version from the factory, and a CPU that was just a bit too new for it to work. The board supported AMD Ryzen 5000 series CPU-s, not 5000 G-series CPU-s. I also didn’t have an older CPU available that I could use to perform this upgrade on my own. Luckily I was able to find a loaner 5000 series CPU, perform the upgrade and finally assemble the machine.

At least ASRock puts a sticker with the UEFI version on the chip itself so that you can check the CPU support once you physically have the motherboard, but it’s pretty much useless when buying this board online. There’s no indication as to which version it will ship with, the best you can do is to look for some Reddit threads where others have shared their experiences with the board and the UEFI version it shipped with from the factory.

Pro tip for CPU and motherboard manufacturers: if you’re going for long-term support on your platform, please include a CPU-less UEFI upgrading solution. Some high-end motherboards already ship with that type of functionality, but I believe it should be made mandatory for all boards. Avoiding headaches like that will be worth the extra cost.

Verdict

It’s fine, looks good and gets the job done. Can’t recommend it for someone who needs a lot of performance and PCIe lanes, but for hobbyist workloads it will probably be just fine. UEFI flashing situation really soured the experience though.

I’ve been running my all-in-one PC for a while now. It was my desktop, my NAS and my gaming PC. However, during the next couple of months I kept hitting small bumps along the road. Most of these were quite straightforward to fix, but there is one that finally convinced me to go back to a simpler setup.

iSCSI

As mentioned in one of my previous posts, I had set up the storage on my gaming VM over iSCSI. If you don’t know what iSCSI is: think of it like a hard drive that you can hook up to your machine over the network. This setup had a couple of benefits: the amount allocated to the VM was flexible and I would always have a snapshot of my game data in case the Windows VM decided to nuke itself.

What I didn’t take into account was the configuration. tgt allows you to define the IP address of the client machine, which can be seen as an additional layer of security. My gaming VM has two virtual NIC-s, one pointing to the actual physical network, and another one to a virtual one. This resulted in the iSCSI connection being flaky on startup: the disk was not present in the VM on first boot, but was there after a reboot.

This issue was gone once I removed this restriction from the iSCSI target configuration on the Linux VM. My best guess is that the iSCSI stack in Windows might have tried to initiate a connection over the wrong NIC, depending on whichever one was brought up first.

After “solving” this issue, it had been smooth sailing.

GPU audio issues

When the Windows VM had been on for a while, the HDMI audio output would start to glitch out after a while, with the audio sounding slow, robotic and garbled. After doing some research, I found some resources that mentioned enabling MSI interrupts in Windows. The guide for this can be found over at Guru3D forums.

After tweaking the Windows registry settings and reminding myself that I do have snapshots to fall back to in case I mess up, I got the issue fixed.

GPU driver issues

Whenever there’s a new big and popular game released, NVIDIA usually releases an updated version of their GPU drivers. I try to keep my software up to date, so I usually install these updates when they come out. Just one problem: when I upgrade the GPU drivers in the VM, the screen will stay black. After some time passes, I have to force a reboot using virt-manager.

I have not looked into this issue yet as it only happens during driver updates, but it’s definitely an annoying thing to have.

UEFI and CSM

UEFI updates have a bad habit of resetting all the custom settings that you have made, and at least with the motherboards I have used, the settings cannot be restored if the backed up settings are from an older UEFI version. This means that I have to navigate the UEFI quite often to configure all the settings that I need to have in place for VFIO to work.

There is one setting that I didn’t expect to cause that much trouble though: CSM support. CSM (compatibity support module) is generally enabled by default and allows you to boot off of legacy operating systems if needed. I don’t have that requirement, so I went ahead and set my boot options to boot in UEFI mode only.

Reboot, start up Fedora, and boom, anything VFIO related is spewing errors like crazy.

Turns out that enabling UEFI only mode in UEFI settings somehow grabs the GPU and causes problems when you try to then pass the GPU to a VM. While in this mode, I noticed that the GPU was also shown in a separate menu in UEFI settings. Enabling CSM support fixed this issue for me.

VR and USB issues

With the situation in the world being less than optimal, I decided to spice up my gaming setup with the addition of a HTC Vive Cosmos in case in-door activities become much more popular suddenly.

First issue: I can’t get the damn thing working. My gaming VM setup has the inateck KT4006 passed through so that I can plug peripherals to the PC and make them available to the gaming VM with ease. After plugging everything in according to instructions and turning the link box on, I’m met with a barrage of USB connected-disconnected sound effects. 6 or 7 cycles later the sound stops, and with that do the rest of my USB devices.

I took the USB 3.0 card and tried it in an older machine that doesn’t have a fancy VFIO setup on it. Same issue. However, when connected to an USB 2.0 port, the setup shows a warning in VIVE Console, but at least it works! Since my PC literally has no USB 2.0 ports, I had to get a bit creative and plug an USB 2.0 extension cable between the PC and the USB 3.0 cable coming from the link box.

With the help of my friend I got confirmation that the issue is indeed with my USB ports. I also ordered a different USB PCIe card from inateck, this time the inateck KTU3FR-4P. I made the assumption that since the VIVE support page also recommends an inateck card, then they must be fit for purpose.

Nope. The second card apparently has the same chipset and the same issue.

What did end up working was a random VIA USB 3.0 PCIe card, which shows up as VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller on my machine. Just one issue: it has PCIe device reset issues. If I shut down the Windows VM and start it up again, I would sometimes get a bunch of IO_PAGE_FAULT errors in my kernel logs. No worries, guess I’ll just avoid rebooting a notoriously reboot-happy OS.

Latency: the straw that broke the camel’s back

If there’s one thing you don’t want to see in a VR gaming setup, it’s latency.

I’ve covered the tweaks I’ve made to my setup in a previous post. Turns out that those might not be enough. The CPU isolation using systemd works well in general, but not with kernel threads.

Whenever I did something IO heavy on an NVMe-based btrfs file system, it would introduce stutter to the gaming VM. Running a scrub operation on the filesystem was enough to put a huge load on the CPU, likely due to the raw speed that the NVMe SSD-s can provide. This results in the gaming VM experiencing stutter that ruins the whole experience.

Here’s an illustration.

Similar issues could be observed whenever I tried to run another VM that shares the same cores.

I looked into alternative solutions as well, and it does seem to be possible to also isolate those CPU cores from the kernel. They will probably work to some extent, but implementing them feels a bit hacky at the moment. I’m hoping that in the future these kinds of tweaks will be exposed as a simple checkbox in virt-manager itself.

At that point I had enough. If I’m not going to be able to use the machine for other purposes at the same time without experiencing all these issues and having to work around them, then I’ll just go back to a simpler setup. I’ll lose some of the benefits, but at least I don’t have to spend time debugging all these issues.

Conclusion

VFIO and virtualization are still interesting topics to learn about and try out, but for my use case they are just too limiting. I’d imagine that an use case that didn’t have such strict latency requirements would still be able to run just fine, unfortunately gaming just isn’t one of them.

This experience has been quite fun in general, but I’m calling quits, at least for now. The one machine that does it all turned out to be a jack of all trades, master of none.

As you might have read from my previous post on this topic, I have a pretty neat “cloud” gaming setup running. I have one powerful desktop PC, one virtual machine with a GPU attached to it, and client machines that can be used to stream games from. To keep things simple (and because my current ISP is unbelievably bad), I have so far used streaming only over my local network.

However, there were some pretty annoying issues with this setup:

• The AMD GPU encoder is bad. 720p streaming was the best I could do if I wanted reasonable performance.
• The Windows 10 VM would sometimes not be able to start the host encoder, and the GPU drivers reported issues. This was fixed with a couple of reboots to the VM and I suspect that this may be related to the notorious GPU reset issues that AMD cards are infamous for in the VFIO world, but I wasn’t interested in digging that deep into that topic.
• The stream would sometimes randomly crash or lag. This might have been down to the AMD GPU or Parsec itself.

I read about this topic for a while and found that the overwhelming number of threads related to this topic recommend an NVIDIA GPU and Moonlight for the best streaming experience. And so I went out to get one.

The GPU

It’s 2021. The world is on fire and GPU prices are insane. With that in mind, I tempered my expectations and went for something simple: a NVIDIA GTX 1060 6G by MSI. Simple, relatively modern for my games and the price was around the retail price 5 years ago (260 EUR), which is the best you can do in these lousy times.

But hey, it works! From the testing I did in a previous post, I knew that NVIDIA had finally stopped intentionally throwing error code 43 in their drivers, which made this option viable. Replacing the GPU was simple, I just had to change the PCI device ID-s according to the VFIO setup guide and changed the devices attached to the Windows 10 VM using virt-manager, and that was it!

Parsec: AMD vs NVIDIA

Since I was already using Parsec for my streaming purposes, I decided to do a quick test to see what the difference is between the two cards.

The previous GPU was an AMD Radeon RX 570 4GB by Sapphire. These two GPU-s are in the same performance class, which makes for a good comparison.

Here are some results taken in two quick testing sessions in Dirt Rally. Take these results with a spoonful of salt as these have not been conducted with any kind of standards. I just picked the same graphics preset, resolution and let it run with Parsec statistics view open.

H.264

AMD RX 570: 10ms

NVIDIA GTX 1060: 3ms

H.265

AMD RX 570: 8ms

NVIDIA GTX 1060: 3ms

The difference in the encode performance is huge, especially if you consider that the time between two frames is 16.6ms for a 60 FPS stream. Some of that budget is also taken up by network and decode latency in the client, so the shorter the encode time is, the better.

Parsec is fine, but I have had some trouble getting GPU-accelerated decode working on my Linux clients, and it also lacks an iOS client. While the latter isn’t a deal-breaker, the first one definitely is.

Moonlight: an open-source gem

Moonlight is an alternative to Parsec. It implements the NVIDIA GameStream protocol and has clients for most common platforms. My testing has been performed on Linux clients running Fedora 34, a Windows 10 laptop (ThinkPad X230), iPhone SE 2020 and an old Google Nexus 5. I also made an attempt to run the Moonlight client on a Raspberry Pi 1 B+, but it seems that it is simply too old for the Moonlight client, resulting in an error on startup.

To set up Moonlight, I opened up the client and tried the automatic detection over the network. Unfortunately that did not yield any results, so I input the IP address of the Windows VM manually. That worked well. This initiated a pairing process that I had to continue using Parsec, because you need to input a code on the host machine. After that, I had to quit Parsec and start streaming using Moonlight.

OK, not quite yet. Moonlight is able to detect the games installed on your system, but for a remote desktop experience, you have to do a small tweak for it to be equivalent to the experience you get from Parsec.

One thing that I noticed at the start was that the image quality on the desktop isn’t as good as on Parsec. However, it more than makes up for it in-game. Moonlight just works. There are plenty of knobs you can turn as well, including configuring the image quality by setting the resolution, framerate and network limits.

What makes Moonlight more suitable for me is that the Linux client packaged as a Flatpak is able to use the GPU for hardware-accelerated decode. The experience is great on a Linux machine and it just works, which is not something you can always expect to happen. Even an older XBOX ONE controller connected over Bluetooth works just fine over Moonlight!

The only issues that I had with Moonlight were later determined to be caused by networking issues. This shows up well on the integrated statistics that Moonlight provides using the Ctrl + Alt + Shift + S key combination under the Network jitter field. After stopping some services on my network, the issues went away.

The setup: one month later

After a month of use, I have made some adjustments to the setup. I used to keep the VM running so that I could just jump in and start playing whenever I liked it, but due to a tweak I made in the name of improving gaming performance, half the CPU cores on my system were not available for other purposes, such as suffering through IntelliJ indexing the project again. I decided to only power on the VM if I actually was about to take the time to play it. I haven’t yet figured out the best way to make this more convenient, though. I did make the “Calculator” key on my keyboard start/stop the VM on demand, but then GNOME started ignoring the shortcut I had setup and opened the calculator anyway, so that was a short-lived tweak.

A bigger change that I made to the setup was to store all my games on my ZFS pool that lives in another VM. For that, I setup an iSCSI target using this guide as a starting point . I used the tgt package instead. That has worked out surprisingly well, but the iSCSI target does not want to properly connect in the Windows VM on the first boot, which is mildly annoying. I soon discovered that spinning rust just does not cut it and got more SSD-based storage, otherwise other processes on the NAS could wreak havoc on the gaming experience in games like GTA V. Those issues were manifesting as long loading times and objects popping in too late, resulting in half the roads and building being missing while driving around the map. The iSCSI target solution also comes with the added bonus of having a proper backup of all my games. If Windows decides to blow itself up again and I don’t have a backup of the VM ready, then I could just reinstall it and not worry about downloading and installing all the games again.

Verdict

Overall, I’m very happy about the setup. I don’t use it all too often, but occasionally launching art of rally or BeamNG.drive on any random device I own is great. I do have some concerns about the future of this setup, especially with Windows 11 having more strict system requirements and newer games demanding more performance. For the time being, this works and I intend to keep it that way.

The one box that does it all: so far so good!

This is a follow-up to my previous post where I covered the VFIO setup in general. For many people that would have been good enough, but my goal with this setup was to have a powerful gaming setup that I could access from my living room PC with 20 meters of Ethernet cables between the two.

Cloud gaming?

The concept of cloud gaming has become more popular lately, with Microsoft, Google, NVIDIA and others offering services that allow you to stream games to your device. These setups usually include a powerful server in a datacenter rendering the game and sending the compressed video stream to your device. This works surprisingly well, but due to high cost, poor internet connectivity in many places in the world and licensing issues , this setup might not work well for everyone.

However, if you already own a moderately powerful gaming PC and would like to use it to play games you already own over the network, then there are plenty of solutions that can get you there.

Here are some options:

• Steam remote play: works for games launched via Steam.
• Parsec: able to stream the whole desktop, including games.
• Moonlight: same, but only usable on hosts with NVIDIA GPU-s due to the implementation relying on NVIDIA’s GameStream protocol.

In the past, I’ve had OK results with Steam remote play, but the limiting factor has been the reliance on Steam. I have bought a couple of games from GOG as well, which I could probably also play remotely by launching them via Steam, I just don’t want to go through the hassle.

For this setup I’ve opted to go with Parsec. It’s not perfect, but it’s still good enough for our purposes. The GPU we’re using (AMD Radeon RX 570) also limits our options because it won’t work with Moonlight.

The installation of Parsec is pretty straightforward: just install it on your host and client machines. In my case the client machine is a Lenovo ThinkPad X230 that was brought back from the dead . It’s not powerful, but at least it can do H.264 hardware decoding and it uses only 12W of power when idle, making it a perfect candidate for testing this out. It’s also worth mentioning that both machines are connected on the local network using an Ethernet connection to avoid Wi-Fi becoming a bottleneck.

VFIO, gaming and you

What turned out to be a bigger hurdle was the performance of games inside the VM. After getting everything running with the VFIO setup, I didn’t really spend any time trying to optimize the setup for the smoothest experience. When trying to actually run this setup, I ran into quite a few problems with performance which mainly manifested as stutters and unusually low framerates. Turns out that gaming has stricter latency requirements than other server workloads.

Thanks to the great guide in Arch Wiki, I could get a lot of ideas on what to try out to improve the performance of this gaming virtual machine.

Before doing anything, I ran some benchmarks to get a sense of what performance levels I’m dealing with. To test out each individual change, I used GTA V as the gaming benchmark since it was pretty good at pointing out any performance issues.

Here’s a list of things that I ended up doing:

• setting the CPU model to host-passthrough in virt-manager
• isolating CPU-s dynamically whenever the VM starts and pinning those CPU-s to the VM to avoid the host OS and other VM-s from using those cores
• setting the CPU governor to performance on isolated cores to rule out issues with the CPU not clocking high enough and switching between idle/load power modes: echo performance > /sys/devices/system/cpu/cpu[4-7]/cpufreq/scaling_governor
• enabling static hugepages to rule out issues stemming from poor memory access speeds
• disabling SMT in UEFI settings to get rid of one additional variable
• getting a 64 GB DDR4-3600 memory kit to allocate more RAM to the VM (16 GB) and leave enough for the host machine and other services

The golden rule of troubleshooting is to change one variable at a time and comparing results. Not every change that I made ended up being a positive one. For example, when configuring the VM to use 4 cores and 8 threads instead of plain 4 cores, I saw the framerates in GTA V drop 50%. I assume that the VM treated the “SMT cores” as real ones, causing the Windows scheduler to make incorrect decisions.

After playing around with this setup and accumulating these tweaks, I managed to get rid of most of the issues that bothered me during gaming, resulting in a much smoother experience. It finally felt like the gaming VM behaved like a machine with a 4 core CPU, 16 GB of RAM and an AMD RX 570 inside it.

At this point I’d consider the setup to be fantastic for someone who wants to play games with a VFIO setup. Since I was using this setup over the network with Parsec, I soon ran into more issues, but this time with the GPU.

GPU issues

Parsec and other similar solutions rely on the host GPU to encode the image into a video stream using the encoder present on the GPU die itself. You might see this being referred to as hardware encoding in settings. What I didn’t know before trying this setup is that AMD GPU encoders have a reputation for just being plain bad:

If the host has an AMD card, AMD is usually known for having worse encode than NVIDIA and even Intel. You should be fine at low resolutions. If all guests support H.265 and have it turned on, you may see better performance than with it off

And this shows. When trying to stream at 1080p, the result was a stuttery and inconsistent mess. Imagine trying to play at around 30fps with the frame timings graph resembling a heart rate monitor, that’s what it felt like. At 720p, the experience was so much smoother. Yes, the image quality suffers because of that, but at least it was mostly playable.

I decided to also try out if changing the codec from H.264 to H.265 has a significant impact. I did a quick test in Dirt Rally due to it having a benchmark loop mode and checked the statistics that Parsec shows. With H.264, I saw encode latency of about 10ms. With H.265, this latency was now at 8ms. Still not great, but it is technically a ~20% improvement. The downside of this for my setup is that the client laptop simply does not support hardware accelerated H.265 decode, which is something that Intel integrated graphics received support for in 7th gen CPU-s.

The GPU seems to also cause some trouble for Parsec, as it would fail to connect occasionally and reports a host encoder issue. This is usually overcome by rebooting the VM, which can get quite annoying after a couple of times. At other times, Parsec sometimes just froze during gameplay and caused the client to freeze at 100% CPU usage. Not what I’d call smooth sailing.

Another issue that I noticed by using MSI Afterburner is that the framerate was still not that stable in some games, such as GTA V. After all the fixes and tweaks, there were still small stutters, even with vsync enabled. I decided to look over AMD Radeon settings to see if a driver feature had an unintended side effect. I had picked the Gaming preset because that was what I was using this GPU for, but decided to switch to the Standard mode instead. And just like that, the stutters were gone! I suspect that the Radeon Anti-Lag feature might have been the cause to this, as that was one of the main settings that was disabled after switching to the Standard settings preset.

Is it worth it?

I like a technical challenge and going through all the attempts to get more and more performance out of this setup was interesting for me. However, it might not be the same way for everyone else. If you still feel like you want to go through this and learn something along the way, then feel free to use this as a guide on what you can try. Make sure to also read up on experiences that others have had with these tweaks and always measure your results to see if they had the intended effect.

For those who just want to play games and not worry too much about getting the expected performance out of your machine, I’d still recommend going with building a separate gaming PC.

If I had to build a machine specifically for this type of workload, then I’d make these changes:

• Replace the CPU with a non-APU model, such as the Ryzen 9 5950X. Due to the physical core layout you could assign one CPU core complex to the gaming VM and leave everything else to the host. These CPU-s also have a lot of L3 cache, which should help in workloads that require low latency, such as games.
• If you’re going for a streaming setup, then I’d try it first with an NVIDIA GPU. The error code 43 issues are gone now, making it a viable option with solutions like Moonlight.
• More SATA or M.2 slots on the system. This makes passing through storage devices so much easier.

Future plans

I’m not planning on stopping this adventure just yet. I’ve recently looked around for an NVIDIA GPU to do a comparison against the AMD RX 570 and picked out an NVIDIA GTX 1060, which is from the same era and performance bracket. Should make for an interesting comparison in both Parsec vs Parsec and Parsec vs Moonlight scenarios.

I’m also hoping to eventually move to a more modern client PC that can support H.265 decoding and output at higher resolutions. Yes, 4K at 60Hz is demanding for a laptop from 2012. Given that the laptop is an older ThinkPad, I expect that to happen after 2025.

Regarding the storage setup, I’ve got input from a friend saying that I might want to try setting up a Samba share on my “NAS” VM and host my Steam library and other game files on that. After all, the virtual LAN has managed hit 2-3 Gbit/s in my testing and with L2ARC being persistent in ZFS 2.0, I might be able to take advantage of that as well ( assuming that I even need L2ARC, ARC efficiency is pretty good!). Currently I just have a Syncthing sync set up between the “NAS” VM and the gaming VM so that there exists at least a basic backup of all my games.

Introduction

Before we jump into all the nitty-gritty details, I’d like to go over what we are dealing with here since these topics may be unfamiliar to you. VFIO is quite a niche topic and not everyone knows about it.

GPU passthrough: the process of allowing a VM (virtual machine) to use a dedicated GPU. This allows you to run GPU-heavy workloads inside the VM, such as gaming or anything that benefits from GPU compute power.

VFIO: the framework that allows us to perform this operation.

IOMMU: hardware feature that supports all of this.

virt-manager: GUI application that I use to manage VM-s.

Reasons why you may want to go through all the hassle and use this approach:

• you’re on Linux but want to play games occasionally without rebooting into Windows
• you want to do all your work, gaming and server workloads on one machine
• you simply don’t want to build a dedicated PC for playing games and would like to utilize the existing resources on your main machine
• you want more control over the Windows installation because you simply don’t trust Microsoft
• you want to have more control over the Windows installation and the ability to revert the installation to an earlier point in time using a filesystem that supports snapshots (BTRFS, ZFS)
• you have multiple separate VM-s for different purposes, but you would like to use one single GPU for all of them

One thing to keep in mind is that the hardware you use matters a lot here as that provides the foundation for getting this solution working. Here are some of the requirements you should know about:

• The GPU you want to pass through has to support UEFI. GPU-s from the past 5 years should be good on that, but if you have a really old GPU that you want to use for testing, then that might not work out that well.
• You need to have a good-enough CPU and plenty of RAM. After all, you’re essentially running a full PC within another PC.
• Your motherboard has to support IOMMU and have IOMMU groups that allow you to isolate the GPU and only pass the GPU to the VM. Level1Techs does quite a few motherboard reviews and Wendell goes over the IOMMU groups and suitability for VFIO in them. It also helps if you look around the internet for enthusiasts that have already bought the same motherboard, they may have posted information about the groups. Here’s a handy script that allows you to check your IOMMU groups.
• Some games may ban you if they detect that you are running the game inside a VM. It’s great that they’re trying to do something against cheaters, but unfortunately VFIO users get unfairly treated here. Please do some research beforehand if you play competitive multiplayer games a lot!

I hope that this introduction helped you understand the basics. Now, let’s just jump right into it.

Setup

We’re going to test this setup on hardware that I have covered previously. What makes this setup a bit special is that we’re using an AMD Ryzen 7 5700G APU on an mITX motherboard and a single dedicated GPU, allowing us to do big things in a small package. Yes, there are setups where you can do VFIO with a single GPU and pass it between the host OS and the VM, but that setup might be a bit tricky to use.

The OS is Fedora 34. To get this working, I’ve used various resources:

• beginners guide on Level1Techs forums
• Fedora-specific guide by Wendell from Level1Techs
• this brilliant Arch Wiki guide that covers most of what you need to know about setting this up. It’s not only useful for Arch, a lot of what is written here applies for any distro.

Testing

I decided to do the initial testing with the Nvidia GT 710. It’s slow, buggy on Linux on Nouveau open source drivers, and I had it available. I also recently heard that Nvidia stopped being hostile towards their customers in one aspect by allowing GeForce GPU-s to be used in a Windows 10 VM without having to resort to workarounds.

The testing itself was relatively straightforward. The only issues I had were PEBCAK issues, possibly related to me performing this testing after work. The main issues I ran were either small typos in dracut configuration or the wrong device ID-s being added to kernel boot parameters. Once I discovered and fixed those, it was all smooth sailing.

I’m not going to rewrite a full guide here, please refer to the previously linked resources if you need more details than that. Those resources are also much more likely to receive updates about new finds and features.

The steps I took were roughly these:

• Make sure that IOMMU was enabled in UEFI settings. It’s probably set to auto by default, make sure to set it to enabled.
• Install the GPU that I want to pass through and connect it to a monitor.
• Install virtualization packages on Fedora: sudo dnf install @virtualization
• Enable IOMMU and preload the VFIO kernel module by adding amd_iommu=on rd.driver.pre=vfio-pci to Linux kernel parameters.
  • I use GRUB2, these parameters live in /etc/sysconfig/grub.
  • To apply these changes, you have to regenerate the GRUB configuration. Under Fedora, this is done using grub2-mkconfig -o /etc/grub2-efi.cfg
• Get the device ID-s for the GPU and the related audio device that you’re planning on passing through
  • To see the devices and ID-s, run lspci -nnk. The ID-s look something like 1002:aaf0.
• Bind the GPU to vfio-pci to avoid the GPU driver from taking control of the GPU, otherwise you cannot pass it to the VM.
  • I opted to go for the simple approach and added the device ID-s to kernel parameters: vfio-pci.ids=1002:67df,1002:aaf0
  • Since I went with this approach, then I have to regenerate GRUB configuration again.
• Make sure that the initramfs loads the necessary vfio drivers early in the boot.
  • With Fedora 34, this means creating a file /etc/dracut.conf.d/10-vfio.conf with contents add_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd ".
  • Make sure that you don’t have any typos in that.
  • Regenerate the initramfs: dracut -f
• Reboot!
• Using virt-manager, select the VM you want to pass the GPU through and add two PCIe devices: the GPU and its associated audio device.
  • If you don’t have the VM set up yet, then proceed with the normal installation without the GPU passed through yet. Make sure to create an UEFI VM, otherwise you might run into issues. This can be configured in virt-manager _ Overview_ section by selecting the Q35 chipset and setting the firmware to OVMF_CODE.df.
  • If you have passed through the GPU, then make sure to remove the Display Spice device from the VM.
  • If you want to also control the VM, you need to also pass through your USB devices, such as a Logitech wireless receiver.
• Start the VM and cheer once you notice TianoCore appearing on the monitor that’s connected to the passed through GPU!

Results: Nvidia GT 710

Initial testing with Nvidia GT 710 was successful. By successful, I mean that the GPU displayed an image and did not install GPU drivers automatically.

To overcome that last issue, I downloaded the latest official Nvidia GPU drivers and was good to go. The news were true, error code 43 was no longer an issue!

Things got a bit less exciting when I got reminded that this GPU is weak. Very weak. Regardless, I decided to demonstrate its computing prowess by downloading the art of rally demo. Side note: it’s a great game, go try it out!

The demo ran! Not well, but it still ran!

Satisfied with the results, I decided to go for gold and replace this GPU with the AMD RX 570 that I “borrowed” from this PC I built recently.

Results: AMD RX 570

After switching the GPU, I replaced the device ID-s with the new values and continued with the VFIO adventure. This time, the GPU drivers were automatically installed by Windows and everything just worked. That’s not supposed to happen, at least in my experience something always goes wrong. Always.

I installed the latest drivers from AMD-s website and continued with testing.

Furmark? Runs as expected.

GTA IV? A stuttery mess, so yeah, runs as expected.

At this point the VM was assigned 4 CPU cores and I had performed no CPU pinning or optimizations of any kind, so the results were pretty good.

Storage

Because I built this setup on my current workstation/server machine, the storage situation is a bit tricky. The other VM that runs all the services has full access to the two 12TB hard drives and I wasn’t interested in setting up a networked storage setup. The only free spots I had were as follows:

• 120GB partition from the NVMe SSD. Good enough for storing the Windows 10 system files.
• 2x 250GB partitions on the Samsung SATA 1TB SSD-s. Just enough to hold my recently played games, but not much more. Running in striped configuration under Windows.

I later opted to expand the game storage partitions to 375GB, which meant that I had to get rid of the extra overprovisioning space that I had left aside. This setup is fine, but I’m losing out on some of the benefits that come with a virtualized Windows setup.

With regards to passing this storage to the VM, I had two options:

• keep using the SATA virtual disk: works out of the box, but might not perform as well.
• use virtio: you need to manually load and install the drivers for Windows to recognize these disks, but allegedly this has better performance than the SATA implementation.

I started off with SATA and did a comparison against virtio using CrystalDiskMark.

virtio did come ahead in these comparisons, but I would probably have been fine with the SATA performance as well.

Conclusion

One of the main reasons I do these kinds of setups is the technical challenge. It sounds weird, but sometimes I am happier after having completed something technical and novel compared to actually using the setup.

Previously I’ve had mixed results with GPU passthrough, with ThinkPad T430 and eGPU somehow managing to run GTA V in a VM , while in another testing session I ran into issues which turned out to be related to the CPU being faulty (lots of PCIe errors).

I do have more practical future plans with this. The plan is to use this VM as a gaming VM that I can stream games from using Parsec to any device that I wish. One idea is to use the Nvidia Shield TV as a low-power box that is capable of performing streaming. Alternatively, I could also get a super tiny form factor Dell/Lenovo/HP PC that has plenty of power to drive a 4K display and uses a reasonable amount of power. More on that in a future post.

Regarding storage I foresee an upgrade coming soon, either to the drives themselves or the whole setup since ATX motherboards support more SATA ports and expansion cards. If you also pair that up with a case like Masterbox Q500L with a good power supply placement, then you’d still have a relatively small setup.