Freezing Linux VMs and sporadically connection issues regarding a node: diagnose from IOMMU to ZFS ARC

Freezing Linux VMs and sporadically connection issues regarding a node are almost never random. It’s IOMMU, ZFS ARC, or a NIC driver — in that order. Here’s how to check each without guessing.

Start with IOMMU grouping

If a VM freezes solid and the host is fine, it’s often a PCIe device passed through that’s sharing an IOMMU group with something else. The VM grabs it, the host loses it, and everything locks. Check the groups:

for d in /sys/kernel/iommu_groups/*/devices/*; do
  echo "IOMMU group $(basename $(dirname $(dirname $d))): $(basename $d)"
done | sort -V

If your passed-through NIC or GPU sits in a group with anything else, that’s your problem. You can try the ACS override patch, but honestly, on a homelab I’d just move the card to another slot or drop the passthrough. Less to break later.

ZFS ARC eating all the RAM

Linux VMs freezing while the host seems okay? Check memory pressure. ZFS ARC will happily consume every byte you give it, and when a VM needs memory, the host starts swapping. That looks like a frozen VM and sporadic node connection drops because the host is thrashing.

arc_summary | grep -E "size|memory_throttle"
free -h

If ARC is at max and swap is active, limit it in /etc/modprobe.d/zfs.conf:

options zfs zfs_arc_max=8589934592

That’s 8GB. Adjust to your box, then update-initramfs -u and reboot. I’d also set a swapiness of 10 so the kernel doesn’t swap out VM pages too eagerly.

NIC driver flakiness

Sporadic connection issues to the node itself — not the VMs — are usually the NIC driver. Realtek and some older Intel drivers drop packets under load. Check dmesg for resets:

dmesg | grep -iE "link is down|link is up|reset|NIC"

If you see repeated link flaps, try a different driver version or just throw in an Intel I350. They’re cheap on eBay and rock solid. Also check ASPM settings — power management on the PCIe bus can cause the same symptoms.

cgroup limits strangling the VM

If you’ve set CPU or memory limits in the VM config, a burst of I/O can push the VM over and the kernel starts killing processes. The VM looks frozen but it’s actually OOM. Check journalctl -k inside the VM for oom-killer messages. If you see them, raise the limit or remove it. Proxmox defaults are usually fine, but if you tweaked something, undo it.

Quick checklist

• IOMMU groups — any passthrough sharing a group?
• ZFS ARC — is it maxed and swapping?
• NIC driver — link flaps in dmesg?
• cgroup limits — OOM killer inside the VM?

That covers 90% of what I’ve seen. If it’s still freezing after that, check I/O pressure stalls — that’s a whole other rabbit hole.