Technical

Extremely high IO delay / IO Pressure Stall: isolate LXC limits vs TrueNAS NFS saturation

Extremely high IO delay / IO Pressure Stall: isolate LXC limits vs TrueNAS NFS saturation

Extremely high IO delay / IO Pressure Stall on Proxmox is almost always either an LXC I/O limit or the TrueNAS/NFS backend choking. Here’s how to tell which one without guessing.

Detailed view of Ethernet and VGA ports on a server highlighting connectivity features.

Check PSI first

Pressure Stall Information tells you if the host is actually hurting. Run this on the Proxmox node:

cat /proc/pressure/io

You’ll see three numbers: some avg10=, avg60=, avg300=. If avg10 is above 40-50%, the host is stalling on I/O. If it’s low but your container is slow, the limit is inside the container.

Is it an LXC limit?

Check the container’s config for I/O limits:

grep -E '^(lxc.cgroup2.io|limits)' /etc/pve/lxc/CTID.conf

If you see something like lxc.cgroup2.io.max with a tiny number, that’s your culprit. Remove it or raise it. The default is unlimited, so if it’s there, someone set it.

System with various wires managing access to centralized resource of server in data center

Is it the NFS/TrueNAS backend?

If the host PSI is high and there are no LXC limits, look at the storage. On the Proxmox node, run:

iostat -x 1

Watch the %util column for the disk backing your NFS mount. If it’s pegged at 100% and await is high, the storage is saturated. Then check TrueNAS: is the pool healthy? Are you using SMR drives? Is sync writes enabled on the dataset? NFS with sync=always on a slow pool will absolutely tank performance.

I’d also check the network between Proxmox and TrueNAS. A bad cable or a switch port flapping can cause retransmits that look like I/O stalls. iftop or nload on both ends will show if you’re getting packet loss.

My take

Most of the time it’s a stupid LXC limit someone set and forgot. Second most common is sync writes on NFS. Fix those before you start blaming the hardware. And if you’re running unprivileged containers with large UIDs, check this post on UID/GID mapping—it can cause weird I/O behavior too.

If you want to go deeper on PSI and ZFS hangs, I wrote a whole walkthrough on IO Pressure Stall that covers the same ground with more detail.

Leave a comment

Comments are reviewed before they appear. Your email is never published.