Technical

Rancher cluster node has disk pressure — fix it in a Proxmox homelab

Rancher cluster node has disk pressure — fix it in a Proxmox homelab

Rancher cluster node has disk pressure — and of course the usual Kubernetes eviction threshold advice doesn’t help. In a Proxmox homelab running Rancher on LXC containers, the real culprit is usually the container’s disk allocation and Rancher’s own image garbage collection settings. Here’s what actually works.

Detailed view of a server rack with a focus on technology and data storage.

Why you’re seeing disk pressure in the first place

Rancher’s UI will flag a node with disk pressure when the filesystem usage goes above 85% by default. In a native Kubernetes setup, you’d tweak kubelet eviction thresholds. But on Proxmox, your Rancher node is likely an LXC container with a fixed-size root disk. That disk fills up fast with container images and logs, and the node doesn’t know it’s a container — it just sees the disk getting full.

Check the actual disk usage

Jump into the Rancher node (the LXC container) and run df -h. You’ll probably see / at 90% or more. The culprit is almost always /var/lib/docker or /var/lib/containerd, depending on your Rancher version. Images pile up, especially if you’re pulling new ones for testing.

Fix #1: Resize the LXC container disk

This is the simplest fix if you have spare storage on the Proxmox host. In the Proxmox UI, go to your Rancher node container, Resources, and increase the root disk size. Then inside the container, grow the filesystem:

resize2fs /dev/mapper/pve-root

If you’re using ZFS, the command is different — but honestly, in a homelab, just give it more space and move on. You can also check out How to Resize a Linux Container in Proxmox for the full steps.

Two computer hard drives and a shiny disk displayed on a wooden surface.

Fix #2: Clean up Docker images properly

Rancher runs on Docker or containerd, and its own garbage collection isn’t aggressive enough. First, prune unused images:

docker image prune -a

That’ll free up a bunch of space. Then, set Rancher’s image cleanup to be more aggressive. In the Rancher UI, go to the cluster settings, edit as YAML, and add under rancherKubernetesEngineConfig:

services:
  kubelet:
    extra_args:
      image-gc-high-threshold: 70
      image-gc-low-threshold: 50

This tells kubelet to start cleaning images when disk usage hits 70% and stop when it’s down to 50%. The defaults are 85% and 80%, which is way too high for a small LXC disk.

Fix #3: Move Docker data to a separate volume

If you keep hitting the limit, attach a second virtual disk to the LXC container and move Docker’s data directory there. Mount it at /var/lib/docker and you’ll never have to worry about the root disk filling up again. It’s a bit more work, but it’s the permanent fix.

My take

Rancher’s disk pressure alert is a pain in the butt on Proxmox because it doesn’t account for the LXC layer. The quickest path is to resize the container disk and prune images. If you’re running a lot of clusters, just move Docker to its own volume and forget about it. Don’t overthink the eviction thresholds — they’re not the real problem here.

Leave a comment

Comments are reviewed before they appear. Your email is never published.