Proxmox HA rebooted all nodes because the watchdog and fencing were misconfigured. It’s not a kernel bug or bad hardware. It’s the HA stack doing exactly what you told it to do, badly.
What actually happened
HA works by having each node run a watchdog. If the watchdog doesn’t get kicked, the node assumes it’s hung and reboots itself. That’s the self-fencing part. If the cluster loses quorum, the nodes that are left can fence the others using watchdog or external devices. When every node reboots at once, it’s usually because the watchdog timer is too short, or fencing is set to a mode that reboots everything instead of just the dead node.
Check the watchdog first
Run this on each node:
ha-manager status
cat /etc/pve/ha/manager_status
Look for the watchdog line. If it says watchdog: hardware but you don’t have a real hardware watchdog, that’s your problem. The software watchdog (softdog) is easier and works fine for a homelab. Load it and set it as the default:
modprobe softdog
echo softdog > /etc/modules-load.d/softdog.conf
ha-manager set --watchdog softdog
Then check the timeout. The default is 60 seconds. If your nodes are under heavy I/O, that might not be enough. Bump it to 120:
ha-manager set --watchdog_timeout 120
Fencing: stop the reboot loop
Fencing is what the cluster does when it thinks a node is dead. By default, Proxmox uses watchdog fencing, which means it tells the dead node’s watchdog to fire. But if the watchdog is misconfigured, it can trigger on the wrong node. Check the fence mode:
ha-manager set --fence_mode watchdog
That’s the default. If you have a real fence device (like a PDU), use hardware mode instead. For a homelab, watchdog is fine as long as the watchdog itself is right.
Also check the HA resources. If you have a resource with max_restart set too high, and the service keeps failing, the node will reboot to try to recover it. Look at ha-manager status and see if any resource is in a loop. If so, set max_restart to something sane like 3.
One more thing: quorum
If you have an even number of nodes, you need to set quorum_votes correctly. With two nodes, one has to have two votes or you’ll lose quorum every time one node hiccups. That’s not a reboot, but it makes HA act weird. Check pvecm status and make sure quorum is stable.
This is the same kind of pain as the pve-ha-manager update stuck at processing triggers issue — the HA stack is finicky and the defaults are not your friend.
Fix the watchdog, check the fencing mode, and the mass reboots stop. It’s not magic, it’s just configuration.
