Technical

VM processes stall when rebooting adjacent node — fix it in a two-node Proxmox cluster

VM processes stall when rebooting adjacent node — fix it in a two-node Proxmox cluster

VM processes stall when rebooting an adjacent node in a two-node Proxmox cluster because of corosync quorum loss and fence delays. Here’s how to fix it without just waiting it out.

Blue plastic wires with white tips connected to server and provide access to information

Why it stalls

In a two-node setup, when one node reboots, the remaining node loses quorum. Corosync expects at least two votes, and with only one left, it freezes all HA-managed services. The VMs don’t actually crash—they just hang there, stuck, until the other node comes back or the fence kicks in. And the default fence delay is way too damn long.

Test it yourself

Don’t guess. Reboot one node and watch what happens on the other. Run this on the surviving node:

watch -n 1 'ha-manager status; pvecm status'

You’ll see the VMs go into a “fence” state and just sit there. That’s the stall. If you’re not using HA, you might still see weirdness if the VMs have any cluster dependencies—like shared storage that briefly disappears.

The quick fix: lower fence delay

The dumbest part is the default fence delay. It’s set to something like two minutes, which is an eternity when you’re staring at a frozen VM. Cut it down to 30 seconds or less. Edit /etc/pve/ha/fence.cfg on any node:

fence: default
	delay 30

Then restart the HA services:

systemctl restart pve-ha-lrm pve-ha-crm

Now when a node goes down, the fence triggers faster and the VMs get unfrozen sooner. Still not instant, but way less painful.

From below of long thin identical blue cables connected to small round electrical connectors

If you don’t need HA, disable it

Honestly, for a two-node homelab, HA is often more trouble than it’s worth. If you’re not running critical services that need automatic failover, just turn it off. Remove all resources from HA and stop the services:

systemctl stop pve-ha-lrm pve-ha-crm
systemctl disable pve-ha-lrm pve-ha-crm

No HA, no quorum headaches. Your VMs will still run fine, they just won’t try to fence or migrate when a node reboots. Simple.

Give the second node a second vote

Another option: give the remaining node two votes so it can maintain quorum alone. This is a bit hacky but works. Edit /etc/pve/corosync.conf and set quorum_votes: 2 for each node. Then restart corosync:

systemctl restart corosync

Now a single node can reach quorum by itself. The downside? If the other node comes back and they split-brain, you might have a mess. But for a quick test or a lab where you don’t care, it’s fine.

My take

This is one of those things that makes two-node clusters a pain in the ass. The “right” fix is a third node or a QDevice, but who has spare hardware just for quorum? I’d lower the fence delay and call it a day. If you really need HA, look into a QDevice on a Raspberry Pi—there’s a post on thin provisioning that might give you ideas for keeping it lightweight. But for most homelabs, just disable HA and move on.

If you’d rather pay for this

The uncomfortable part of this one is that the workaround is not a fix. A two-node cluster cannot establish quorum when one node goes away, because one vote out of two is not a majority. Everything above manages that; nothing above removes it.

The actual fix is a third vote, and it is cheap. A QDevice needs almost nothing — any always-on machine or a small VM at a VPS provider running the corosync qnetd daemon, which is single-digit dollars a month if you do not already have somewhere to put it. That is the entire purchase, and it converts a cluster that hangs on a reboot into one that does not.

A Proxmox subscription is the separate, adjacent buy: it is priced per socket per year and gets you the enterprise repository plus support. It will not give you quorum, and any vendor that implies otherwise is selling you the wrong thing.

Rule of thumb: if the cluster runs anything you would be annoyed to lose during a reboot, buy the third vote this week. It is the cheapest reliability you will ever purchase.

Nguyen

I write two things here: notes from a homelab that mostly runs Proxmox, and essays in Vietnamese about a country I left. The technical posts are whatever broke that week, written down so it costs you less time than it cost me.

All posts →

Leave a comment

Comments are reviewed before they appear. Your email is never published.