Technical

pve-ha-manager deadlock apt dist-upgrade: fix it with a systemd override and a script

pve-ha-manager deadlock apt dist-upgrade: fix it with a systemd override and a script

pve-ha-manager deadlock apt dist-upgrade is a real thing. The HA stack restarts services mid-update and the whole upgrade hangs. Here’s the fix: a systemd override plus a script that stops the cluster stack before apt runs.

A set of black power and data cables neatly arranged on a vivid yellow background.

What actually deadlocks

During apt dist-upgrade, the pve-ha-lrm and pve-ha-crm services get restarted as part of package updates. If the cluster is quorate and HA is active, the watchdog can fire while the services are down. The node fences itself, or the upgrade just hangs waiting on a lock that never releases. Either way, you’re staring at a frozen terminal.

The forum thread that proposed the fix is the only real documentation. No official bug report, no patch in the current repos. It’s been like this for a while, which is annoying.

The fix: stop HA before you upgrade

Don’t try to uninstall or mask the services. That breaks HA permanently. Instead, stop the cluster stack cleanly before the upgrade, then start it again after. The clean way is a systemd override that makes pve-ha-lrm and pve-ha-crm stop cleanly instead of getting killed mid-restart.

Create the override:

mkdir -p /etc/systemd/system/pve-ha-lrm.service.d
cat > /etc/systemd/system/pve-ha-lrm.service.d/override.conf <<EOF
[Service]
TimeoutStopSec=30
KillMode=mixed
EOF

mkdir -p /etc/systemd/system/pve-ha-crm.service.d
cat > /etc/systemd/system/pve-ha-crm.service.d/override.conf <<EOF
[Service]
TimeoutStopSec=30
KillMode=mixed
EOF

systemctl daemon-reload

Then run this before apt dist-upgrade:

systemctl stop pve-ha-lrm pve-ha-crm
apt dist-upgrade
systemctl start pve-ha-lrm pve-ha-crm

That’s it. The services stop cleanly, apt does its thing, and HA comes back up without fencing. I’d put this in a small script and run it every time, because forgetting once is all it takes.

Detailed view of a music pedalboard with cables and equipment on a stage.

Why the override matters

The default KillMode=control-group sends SIGKILL to the whole cgroup after a short timeout. That’s what triggers the watchdog. KillMode=mixed sends SIGTERM to the main process, gives it 30 seconds, then SIGKILL only if needed. It’s a small change but it prevents the deadlock entirely.

If you’re running a cluster with more than one node, do this on all of them before upgrading any. A mixed-version cluster with one node still running the old behavior can still hang.

If you’d rather pay for this

Proxmox has a paid support subscription that includes access to the enterprise repository and tested upgrade paths. It’s about €110 per CPU per year for the basic tier. Worth it if you run production workloads and can’t afford a hung upgrade. For a homelab, the script above is free and works fine.

Related: Fix Proxmox Update Stuck at Processing Triggers for pve-ha-manager

Leave a comment

Comments are reviewed before they appear. Your email is never published.