CapturedByNguyen

Homelab & Vietnam

Proxmox in a homelab

Proxmox in a homelab breaks in four places: the host underneath, the containers on top, the storage behind it, and the cluster between the nodes. Almost everything on this page is one of those four wearing a different error message — which is why they are collected together rather than scattered across an archive by date.

All of it is written from a single Proxmox host running a working homelab, in the order the problems tend to arrive.

Backup and storage

Every guide in this section exists because of the same discovery, arrived at from four different directions: a backup can look completely successful and protect nothing.

Not a corrupted archive, not a failed job, not a red light anywhere. A green tick, a sensible retention policy, a datastore filling up at a believable rate — and a gap you only find on the day you need the data back. That is the failure mode this collection is about, because it is the one that actually costs people their data. A backup that visibly fails gets fixed on Tuesday.

1. What is actually inside the backup

The first assumption worth destroying is that backing up a container backs up what the container serves. Bind-mount a NAS dataset into an unprivileged LXC and the data lives outside the container’s rootfs — so it is not in the container backup, and nothing tells you that.

Sharing TrueNAS Datasets with Unprivileged LXC Containers in Proxmox is the setup itself: how the UID mapping works and how to get the permissions right. Read it for the mechanics, and note what the resulting backup does and does not contain.

2. What retention actually guarantees

Once the contents are right, the next assumption is that a retention policy is a protection policy. It is not. Prune and garbage collection decide what is kept; they decide nothing about who can delete it. Anyone holding datastore credentials can still take the whole history.

Immutable backup on Proxmox is about the gap between those two things — what you can genuinely enforce on your own hardware, what needs object storage with a lock on it, and what you should stop calling immutable.

3. When the backup system is the thing that breaks

Backups protect the machines they run against. They do not protect themselves. A kernel update took the backup host down here, and the question stopped being “are the backups good” and became “can I prove anything about the state of the host that holds them”.

PBS Backup Host Outage: Evidence Checklist After a Kernel Update is the checklist that came out of it — what to collect, in what order, before you start changing things and destroy the evidence.

4. Copies are not recovery

The last assumption is the biggest one. Replication gives you the data somewhere else. Recovery is an ordered runbook, boot sequencing, re-addressing, and a test you can actually run without breaking production — which is what the commercial products were selling all along.

What Replaces SRM and vSphere Replication on Proxmox? covers ZFS send/receive and the scripting around it, and is honest about the part scripting does not replace.

5. When the storage underneath is the problem

The rest of this section is the layer below all of that. LVM thin provisioning and cache, qcow2 images that come back carrying a previous guest’s data, NFS mounts that connect from the frontend and fail from the backend, and the one that wastes the most time of any of them: I/O pressure stalls, where nothing is broken and everything is slow.

If you are chasing a stall, start with Proxmox IO pressure stall and read PSI before you touch anything else — it is one command and it tells you whether to look at the host or inside a single container. Guessing that wrong is how an afternoon disappears.

The short version

If you read nothing else here: check what is inside your backups before you trust the schedule, and restore something on purpose, on a day you chose. Every problem in this section is cheap to find on a Tuesday afternoon and expensive to find any other way.

Containers and permissions

Unprivileged LXC containers are the right default and they cost you one thing: the UID map. Proxmox hands each container 65536 IDs starting at host UID 100000, and every “permission denied”, every nobody:nogroup on a bind mount, and a surprising number of stranger symptoms trace back to data whose ownership does not fit inside that window.

Proxmox unprivileged container UID/GID mapping explained is the reference — remapping the range, carving a hole in it, and the /etc/subuid entry whose absence makes the container refuse to start with an error that says nothing useful.

The rest of this section is what goes wrong once containers are running: a monitoring agent reporting the host’s RAM instead of the container’s, a TTY that answers in gibberish, screen refusing to attach, and resizing a container’s disk without rebuilding it.

Host and hardware

Before you debug a VM or a container, rule out the host. That is not a slogan — it is the order that saves time, because a host-level fault presents as an application-level symptom often enough to send you looking in the wrong place for an hour.

This section covers getting Proxmox installed at all (USB installers that fail on current versions, PXE that cannot find its ISO), the boot chain when it stops halfway, hardware that misbehaves quietly (an NVMe controller dropping off the bus, a kernel warning about a corrupt EDID header), a hypervisor that restarts itself, and the console you are reading all of it through.

Cluster and networking

The smallest cluster is the one that teaches you the most, and usually the hard way: two nodes, no quorum device, and a reboot on one of them that stalls processes on the other. This section is that, plus the network side — a container with no internet, multiple IPs on one subnet, finding a VM’s address when the guest agent is not installed, and taking a cluster apart properly when you want the node back as a standalone host.

What is not here

This page grows as things break. It does not cover backing up Proxmox’s own configuration, Ceph, or anything at genuine business scale, because none of those has happened here yet — and there is enough advice online written by people they also have not happened to.