Technical

Immutable backup on Proxmox: what you can actually enforce on a homelab budget

Immutable backup on Proxmox: what you can actually enforce on a homelab budget

Immutable backup on Proxmox Backup Server is something you can get close to, but you can’t fully achieve on a pure homelab setup without spending money on external storage. PBS prune policies and garbage collection look like immutability from a distance, but they’re not—an attacker who compromises your datastore credentials can still wipe every snapshot. This post is about what you can actually enforce and what you need to stop calling immutable.

Detailed view of a server rack with a focus on technology and data storage.

The gap between prune policies and immutable backup

PBS gives you two powerful tools for managing backup history: prune and garbage collection. Prune removes snapshots older than the retention policy you set, and GC reclaims the chunks that are no longer referenced. Together they keep your datastore from ballooning. A typical prune schedule on a homelab looks like this:

# /etc/proxmox-backup/prune.cfg
keep-daily 7
keep-weekly 4
keep-monthly 6

That gives you daily snapshots for a week, weekly for a month, and monthly for half a year. It feels solid. You can say “nobody can delete my backups because the policy holds them.” But that’s not immutability—it’s a cleanup rule written in a config file. PBS honors that rule when it runs prune as a scheduled task. It does not protect backups from a user with datastore credentials who decides to run prune manually with a different retention or, worse, goes straight to deleting snapshots with the web UI or CLI. If someone gets those credentials, they can shred your history in seconds.

Immutable backup means that once written, a backup cannot be altered or deleted for a fixed period—by anyone, including the administrator who created it. That’s a technological guarantee, not a policy. Think of it as a write-once-read-many (WORM) storage layer. PBS has no built‑in mechanism to enforce that at the chunk level. It relies entirely on access control and hope. In a homelab, that hope often evaporates the moment ransomware hits a perfectly patched machine and you discover that the same NFS mount that holds your backups is writable from every VM because it was convenient.

What the attacker actually sees

Picture a typical single‑host PBS setup: a small Debian VM or bare‑metal box with a ZFS pool for storage, the datastore shared over the network, and the root password reused from some other service because you’ve got enough to remember already. Now an attacker compromises your workstation—maybe through a browser zero‑day—and pivots. They find a credentials file with the PBS API token pasted from a setup guide. With that token, they can list all snapshots, delete any of them, and trigger garbage collection to make sure the chunks are really gone. They could even encrypt the datastore in place by writing corrupt chunks if they wanted to be thorough.

Your prune policy doesn’t matter. Your carefully spaced schedule doesn’t matter. The attacker isn’t playing by the cron job; they’re playing by the API. And because PBS treats the datastore owner as fully trusted, there’s no safety net. This is not a design flaw—PBS assumes you’ll secure the credentials and layer on immutability elsewhere. That elsewhere is the part homelabs often skip.

What you can actually enforce without a second mortgage

True immutability requires a storage backend that supports object locking or a physical air gap. On a homelab budget, you’re probably not buying a tape library, and a cloud service with S3 Object Lock might still feel like overkill for 20 TB of Linux ISOs. But you can raise the bar significantly by reducing the blast radius of a PBS credential compromise and ensuring that even if the main backup target gets nuked, you have a second copy somewhere the attacker didn’t reach.

1. Keep the PBS datastore on a dedicated host with minimal access

Don’t mount your PBS datastore over NFS from a NAS that everything else talks to. Run PBS on its own machine (or VM) and give it its own storage. Backups arrive via PBS client connections, not by writing files directly to the datastore. This means the PBS-host’s root account is the only local door to the chunks. If your Proxmox cluster nodes are compromised, they still have the API token to push new backups, but that token should not have admin privileges—just backup permissions. Limit the token to the specific datastore and set the privilege level to DatastoreBackup only. That prevents it from deleting snapshots or altering prune settings.

Combine this with SSH key‑only access and a firewall that allows inbound connections only from the Proxmox hosts and your own workstation. It won’t stop a determined attacker who also pawns the PBS host itself, but it contains the damage of a stolen API key from a cluster node.

2. Off‑host backup copies that aren’t reachable over the network

The classic “poor man’s air gap”: attach a USB hard drive to the PBS host, run a nightly rsync of the datastore directory, and then unmount the drive. Yes, it’s manual. Yes, it’s annoying. But a drive sitting on a shelf, unpowered, cannot be encrypted by ransomware over the wire. If you script the mount‑copy‑unmount sequence on a cron job, ensure the script runs as a user that has no write access to the PBS configuration—otherwise a compromised PBS host could rewrite your script to copy garbage. Actually, even better: run the script on a completely separate machine that pulls the backups from PBS via the API (with read‑only credentials) and writes to the USB disk. That way a compromise of the PBS host doesn’t poison the archive.

I’d expect PBS to handle that fine, though I can’t point at a run that proves it. The API client proxmox-backup-client can download entire snapshot contents, and you could cron that pull onto a Raspberry Pi with a hard drive. It’s slow, but it’s offline once the pull is done.

3. ZFS snapshots on the PBS datastore (if you have root on PBS)

If your PBS datastore sits on ZFS, you can periodically snapshot the dataset and keep those snapshots on a separate pool or send them to another machine. ZFS snapshots themselves are read‑only, so an attacker who gains PBS credentials but not root on the PBS host cannot delete them. The catch: if the attacker gets root on the PBS host, they can destroy ZFS snapshots just as easily. This is why I’d only count ZFS snapshots as a layer, not a guarantee. It’s the same story: you’re moving the trust boundary from the application to the host. That might be enough if you practice good host hygiene—separate management network, no public services, patched kernel. For more on keeping that host alive and diagnosing surprises after an update, I’ve covered the post‑reboot evidence checklist in another post.

Detailed view of computer hard disk drive components, showcasing circuitry and connectors.

Where prune schedules fit into a defense‑in‑depth strategy

Even though prune alone isn’t immutable backup, a correctly tuned prune schedule still helps by limiting what an attacker can delete in one go if they only have the API key for a few minutes. If you keep 180 days of backups, an automated attack that triggers prune with a bad retention will still need to wait for the next prune window—by default, PBS runs prune daily. That gives you a detection window. You can also set up alerts on mass snapshot deletions by tailing the PBS access log. None of this is bulletproof; it’s just making the attacker work harder while you hopefully notice something’s off.

The biggest risk in a homelab remains the human one: accidentally running a prune command with too few “keep” values, or fat‑fingering the web UI and deleting a whole VM’s backup chain. PBS has a trash protection for the UI that delays actual deletion for 24 hours, but that’s easily overridden. I’ve seen colleagues do it. I’ve nearly done it. No policy saves you from your own mouse click unless the storage itself says no.

What you get when you add actual immutability

To move from “retention policy with confidence” to genuine immutable backup, you need a storage layer that enforces WORM semantics. In the PBS ecosystem, the cleanest path is using a remote S3‑compatible object store that supports Object Lock. Providers like Backblaze B2, Wasabi, or even a self‑hosted MinIO with locking enabled can serve as a PBS datastore via the “remote” sync feature (note: PBS doesn’t natively write to S3; you’d use replication to push snapshots from a local datastore to the S3‑backed datastore).

Configuring PBS replication to an S3 datastore is beyond the scope of this piece, but the critical bit is that once the remote store has Object Lock enabled with a retention period, no credentials—admin or otherwise—can delete that data until the lock expires. If ransomware owns your entire house, those backups survive. That’s the actual promise of immutable backup, and on a homelab budget it usually costs more than the electricity to run the server.

For those already using ZFS replication between Proxmox nodes as a quick‑and‑dirty DR strategy, you’ve seen how well replication works for keeping uptime; the same mindset can be applied to backups, just one layer deeper. I wrote about that when replacing VMware SRM with Proxmox tools, and the lesson holds: replication is cheap, fast, and beats the hell out of doing nothing. It doesn’t make your backups immutable, but it adds geographic distance and a second set of credentials to defeat.

If you’d rather pay for this

If you’re done pretending that an rsync to a USB stick qualifies as immutable backup and you want a service that actually delivers WORM semantics without building anything, Backblaze B2 with Object Lock is the straightforward answer. You’ll pay about $6.95 per terabyte per month for storage at the time of writing, plus negligible API call costs. Object Lock is free to enable; you set a compliance or governance retention period (say 30 days) and after that, nothing can delete your data until the lock expires. Integration with PBS means setting up a replication job to a B2 bucket after you’ve enabled Object Lock in the bucket settings. It works, and the bill is transparent.

At homelab scale—let’s say you back up 2 TB of important data and not the whole media library—you’re looking at about $14/month. For that, you stop worrying about accidentally pruning the wrong backup chain or a stray script nuking your datastore. It’s worth paying when the data you’re protecting would cost more than a few months of coffee money to recreate. If your backups are mostly replaceable Linux ISO repacks and you’re okay losing a week’s worth of configs, then no, it’s not worth it.

Immutability is one of four places a Proxmox backup can look fine and not be; the rest are collected in Proxmox Backup & Recovery.

Leave a comment

Comments are reviewed before they appear. Your email is never published.