Proxmox VE ZFS benchmark with NVMe isn’t about chasing the highest numbers—it’s about finding the settings that don’t wreck your VM performance. Here’s a practical workflow that tests ashift, recordsize, and sync side-by-side.
Start with ashift
ashift is the first thing to get right, and you can’t change it after the pool is created. For NVMe drives, ashift=12 (4K sectors) is the safe default, but many modern NVMe drives report 4K logical sectors anyway. Check with:
cat /sys/block/nvme0n1/queue/logical_block_sizeIf it says 4096, ashift=12 is correct. If it says 512, you might get slightly better space efficiency with ashift=9, but the performance difference is negligible and 12 is future-proof. Just use 12 and move on.
recordsize: match your workload
recordsize is the ZFS block size for datasets. Default is 128K, which is fine for general VM storage. But if you’re running databases or other random I/O workloads, a smaller recordsize like 16K or 8K can reduce read amplification. Test with fio inside a VM:
fio --name=randomread --ioengine=libaio --iodepth=32 --rw=randread --bs=4k --size=1G --numjobs=4 --runtime=60 --time_basedThen change recordsize on a test dataset and compare. Don’t go crazy—128K is fine for most homelab stuff.
sync: the silent killer
sync=standard is the default, and it means ZFS honors sync writes from VMs. That’s safe but slow on NVMe without a SLOG. If you’re benchmarking and see terrible write performance, check if your workload is doing sync writes. You can set sync=disabled on a dataset to see the raw speed, but don’t run production VMs that way unless you’re okay losing data on power loss. For a homelab, sync=disabled is often fine and gives a huge boost.
Test procedure that actually works
- Create a test pool with the settings you want to compare.
- Create a dataset for each scenario: e.g.,
tank/test-ashift12-rec128-syncstd. - Install fio in a VM or container on that dataset.
- Run the same fio jobs against each dataset and note IOPS and latency.
- Change one variable at a time—don’t mix ashift, recordsize, and sync changes together or you won’t know what helped.
This is boring but it’s the only way to get numbers you can actually use. There’s no magic command that tells you the “best” settings; you have to test your own workload.
My take
Most of the time, the defaults are fine. ashift=12, recordsize=128K, sync=standard. If you’re not hitting performance problems, don’t benchmark—just use it. But if you do need more speed, start with sync=disabled and see if that’s enough before messing with recordsize. And if you’re still not happy, check out IO Pressure Stall – Unexplained: Diagnosing Proxmox LXC and ZFS I/O Hangs for some deeper troubleshooting.
