Proxmox Host Crashing How to Debug: Persistent Journal, Netconsole, and kdump

When your Proxmox host crashes and reboots, the default journal is gone. You need to make it persistent, set up netconsole, and configure kdump so the crash leaves evidence somewhere readable. Here’s the order I’d do it in.

Make the journal persistent

The systemd journal lives in /run/log/journal by default, which is tmpfs. That means it’s wiped on every reboot. To keep it across crashes, create the directory and set permissions:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

Now check that it’s actually persistent:

journalctl --list-boots

If you see boots from before the crash, you’re good. If you only see the current boot, the directory isn’t set up right.

Set up netconsole

Netconsole sends kernel messages over the network to another machine. That way, even if the disk is corrupted or the journal is lost, you get the last kernel messages before the crash. You need a second machine on the same network to receive the logs.

On the receiving machine, set up a listener. I’d use netcat:

nc -u -l -p 6666 | tee netconsole.log

On the Proxmox host, load the netconsole module with the right parameters. Replace the IPs with your own:

modprobe netconsole netconsole=@/,@192.168.1.10/

The first IP is the source (can be empty), the second is the receiver. The port defaults to 6666. To make it permanent, add it to /etc/modules-load.d/netconsole.conf and put the options in /etc/modprobe.d/netconsole.conf.

Test it by triggering a kernel message:

echo test > /dev/kmsg

You should see the test message in your netcat listener. If not, check your firewall and network.

Configure kdump

kdump captures a full kernel crash dump. It needs a reserved memory area and a place to save the dump. Proxmox uses the Debian kernel, so kdump is available but not installed by default.

Install it:

apt install kdump-tools

Then edit /etc/default/kdump-tools and set KDUMP_CMDLINE_APPEND to reserve memory. Something like:

KDUMP_CMDLINE_APPEND="crashkernel=256M"

You’ll need to reboot for the crashkernel parameter to take effect. After reboot, check that the memory is reserved:

cat /proc/cmdline

You should see crashkernel=256M in there. Then enable kdump:

systemctl enable kdump-tools
systemctl start kdump-tools

Now when the kernel panics, kdump will save a vmcore file to /var/crash. You can analyze it with crash or just check the kernel log for the panic message.

Check the kernel ring buffer

Sometimes the crash leaves messages in the kernel ring buffer that survive a soft reboot. After the host comes back, run:

dmesg | tail -50

Or look at the previous boot’s kernel messages:

journalctl -k -b -1

That often shows the panic or oops right before the crash. If you see something like “Kernel panic – not syncing” or a hardware error, you have a lead.

If none of this works, you might have a hardware issue. But at least now you have a way to see what the kernel was thinking when it died. Check out Proxmox Unresponsive After 1 or 2 Days: Check the Kernel Ring Buffer First for more on that.