Ceph OSDs not being able to be configured in fully routed Environment: fix the network first

Ceph OSDs not being able to be configured in a fully routed environment is almost always a network reachability problem. The OSD can’t talk to the monitors, so it never gets created. Fix the routing and the OSD config works.

What breaks in a routed setup

Proxmox defaults to a flat network for Ceph. The public and cluster networks are usually on the same subnet, and the installer assumes that. In a fully routed environment, those subnets are different and often on different VLANs. The OSD needs to reach the monitor IPs over the public network, and the monitors need to reach the OSD over the cluster network. If either path is missing, the OSD can’t be configured.

Check the basics first

On the node where you’re adding the OSD, test connectivity to the monitor IPs. Use ping and then a TCP check on port 6789, the Ceph monitor port.

ping -c 3 10.10.10.10
timeout 1 bash -c 'echo > /dev/tcp/10.10.10.10/6789' && echo open || echo closed

If ping works but the TCP check fails, something is filtering the port. If ping fails, you have a routing or firewall issue. Fix that before touching Ceph config.

Set the public and cluster networks explicitly

Don’t let Ceph guess. In /etc/pve/ceph.conf, define the public network and cluster network for each node. Use the actual subnets, not the interfaces.

[global]
     public network = 10.10.10.0/24
     cluster network = 10.10.20.0/24

Then restart the Ceph services on that node. The OSD will pick up the correct networks when it starts.

If the OSD still won’t configure

Check the OSD log. It’s usually in /var/log/ceph/ceph-osd.*.log. Look for lines about “unable to bind” or “cannot reach monitor”. That tells you which network is broken.

Also verify the monitor addresses in the Ceph config match what the OSD expects. You can see the current monitor map with:

ceph mon dump

If the monitor addresses are on a subnet the OSD can’t reach, you need to fix the routing or add a route on the OSD node.

My take

Fully routed Ceph is more work than it should be. The docs assume a flat network, and the error messages are vague. If you can, keep Ceph on its own VLANs and make sure every node can reach every other node on both the public and cluster networks. That’s the whole fix.

If you’re still stuck, check the Proxmox VM unreachable guide for more network debugging steps.