Ceph Cluster Guide

Appearance

Cluster Management ​

Operational commands for managing your Ceph cluster, scaling, and maintenance. Run these from the Admin node (ceph-mgr-1).

1. Host Labels ​

Labels tell the orchestrator (cephadm) where to place specific daemons.

View & Add Labels ​

bash
# View current host status
sudo ceph orch host ls

# Add 'mon' label to a host
sudo ceph orch host label add ceph-osd-1 mon

# Remove a label
sudo ceph orch host label rm ceph-osd-1 mon

2. Monitor Placement (MON) ​

Ceph requires an odd number of monitors (3 or 5) for quorum.

Placement Strategy

You can either place monitors on specific hosts or use labels for auto-deployment.

bash
# Option A: Explicit placement
sudo ceph orch apply mon --placement="ceph-osd-1,ceph-osd-2,ceph-osd-3"

# Option B: Label-based placement
sudo ceph orch apply mon --placement="label:mon"

3. Storage Management (OSD) ​

Add OSDs ​

bash
# View available empty disks
sudo ceph orch device ls

# Add a specific disk
sudo ceph orch daemon add osd ceph-osd-1:/dev/sdb

Remove OSDs Safely ​

Data Evacuation

Never pull a disk physically before evacuating data via the orchestrator.

  1. Schedule removal: sudo ceph orch osd rm <ID>
  2. Check progress: sudo ceph orch osd rm status
  3. Physical removal: Only after status is "done".

4. Maintenance Mode ​

Use these flags to prevent "rebalance storms" when rebooting nodes for updates.

bash
# Set maintenance flags
sudo ceph osd set noout
sudo ceph osd set norebalance

# ... Reboot Nodes ...

# Return to normal operations
sudo ceph osd unset noout
sudo ceph osd unset norebalance

5. The Nuclear Option (Purge) ​

Irreversible Data Loss

This obliterates the entire cluster and wipes all data. Use with extreme caution.

  1. Get Cluster ID: sudo ceph fsid
  2. Destroy Cluster: sudo cephadm rm-cluster --zap-osds --fsid <FSID> --force

6. Troubleshooting Placement ​

Stuck Removal? ​

If an OSD removal is stuck, you can abort:

bash
sudo ceph orch osd rm stop <ID>
sudo ceph osd in <ID>
sudo ceph orch daemon start osd.<ID>

Unmanaged Services ​

Orchestration Logic

If ceph orch ls shows <unmanaged>, it means the daemons were added via imperative commands rather than declarative YAML specs. Ceph will not automatically replace these if they fail.