Ceph Cluster Guide

Appearance

Multi-Tenant Subvolumes ​

This guide details how to provision isolated storage for 400 users, each with a strict 10GB quota, using CephFS Subvolumes.

1. Strategy: Subvolumes ​

Instead of manual directory management, we use the Native Subvolume abstraction.

  • Isolation: Each subvolume is managed like a separate volume.
  • Quotas: Enforced at the subvolume level.
  • Security: Unique cryptographic keys for each user.

2. Create Subvolume Group ​

Create a logical group to organize student volumes.

bash
# Run on the Admin Node
ceph fs subvolumegroup create cephfs students

3. Automated Provisioning ​

Batch Operations

Use this script to create 400 subvolumes, set quotas, and generate unique access keys.

bash
#!/bin/bash
# create_students.sh
FS_NAME="cephfs"
GROUP_NAME="students"
USER_COUNT=400
QUOTA_SIZE=$((10 * 1024 * 1024 * 1024))
OUTPUT_FILE="student_credentials.csv"

echo "Username,Subvolume,MountPath,SecretKey" > $OUTPUT_FILE

for i in $(seq -f "%03g" 1 $USER_COUNT); do
    USER_ID="student_$i"
    # 1. Create subvolume
    ceph fs subvolume create $FS_NAME $USER_ID \
        --group_name $GROUP_NAME \
        --size $QUOTA_SIZE > /dev/null 2>&1

    # 2. Authorize client
    ceph fs subvolume authorize $FS_NAME $USER_ID $USER_ID --group_name $GROUP_NAME > /dev/null 2>&1

    # 3. Fetch key and path
    KEY=$(ceph auth get-key client.$USER_ID)
    PATH_ADDR=$(ceph fs subvolume getpath $FS_NAME $USER_ID --group_name $GROUP_NAME)

    echo "$USER_ID,$USER_ID,$PATH_ADDR,$KEY" >> $OUTPUT_FILE
done

4. Cleanup ​

Destructive Script

This script permanently deletes all 400 subvolumes and their associated authentication keys.

bash
#!/bin/bash
# cleanup_students.sh
for i in $(seq -f "%03g" 1 400); do
    USER_ID="student_$i"
    # Remove volume and key
    ceph fs subvolume rm cephfs $USER_ID --group_name students > /dev/null 2>&1
    ceph auth del client.$USER_ID > /dev/null 2>&1
done

5. Mounting (Client Side) ​

Kernel Mount ​

bash
sudo mount -t ceph <MON_IP>:<PATH_FROM_CSV> /mnt/storage \
  -o name=<USER_ID>,secret=<KEY_FROM_CSV>

FUSE Mount (Non-Root) ​

User Space

Students can use ceph-fuse to mount their data without requiring root privileges.

bash
ceph-fuse -n client.student_001 --client_mountpoint=<PATH> /home/student/data

6. Management ​

Check Quota Usage ​

bash
ceph fs subvolume info cephfs student_001 --group_name students

Resize Quota ​

bash
ceph fs subvolume resize cephfs student_001 --size 20G --group_name students