Multi-Tenant Subvolumes

This guide details how to provision isolated storage for 400 users, each with a strict 10GB quota, using CephFS Subvolumes.

1. Strategy: Subvolumes

Instead of manual directory management, we use the Native Subvolume abstraction.

• Isolation: Each subvolume is managed like a separate volume.
• Quotas: Enforced at the subvolume level.
• Security: Unique cryptographic keys for each user.

2. Create Subvolume Group

Create a logical group to organize student volumes.

# Run on the Admin Node
ceph fs subvolumegroup create cephfs students

# Verify with
ceph fs subvolumegroup ls cephfs

3. Automated Provisioning

3.1 Simple Sequential Batch

Basic Batch

Use this script to create 400 subvolumes (student_001 to student_400) with a fixed quota.

View create_students.sh

#!/bin/bash
# create_students.sh
FS_NAME="cephfs"
GROUP_NAME="students"
USER_COUNT=400
QUOTA_SIZE=$((10 * 1024 * 1024 * 1024))
OUTPUT_FILE="student_credentials.csv"

echo "Username,Subvolume,MountPath,SecretKey" > $OUTPUT_FILE

for i in $(seq -f "%03g" 1 $USER_COUNT); do
    USER_ID="student_$i"
    # 1. Create subvolume
    ceph fs subvolume create $FS_NAME $USER_ID \
        --group_name $GROUP_NAME \
        --size $QUOTA_SIZE > /dev/null 2>&1

    # 2. Authorize client
    ceph fs subvolume authorize $FS_NAME $USER_ID $USER_ID --group_name $GROUP_NAME > /dev/null 2>&1

    # 3. Fetch key and path
    KEY=$(ceph auth get-key client.$USER_ID)
    PATH_ADDR=$(ceph fs subvolume getpath $FS_NAME $USER_ID --group_name $GROUP_NAME)

    echo "$USER_ID,$USER_ID,$PATH_ADDR,$KEY" >> $OUTPUT_FILE
done

3.2 NIM-Based Provisioning

Advanced Batching

This variant allows batching students by Angkatan (Year) and NIM (Student ID) with customizable counts for each batch.

View create_students_nim.sh

#!/usr/bin/env bash

FS_NAME="cephfs"
GROUP_NAME="students"
QUOTA_SIZE=$((5 * 1024 * 1024 * 1024)) # 5GB
MAX_JOBS=10 # Limits concurrent operations

# Ensure script runs as root
if [ "$EUID" -ne 0 ]; then
    echo "[ERROR] Execute this script as root." >&2
    exit 1
fi

# Secure file generation
umask 077
mkdir -p credentials keyrings scripts/windows scripts/linux

# Configuration: "Year:Count" pairs
# Example: 22:150 means Year 2022 with 150 students
BATCH_CONFIG=(
    "21:120"
    "22:120"
    "23:120"
    "24:120"
    "25:120"
)

DELETE_MODE=false
if [[ "$1" == "--delete" ]]; then
    DELETE_MODE=true
    echo "[INFO] Running in DELETE MODE."
else
    echo "[INFO] Running in CREATE MODE."
fi

process_student() {
    local year=$1
    local nim=$2
    local delete_flag=$3

    if [ "$delete_flag" = true ]; then
        ceph fs subvolume rm "$FS_NAME" "$nim" --group_name "$GROUP_NAME" > /dev/null 2>&1
        ceph auth rm "client.$nim" > /dev/null 2>&1
        rm -f "credentials/${nim}_credential.csv" "keyrings/ceph.client.${nim}.keyring" "scripts/windows/mount.${nim}.bat" "scripts/linux/mount.${nim}.sh"
        echo "  [OK] Deleted $nim"
    else
        # 1. Create subvolume with error checking
        if ! ceph fs subvolume create "$FS_NAME" "$nim" --group_name "$GROUP_NAME" --size "$QUOTA_SIZE" > /dev/null 2>&1; then
            echo "  [ERROR] Failed to create subvolume for $nim." >&2
            return 1
        fi

        # 2. Authorize client
        ceph fs subvolume authorize "$FS_NAME" "$nim" "$nim" --group_name "$GROUP_NAME" > /dev/null 2>&1

        # 3. Fetch key and path
        local key
        local path_addr
        key=$(ceph auth get-key "client.$nim" 2>/dev/null)
        path_addr=$(ceph fs subvolume getpath "$FS_NAME" "$nim" --group_name "$GROUP_NAME" 2>/dev/null)

        if [ -z "$key" ] || [ -z "$path_addr" ]; then
            echo "  [ERROR] Failed to retrieve credentials for $nim." >&2
            return 1
        fi

        # 4. Generate Output Files
        echo "NIM,Subvolume,Path,Key" > "credentials/${nim}_credential.csv"
        echo "$nim,$nim,$path_addr,$key" >> "credentials/${nim}_credential.csv"

        printf "[client.%s]\n\tkey = %s\n" "$nim" "$key" > "keyrings/ceph.client.${nim}.keyring"

        # Windows Mount Script
        cat <<EOF > "scripts/windows/mount.${nim}.bat"
ceph-dokan.exe -l z -n client.${nim} -c C:\\ProgramData\\ceph\\ceph.conf -k C:\\ProgramData\\ceph\\keyring\\ceph.client.${nim}.keyring --root-path ${path_addr} --win-vol-name ${nim}
EOF

        # Linux Mount Script
        cat <<EOF > "scripts/linux/mount.${nim}.sh"
#!/usr/bin/env bash
MOUNT_POINT="/mnt/${nim}"
if [ "\$EUID" -ne 0 ]; then exit 1; fi
mkdir -p "\$MOUNT_POINT"
ceph-fuse -n "client.${nim}" -c "/etc/ceph/ceph.conf" -k "/etc/ceph/ceph.client.${nim}.keyring" -r "${path_addr}" "\$MOUNT_POINT"
EOF
        chmod +x "scripts/linux/mount.${nim}.sh"
        echo "  [OK] Provisioned $nim"
    fi
}

for config in "${BATCH_CONFIG[@]}"; do
    YEAR="${config%%:*}"
    COUNT="${config##*:}"

    echo "[INFO] Processing $COUNT students for cohort 20$YEAR..."

    job_count=0
    for i in $(seq -f "%03g" 1 "$COUNT"); do
        NIM="${YEAR}106050${i}"

        # Execute function in the background for concurrency
        process_student "$YEAR" "$NIM" "$DELETE_MODE" &

        ((job_count++))
        if [[ $job_count -ge $MAX_JOBS ]]; then
            wait -n # Wait for the next background job to finish before spawning more
            ((job_count--))
        fi
    done
    wait
done

echo "[INFO] Batch processing completed."

4. Cleanup

Destructive Script

This script permanently deletes all 400 subvolumes and their associated authentication keys.

View cleanup_students.sh

#!/bin/bash
# cleanup_students.sh
for i in $(seq -f "%03g" 1 400); do
    USER_ID="student_$i"
    # Remove volume and key
    ceph fs subvolume rm cephfs $USER_ID --group_name students > /dev/null 2>&1
    ceph auth rm client.$USER_ID > /dev/null 2>&1
done

5. Mounting (Client Side)

Preparation: Client Keyring

Before mounting, ensure the client has the appropriate keyring file. The provisioning scripts automatically generate these in the keyrings/ directory.

# Example for student NIM: 22106050001
# Copy the generated keyring to the standard location
sudo cp keyrings/ceph.client.22106050001.keyring /etc/ceph/

# Or create a secret file for kernel mount (more secure than passing 'secret' directly)
echo "<SECRET_KEY>" > student.secret
chmod 600 student.secret

Kernel Mount

Use the secretfile option for better security.

sudo mount -t ceph <MON_IP>:<PATH_FROM_CSV> /mnt/storage \
  -o name=<USER_ID>,secretfile=student.secret

FUSE Mount (Non-Root)

User Space

Students can use ceph-fuse to mount their data without requiring root privileges.

ceph-fuse -n client.student_001 --client_mountpoint=<PATH> /home/student/data

6. Management

Check Quota Usage

ceph fs subvolume info cephfs student_001 --group_name students

Resize Quota

ceph fs subvolume resize cephfs student_001 --size 20G --group_name students