Container Storage Interface (CSI) Basics

Comprehensive guide to CSI drivers, volume management, and persistent storage in Kubernetes.

Overview

The Container Storage Interface (CSI) is a standardised API that enables container orchestration systems like Kubernetes to expose arbitrary storage systems to containerised workloads. CSI drivers allow storage vendors to develop plugins once and have them work across different container orchestration platforms, replacing the legacy in-tree volume plugins.

CSI provides dynamic provisioning, volume expansion, snapshotting, and cloning capabilities, making persistent storage management in Kubernetes more flexible and vendor-agnostic.

⤢ zoom

CSI Architecture and Components

Key Concepts

• CSI Driver: Plugin implementing the CSI specification for a storage backend
• External Provisioner: Watches PVCs and triggers volume creation via CSI driver
• External Attacher: Attaches/detaches volumes to/from nodes
• External Resizer: Handles volume expansion requests
• External Snapshotter: Manages volume snapshots and restoration
• Node Plugin: Runs on each node to mount/unmount volumes
• Storage Class: Template defining volume parameters and which CSI driver to use

CSI Driver Lifecycle

⤢ zoom

Installing CSI Drivers

# List available CSI drivers on node
kubectl get csidrivers

# Describe CSI driver
kubectl describe csidriver ebs.csi.aws.com

# View CSI node information
kubectl get csinodes
kubectl describe csinode worker-node-1

# Check CSI driver pods
kubectl get pods -n kube-system | grep csi

Storage Classes

Storage Classes define the provisioner, parameters, and policies for dynamically provisioned volumes.

Basic Storage Class Structure

# storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
mountOptions:
  - discard
  - noatime

Common Storage Class Parameters

AWS EBS CSI Driver

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3                    # gp2, gp3, io1, io2, st1, sc1
  iops: "3000"                 # Only for gp3, io1, io2
  throughput: "125"            # Only for gp3 (MiB/s)
  encrypted: "true"
  fsType: ext4                 # ext3, ext4, xfs
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain

GCP Persistent Disk CSI Driver

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pd-balanced
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-balanced            # pd-standard, pd-balanced, pd-ssd, pd-extreme
  replication-type: regional-pd  # none, regional-pd
  disk-encryption-kms-key: projects/PROJECT/locations/LOCATION/keyRings/RING/cryptoKeys/KEY
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Azure Disk CSI Driver

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-disk-premium
provisioner: disk.csi.azure.com
parameters:
  storageaccounttype: Premium_LRS  # Standard_LRS, Premium_LRS, StandardSSD_LRS, UltraSSD_LRS
  kind: Managed                     # Shared, Dedicated, Managed
  cachingMode: ReadOnly             # None, ReadOnly, ReadWrite
  fsType: ext4
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

NFS CSI Driver

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs-server.example.com
  share: /export/volumes
  mountOptions: vers=4.1,noatime,nodiratime
reclaimPolicy: Retain
volumeBindingMode: Immediate

Storage Class Operations

# List storage classes
kubectl get storageclass
kubectl get sc

# Describe storage class
kubectl describe sc fast-ssd

# Set default storage class
kubectl patch storageclass fast-ssd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# Remove default storage class annotation
kubectl patch storageclass fast-ssd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

# Create storage class
kubectl apply -f storageclass.yaml

# Delete storage class (doesn't affect existing PVs)
kubectl delete sc fast-ssd

Volume Provisioning and PersistentVolumeClaims

Dynamic Provisioning

# pvc-dynamic.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: web-storage
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi
  # Optional: Volume selector for specific PV
  selector:
    matchLabels:
      environment: production
      tier: frontend

Static Provisioning

# pv-static.yaml - Pre-provisioned volume
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-001
  labels:
    type: nfs
    environment: production
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-manual
  mountOptions:
    - hard
    - nfsvers=4.1
    - noatime
  csi:
    driver: nfs.csi.k8s.io
    volumeHandle: nfs-server.example.com:/export/volumes/pv-001
    volumeAttributes:
      server: nfs-server.example.com
      share: /export/volumes/pv-001
---
# pvc-static.yaml - Claim for pre-provisioned volume
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: nfs-manual
  resources:
    requests:
      storage: 500Gi
  selector:
    matchLabels:
      type: nfs
      environment: production

Using PVCs in Pods

# pod-with-pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: web-server
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    volumeMounts:
    - name: web-storage
      mountPath: /usr/share/nginx/html
    - name: cache
      mountPath: /var/cache/nginx
  volumes:
  - name: web-storage
    persistentVolumeClaim:
      claimName: web-storage
  - name: cache
    persistentVolumeClaim:
      claimName: cache-storage

# statefulset-with-pvc.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 100Gi

PVC Operations

# List PVCs
kubectl get pvc
kubectl get pvc -A  # All namespaces

# Describe PVC
kubectl describe pvc web-storage

# Check PVC status and binding
kubectl get pvc web-storage -o jsonpath='{.status.phase}'  # Bound, Pending, Lost

# View bound PV
kubectl get pvc web-storage -o jsonpath='{.spec.volumeName}'

# Expand PVC (requires allowVolumeExpansion: true)
kubectl patch pvc web-storage -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

# Delete PVC (respects reclaim policy)
kubectl delete pvc web-storage

# Force delete stuck PVC
kubectl patch pvc web-storage -p '{"metadata":{"finalizers":null}}'

Access Modes and Reclaim Policies

Access Modes

⤢ zoom

Access Mode Examples

# ReadWriteOnce - Block storage (most common)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-storage
spec:
  accessModes:
    - ReadWriteOnce  # Single node, single or multiple pods on same node
  storageClassName: ebs-gp3
  resources:
    requests:
      storage: 100Gi

# ReadWriteMany - Shared file storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-assets
spec:
  accessModes:
    - ReadWriteMany  # Multiple nodes can read and write
  storageClassName: efs-sc
  resources:
    requests:
      storage: 500Gi

# ReadOnlyMany - Shared read-only content
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: static-content
spec:
  accessModes:
    - ReadOnlyMany  # Multiple nodes read-only
  storageClassName: nfs-client
  resources:
    requests:
      storage: 100Gi

# ReadWriteOncePod - Single pod access (GA k8s 1.29; alpha from 1.22)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: single-pod-storage
spec:
  accessModes:
    - ReadWriteOncePod  # Only one pod cluster-wide
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

Reclaim Policies

Policy	Behaviour	Use Case
Delete	Volume deleted when PVC deleted	Ephemeral data, development
Retain	Volume retained when PVC deleted, manual cleanup required	Production data, backups
Recycle	Deprecated - Volume scrubbed (rm -rf) and made available	Legacy only

# StorageClass with Retain policy
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: retain-storage
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain  # Volume kept after PVC deletion
volumeBindingMode: WaitForFirstConsumer

# Change reclaim policy on existing PV
kubectl patch pv pvc-12345678-1234-1234-1234-123456789012 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

# Reclaim retained PV for reuse
# 1. Delete the PVC (PV enters Released state)
kubectl delete pvc web-storage

# 2. Remove claimRef from PV
kubectl patch pv pvc-12345678-1234-1234-1234-123456789012 -p '{"spec":{"claimRef":null}}'

# 3. PV becomes Available again
kubectl get pv

Volume Binding Modes

# Immediate - Volume provisioned immediately
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: immediate-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: Immediate  # Provision as soon as PVC created

# WaitForFirstConsumer - Volume provisioned when pod scheduled (recommended)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: wait-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer  # Provision in correct AZ/zone

Volume Snapshots and Cloning

Volume Snapshot Classes

# volumesnapshotclass.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-snapclass
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
driver: ebs.csi.aws.com
deletionPolicy: Delete  # or Retain
parameters:
  # Driver-specific parameters
  encrypted: "true"

Creating Volume Snapshots

# volumesnapshot.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: db-snapshot-20231206
  namespace: production
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: database-storage

# Create snapshot
kubectl apply -f volumesnapshot.yaml

# List snapshots
kubectl get volumesnapshot
kubectl get volumesnapshot -A

# Check snapshot status
kubectl describe volumesnapshot db-snapshot-20231206

# View snapshot content (created automatically)
kubectl get volumesnapshotcontent
kubectl describe volumesnapshotcontent snapcontent-12345678-1234-1234-1234-123456789012

# Delete snapshot
kubectl delete volumesnapshot db-snapshot-20231206

Restoring from Snapshots

# pvc-from-snapshot.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-restore
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi
  dataSource:
    name: db-snapshot-20231206
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Volume Cloning

# pvc-clone.yaml - Clone existing PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-clone
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi
  dataSource:
    name: database-storage
    kind: PersistentVolumeClaim

Snapshot Automation with CronJob

# snapshot-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-snapshot
  namespace: production
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: snapshot-creator
          restartPolicy: OnFailure
          containers:
          - name: create-snapshot
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - |
              DATE=$(date +%Y%m%d-%H%M%S)
              cat <<EOF | kubectl apply -f -
              apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                name: database-snapshot-${DATE}
                namespace: production
              spec:
                volumeSnapshotClassName: csi-snapclass
                source:
                  persistentVolumeClaimName: database-storage
              EOF
---
# RBAC for snapshot creation
apiVersion: v1
kind: ServiceAccount
metadata:
  name: snapshot-creator
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: snapshot-creator
  namespace: production
rules:
- apiGroups: ["snapshot.storage.k8s.io"]
  resources: ["volumesnapshots"]
  verbs: ["create", "get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: snapshot-creator
  namespace: production
subjects:
- kind: ServiceAccount
  name: snapshot-creator
roleRef:
  kind: Role
  name: snapshot-creator
  apiGroup: rbac.authorization.k8s.io

Rook/Ceph Integration

Rook is a cloud-native storage orchestrator that automates deployment and management of Ceph storage clusters in Kubernetes.

Rook/Ceph Architecture

⤢ zoom

Installing Rook/Ceph

# Clone Rook repository
git clone --single-branch --branch release-1.20 https://github.com/rook/rook.git
cd rook/deploy/examples

# Create Rook operator
kubectl create -f crds.yaml
kubectl create -f common.yaml
kubectl create -f operator.yaml

# Verify operator is running
kubectl get pods -n rook-ceph

# Create Ceph cluster (customize cluster.yaml for your environment)
kubectl create -f cluster.yaml

# Wait for cluster to be ready (can take 5-10 minutes)
kubectl get cephcluster -n rook-ceph
kubectl get pods -n rook-ceph

# Deploy Rook toolbox for debugging
kubectl create -f toolbox.yaml

Ceph Cluster Configuration

# cluster.yaml - Production Ceph cluster
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v19.2.2  # Ceph Squid (Rook 1.20 default); v17 Quincy is EOL
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 2
    allowMultiplePerNode: false
  dashboard:
    enabled: true
    ssl: true
  monitoring:
    enabled: true
  network:
    connections:
      encryption:
        enabled: false
      compression:
        enabled: false
  crashCollector:
    disable: false
  storage:
    useAllNodes: false
    useAllDevices: false
    nodes:
    - name: "worker-node-1"
      devices:
      - name: "/dev/nvme1n1"
    - name: "worker-node-2"
      devices:
      - name: "/dev/nvme1n1"
    - name: "worker-node-3"
      devices:
      - name: "/dev/nvme1n1"
  resources:
    mon:
      requests:
        cpu: "1000m"
        memory: "2Gi"
      limits:
        memory: "4Gi"
    osd:
      requests:
        cpu: "2000m"
        memory: "4Gi"
      limits:
        memory: "8Gi"
    mgr:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        memory: "2Gi"

Ceph Block Storage (RBD)

# storageclass-rbd.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Ceph Filesystem (CephFS)

# filesystem.yaml
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: ceph-filesystem
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
      requireSafeReplicaSize: true
  dataPools:
  - name: replicated
    replicated:
      size: 3
      requireSafeReplicaSize: true
  metadataServer:
    activeCount: 1
    activeStandby: true
    resources:
      requests:
        cpu: "1000m"
        memory: "4Gi"
      limits:
        memory: "8Gi"
---
# storageclass-cephfs.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: ceph-filesystem
  pool: ceph-filesystem-replicated
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Ceph Object Storage (S3)

# object-store.yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPool:
    replicated:
      size: 3
  preservePoolsOnDelete: false
  gateway:
    port: 80
    instances: 2
    resources:
      requests:
        cpu: "1000m"
        memory: "2Gi"
      limits:
        memory: "4Gi"
---
# object-user.yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: my-user
  namespace: rook-ceph
spec:
  store: my-store
  displayName: "My S3 User"

# Get S3 credentials
kubectl get secret -n rook-ceph rook-ceph-object-user-my-store-my-user -o jsonpath='{.data.AccessKey}' | base64 -d
kubectl get secret -n rook-ceph rook-ceph-object-user-my-store-my-user -o jsonpath='{.data.SecretKey}' | base64 -d

# Get S3 endpoint
kubectl get svc -n rook-ceph rook-ceph-rgw-my-store

Ceph Management Commands

# Access Ceph toolbox
kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- bash

# Inside toolbox - check cluster status
ceph status
ceph health detail
ceph df

# Check OSD status
ceph osd status
ceph osd tree
ceph osd df

# Check pool information
ceph osd pool ls
ceph osd pool stats

# Monitor performance
ceph osd perf

# Check monitor status
ceph mon stat

# View placement groups
ceph pg stat
ceph pg dump

# Access Ceph dashboard
kubectl get secret -n rook-ceph rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d
kubectl port-forward -n rook-ceph svc/rook-ceph-mgr-dashboard 8443:8443
# Open https://localhost:8443 (username: admin)

Local Path Provisioner

Local Path Provisioner dynamically provisions Kubernetes local volumes using hostPath, ideal for single-node clusters or local development.

Installing Local Path Provisioner

# Install via kubectl
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.36/deploy/local-path-storage.yaml

# Verify installation
kubectl get pods -n local-path-storage
kubectl get storageclass local-path

Local Path Configuration

# local-path-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-path-config
  namespace: local-path-storage
data:
  config.json: |-
    {
      "nodePathMap": [
        {
          "node": "DEFAULT_PATH_FOR_NON_LISTED_NODES",
          "paths": ["/opt/local-path-provisioner"]
        },
        {
          "node": "worker-node-1",
          "paths": ["/mnt/ssd/local-storage", "/mnt/hdd/local-storage"]
        }
      ]
    }
  setup: |-
    #!/bin/sh
    set -eu
    mkdir -m 0777 -p "$VOL_DIR"
  teardown: |-
    #!/bin/sh
    set -eu
    rm -rf "$VOL_DIR"
  helperPod.yaml: |-
    apiVersion: v1
    kind: Pod
    metadata:
      name: helper-pod
    spec:
      priorityClassName: system-node-critical
      tolerations:
      - key: node.kubernetes.io/disk-pressure
        operator: Exists
        effect: NoSchedule
      containers:
      - name: helper-pod
        image: busybox:latest
        imagePullPolicy: IfNotPresent

Local Path StorageClass

# storageclass-local.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

Using Local Path Storage

# pvc-local.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: local-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 10Gi

Local Path with Node Affinity

# pod-local-affinity.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-with-local-storage
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - worker-node-1  # Pin to specific node
  containers:
  - name: app
    image: nginx:1.25
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: local-pvc

Longhorn

Longhorn is a lightweight, reliable distributed block storage system for Kubernetes, providing enterprise-grade persistent storage with snapshots, backups, and disaster recovery.

Longhorn Architecture

⤢ zoom

Installing Longhorn

# Install via kubectl
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.12.0/deploy/longhorn.yaml

# Verify installation
kubectl get pods -n longhorn-system
kubectl get daemonset -n longhorn-system

# Install via Helm
helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace

# Access Longhorn UI
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8000:80
# Open http://localhost:8000

Longhorn Prerequisites

# Install open-iscsi (required on all nodes)
# Ubuntu/Debian
sudo apt-get install open-iscsi -y
sudo systemctl enable iscsid
sudo systemctl start iscsid

# RHEL/CentOS
sudo yum install iscsi-initiator-utils -y
sudo systemctl enable iscsid
sudo systemctl start iscsid

# Install NFSv4 client (for ReadWriteMany support)
# Ubuntu/Debian
sudo apt-get install nfs-common -y

# RHEL/CentOS
sudo yum install nfs-utils -y

# Verify prerequisites
curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.12.0/scripts/environment_check.sh | bash

Longhorn StorageClass

# storageclass-longhorn.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"  # 48 hours
  fromBackup: ""
  fsType: "ext4"
  dataLocality: "disabled"  # disabled, best-effort, strict-local

Longhorn with Different Replica Counts

# storageclass-longhorn-single.yaml - Single replica (development)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-single
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "1"
  dataLocality: "strict-local"  # Keep data on same node
---
# storageclass-longhorn-ha.yaml - High availability
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-ha
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"
  dataLocality: "disabled"
  replicaAutoBalance: "least-effort"  # best-effort, least-effort, disabled

Longhorn Recurring Backups

# recurring-job.yaml
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: daily-backup
  namespace: longhorn-system
spec:
  cron: "0 2 * * *"  # Daily at 2 AM
  task: "backup"
  groups:
  - default
  retain: 7  # Keep 7 backups
  concurrency: 2
  labels:
    interval: daily
---
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: hourly-snapshot
  namespace: longhorn-system
spec:
  cron: "0 * * * *"  # Every hour
  task: "snapshot"
  groups:
  - default
  retain: 24
  concurrency: 5
  labels:
    interval: hourly

Longhorn Backup Target Configuration

# backup-target.yaml - S3 backup
apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: longhorn-system
type: Opaque
data:
  AWS_ACCESS_KEY_ID: base64_encoded_access_key
  AWS_SECRET_ACCESS_KEY: base64_encoded_secret_key
  AWS_ENDPOINTS: base64_encoded_s3_endpoint  # Optional for S3-compatible
---
# Apply via Longhorn settings
# Backup Target: s3://bucket-name@region/
# Backup Target Credential Secret: aws-secret

# Configure backup target via kubectl
kubectl edit settings.longhorn.io backup-target -n longhorn-system
# Set value to: s3://my-backup-bucket@us-east-1/

kubectl edit settings.longhorn.io backup-target-credential-secret -n longhorn-system
# Set value to: aws-secret

Longhorn Volume Snapshots

# volumesnapshot-longhorn.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: longhorn-snapshot-vsc
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: driver.longhorn.io
deletionPolicy: Delete
parameters:
  type: snap
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: database-snapshot
  namespace: production
spec:
  volumeSnapshotClassName: longhorn-snapshot-vsc
  source:
    persistentVolumeClaimName: database-storage

Longhorn Disaster Recovery

# disaster-recovery-volume.yaml - Create volume from backup
apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: restored-volume
  namespace: longhorn-system
spec:
  fromBackup: "s3://bucket/backups/backup-abc123"
  numberOfReplicas: 3
  size: "100Gi"

# Restore from backup via PVC
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restored-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 100Gi
  dataSource:
    name: database-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
EOF

Longhorn Management Commands

# List Longhorn volumes
kubectl get volumes.longhorn.io -n longhorn-system

# Check volume details
kubectl describe volume.longhorn.io pvc-12345678-1234-1234-1234-123456789012 -n longhorn-system

# List Longhorn engines
kubectl get engines.longhorn.io -n longhorn-system

# List replicas
kubectl get replicas.longhorn.io -n longhorn-system

# Check Longhorn nodes
kubectl get nodes.longhorn.io -n longhorn-system

# View settings
kubectl get settings.longhorn.io -n longhorn-system

# Trigger manual backup
kubectl create -f - <<EOF
apiVersion: longhorn.io/v1beta2
kind: Backup
metadata:
  name: manual-backup-$(date +%Y%m%d-%H%M%S)
  namespace: longhorn-system
spec:
  snapshotName: snapshot-12345
  labels:
    type: manual
EOF

Volume Expansion

Expanding Volumes

# Check if StorageClass allows expansion
kubectl get sc fast-ssd -o jsonpath='{.allowVolumeExpansion}'

# Expand PVC
kubectl patch pvc web-storage -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

# Check expansion status
kubectl describe pvc web-storage | grep -A 5 Conditions

# For StatefulSet volumes, expand via patch
kubectl patch pvc data-mysql-0 -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

File System Expansion

# Most CSI drivers auto-expand the filesystem
# Some require pod restart or manual intervention

# Check PVC conditions for filesystem resize status
kubectl get pvc web-storage -o jsonpath='{.status.conditions[?(@.type=="FileSystemResizePending")]}'

# If filesystem resize is pending, restart pod
kubectl rollout restart deployment/web-server

# Or delete and recreate pod
kubectl delete pod web-server

Offline Expansion

# For volumes requiring offline expansion:

# 1. Scale down workload
kubectl scale deployment web-server --replicas=0

# 2. Expand PVC
kubectl patch pvc web-storage -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

# 3. Wait for expansion to complete
kubectl wait --for=condition=FileSystemResizeSuccessful pvc/web-storage --timeout=300s

# 4. Scale up workload
kubectl scale deployment web-server --replicas=3

Troubleshooting Storage Issues

Volume Mount Failures

# Check pod events
kubectl describe pod web-server

# Common issues in events:
# - FailedMount: Volume not attached
# - FailedAttachVolume: Volume attach timeout
# - MountVolume.SetUp failed: mount failed

# Check PVC binding
kubectl get pvc web-storage
# Status should be "Bound"

# Check PV status
kubectl get pv
kubectl describe pv pvc-12345678-1234-1234-1234-123456789012

# Check CSI driver pods
kubectl get pods -n kube-system | grep csi

# View CSI driver logs
kubectl logs -n kube-system csi-controller-pod -c csi-provisioner
kubectl logs -n kube-system csi-node-pod -c csi-driver

PVC Stuck in Pending

# Check PVC events
kubectl describe pvc web-storage

# Common causes:
# 1. No matching StorageClass
kubectl get sc

# 2. No available PVs for static provisioning
kubectl get pv

# 3. Insufficient resources in cloud provider
# Check controller logs

# 4. Volume binding mode is WaitForFirstConsumer but no pod scheduled
kubectl get pods -o wide

# 5. Topology constraints not met
kubectl describe pvc web-storage | grep -A 10 "Topology"

PV Stuck in Released

# View PV details
kubectl describe pv pvc-12345678-1234-1234-1234-123456789012

# Check claimRef
kubectl get pv pvc-12345678-1234-1234-1234-123456789012 -o yaml | grep -A 5 claimRef

# Remove claimRef to make Available
kubectl patch pv pvc-12345678-1234-1234-1234-123456789012 -p '{"spec":{"claimRef":null}}'

# Or edit directly
kubectl edit pv pvc-12345678-1234-1234-1234-123456789012
# Remove entire claimRef section

Volume Attach/Detach Issues

# Check volume attachments
kubectl get volumeattachment
kubectl describe volumeattachment csi-abc123

# Check node CSI driver
kubectl get csinodes
kubectl describe csinode worker-node-1

# Force detach (dangerous - ensure volume not in use)
kubectl delete volumeattachment csi-abc123

# Check attach/detach controller logs
kubectl logs -n kube-system kube-controller-manager-master -c kube-controller-manager | grep attachdetach

Snapshot Issues

# Check VolumeSnapshotClass
kubectl get volumesnapshotclass
kubectl describe volumesnapshotclass csi-snapclass

# Check snapshot status
kubectl get volumesnapshot
kubectl describe volumesnapshot db-snapshot-20231206

# Check snapshot content
kubectl get volumesnapshotcontent
kubectl describe volumesnapshotcontent snapcontent-12345

# Check snapshot controller logs
kubectl logs -n kube-system snapshot-controller

# Verify CSI driver supports snapshots
kubectl get csidriver ebs.csi.aws.com -o jsonpath='{.spec.volumeSnapshotDataSource}'

Storage Performance Issues

# Check PV/PVC for IOPS/throughput limits (cloud providers)
kubectl get pv pvc-12345678-1234-1234-1234-123456789012 -o yaml | grep -i iops
kubectl get pv pvc-12345678-1234-1234-1234-123456789012 -o yaml | grep -i throughput

# Check for I/O throttling in pod
kubectl exec -it web-server -- iostat -x 1

# Test disk performance
kubectl exec -it web-server -- dd if=/dev/zero of=/data/testfile bs=1M count=1024 oflag=direct

# For Ceph/Rook
kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph osd perf

# For Longhorn
# Check volume metrics in Longhorn UI or via:
kubectl get volumes.longhorn.io -n longhorn-system -o wide

CSI Driver Debugging

# Enable debug logging for CSI driver
# Edit CSI driver deployment/daemonset
kubectl edit deployment csi-controller -n kube-system
# Add environment variable:
# - name: LOG_LEVEL
#   value: "debug"

# Check CSI driver registration
kubectl describe csinode worker-node-1

# Verify CSI socket
kubectl exec -it -n kube-system csi-node-pod -- ls -la /var/lib/kubelet/plugins/*/csi.sock

# Test CSI driver manually (advanced)
kubectl exec -it -n kube-system csi-node-pod -- csc node get-info --endpoint unix:///var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock

Quick Reference

Common kubectl Commands

# Storage Classes
kubectl get sc
kubectl describe sc fast-ssd
kubectl patch sc fast-ssd -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# PersistentVolumes
kubectl get pv
kubectl describe pv pvc-12345678
kubectl patch pv pvc-12345678 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

# PersistentVolumeClaims
kubectl get pvc
kubectl get pvc -A
kubectl describe pvc web-storage
kubectl patch pvc web-storage -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
kubectl delete pvc web-storage

# Volume Snapshots
kubectl get volumesnapshot
kubectl get volumesnapshotclass
kubectl get volumesnapshotcontent
kubectl describe volumesnapshot db-snapshot

# CSI Drivers
kubectl get csidrivers
kubectl get csinodes
kubectl get volumeattachment

# Debugging
kubectl describe pod web-server | grep -A 10 Events
kubectl logs -n kube-system csi-controller-pod -c csi-provisioner

Access Modes Quick Reference

Mode	Abbreviation	Description	Use Case
ReadWriteOnce	RWO	Single node R/W	Databases, single-instance apps
ReadOnlyMany	ROX	Multiple nodes R/O	Static content distribution
ReadWriteMany	RWX	Multiple nodes R/W	Shared file storage, CMS
ReadWriteOncePod	RWOP	Single pod R/W (GA k8s 1.29)	Strict single-pod access

Reclaim Policies Quick Reference

Policy	Action on PVC Deletion	Use Case
Delete	Volume deleted	Development, ephemeral data
Retain	Volume retained	Production, manual backup
Recycle	Volume scrubbed (deprecated)	Legacy only

Volume Binding Modes

Mode	Behaviour	Use Case
Immediate	Provision on PVC creation	Pre-provisioned volumes, testing
WaitForFirstConsumer	Provision when pod scheduled	Zone-aware provisioning, cost optimization

Storage Provider Comparison

Provider	Block (RWO)	File (RWX)	Object	Snapshots	Cloning	Expansion
AWS EBS	✓	✗	✗	✓	✓	✓
AWS EFS	✗	✓	✗	✗	✗	✓
GCP PD	✓	✗	✗	✓	✓	✓
GCP Filestore	✗	✓	✗	✓	✗	✓
Azure Disk	✓	✗	✗	✓	✓	✓
Azure Files	✗	✓	✗	✓	✗	✓
Rook/Ceph RBD	✓	✗	✗	✓	✓	✓
Rook/Ceph FS	✗	✓	✗	✓	✓	✓
Longhorn	✓	✓*	✗	✓	✓	✓
Local Path	✓	✗	✗	✗	✗	✗
NFS	✗	✓	✗	Varies	Varies	Varies

*Longhorn RWX is native since v1.6 — a built-in Share Manager pod (internal NFSv4) serves each RWX volume; no external NFS provisioner required (earlier versions needed one).

Common Issues and Solutions

Provisioning Issues

Issue	Cause	Solution
PVC stuck in Pending	StorageClass not found	Verify StorageClass exists: kubectl get sc
PVC stuck in Pending	No nodes in correct topology	Check volumeBindingMode and node zones
PVC stuck in Pending	Quota exceeded	Check namespace ResourceQuota and cloud provider limits
PVC stuck in Pending	CSI driver not running	Check CSI controller: kubectl get pods -n kube-system | grep csi
Volume created in wrong zone	VolumeBindingMode: Immediate	Change to WaitForFirstConsumer
Static PV not binding	Selector/labels mismatch	Verify PVC selector matches PV labels

Mounting Issues

Issue	Cause	Solution
FailedMount: volume not attached	VolumeAttachment stuck	Check: kubectl get volumeattachment
FailedMount: timeout waiting for volume	CSI node plugin not running	Check CSI node pods on target node
MountVolume.SetUp failed	Filesystem corruption	Check volume in cloud console, consider restore from snapshot
Multi-attach error	Volume already attached to different node	Ensure pod is deleted from previous node first
Permission denied on mount	Security context mismatch	Check pod securityContext and fsGroup
Read-only filesystem	Volume mounted read-only	Check access mode and mount options

Snapshot Issues

Issue	Cause	Solution
Snapshot stuck in Pending	VolumeSnapshotClass not found	Verify: kubectl get volumesnapshotclass
Snapshot creation failed	CSI driver doesn't support snapshots	Check CSI driver capabilities
Restore from snapshot failed	Snapshot in different region/zone	Ensure snapshot and PVC in same region
Snapshot deleted but backend exists	DeletionPolicy: Retain	Manually delete snapshot from cloud provider

Performance Issues

Issue	Cause	Solution
Slow I/O performance	Wrong volume type	Use SSD-backed storage for databases (gp3, pd-ssd)
Throttled IOPS	Volume too small	Increase volume size or IOPS allocation
High latency	Volume in different AZ	Use WaitForFirstConsumer binding mode
Degraded Ceph performance	Insufficient OSD resources	Add more OSDs or increase OSD resources
Longhorn slow writes	Too many replicas	Reduce replica count for non-critical data

Rook/Ceph Specific

Issue	Cause	Solution
OSD pods not starting	No available disks	Verify disks are empty and unmounted
Ceph health WARN	Clock skew	Synchronise NTP across all nodes
Mon quorum lost	Majority of monitors down	Restart monitor pods, check node health
PG inconsistent	Data corruption	Run ceph pg repair <pgid>
Slow requests	Network latency or disk I/O	Check network and disk performance

Longhorn Specific

Issue	Cause	Solution
Volume degraded	Replica failure	Check replica status in Longhorn UI
Backup failed	Invalid backup target	Verify S3/NFS credentials and connectivity
Replica scheduling failed	Insufficient disk space	Add more nodes or increase disk space
Engine image incompatible	Version mismatch	Upgrade/downgrade engine image
Volume stuck in attaching	Old mount leftover	SSH to node and manually umount

General Debugging Steps

# 1. Check pod events
kubectl describe pod <pod-name>

# 2. Check PVC status
kubectl get pvc <pvc-name>
kubectl describe pvc <pvc-name>

# 3. Check PV status
kubectl get pv
kubectl describe pv <pv-name>

# 4. Check StorageClass
kubectl get sc
kubectl describe sc <sc-name>

# 5. Check CSI driver
kubectl get csidrivers
kubectl get pods -n kube-system | grep csi

# 6. Check CSI driver logs
kubectl logs -n kube-system <csi-controller-pod> -c csi-provisioner
kubectl logs -n kube-system <csi-node-pod> -c csi-driver

# 7. Check VolumeAttachment
kubectl get volumeattachment
kubectl describe volumeattachment <attachment-name>

# 8. For snapshots
kubectl describe volumesnapshot <snapshot-name>
kubectl get volumesnapshotcontent

# 9. Storage-specific debugging
# Rook/Ceph
kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status

# Longhorn
kubectl get volumes.longhorn.io -n longhorn-system
# Access UI at port-forward svc/longhorn-frontend

# Local Path
kubectl logs -n local-path-storage deployment/local-path-provisioner