Available for day contractsFrom 21st September I have availability for day and half day contracts. Please contact for more information.

Contact →
mikepreston.org

Velero

Kubernetes backup and restore tool for cluster resources and persistent volumes.

Velero

Kubernetes backup and restore tool for cluster resources and persistent volumes.

Overview

Velero is an open-source tool for backing up and restoring Kubernetes cluster resources and persistent volumes. It supports scheduled backups, on-demand backups, and migration of resources between clusters. Velero uses cloud provider object storage (S3, GCS, Azure Blob) as backup destinations and supports volume snapshots via native cloud APIs or Kopia-backed file-level backups via the node-agent DaemonSet. CSI snapshot support is built into core Velero since v1.14 (no separate plugin needed).

Cloud StorageKubernetes ClusterWorkloadsDeploymentsPVCsConfigMapsSecretsVelero ServerBackupStorageLocationVolumeSnapshotLocationObject StorageVolume SnapshotsCloud StorageKubernetes ClusterWorkloadsDeploymentsPVCsConfigMapsSecretsVelero ServerBackupStorageLocationVolumeSnapshotLocationObject StorageVolume Snapshots

Backup and Restore Commands

Velero CLI provides commands for creating, managing, and restoring backups of cluster resources.

Key Concepts

  • Backup: Point-in-time copy of cluster resources and volume data
  • Restore: Process of recreating resources from a backup
  • BackupStorageLocation: Defines where backups are stored (cloud object storage)
  • VolumeSnapshotLocation: Defines where volume snapshots are stored
velero backup createCapture ResourcesSnapshot VolumesUpload to StorageBackup Completevelero restorecreateDownload BackupRestore VolumesCreate ResourcesRestore Completevelero backup createCapture ResourcesSnapshot VolumesUpload to StorageBackup Completevelero restorecreateDownload BackupRestore VolumesCreate ResourcesRestore Complete

Creating Backups

# Create backup of entire cluster
velero backup create full-backup

# Create backup with description
velero backup create daily-backup --description "Daily production backup"

# Backup specific namespace
velero backup create app-backup --include-namespaces production

# Backup multiple namespaces
velero backup create multi-ns-backup --include-namespaces app1,app2,monitoring

# Backup excluding namespaces
velero backup create cluster-backup --exclude-namespaces kube-system,velero

# Backup specific resources by label
velero backup create labelled-backup --selector app=nginx

# Backup specific resource types
velero backup create deployments-backup --include-resources deployments,services,configmaps

# Backup excluding resource types
velero backup create no-secrets-backup --exclude-resources secrets

# Backup with TTL (time to live)
velero backup create temp-backup --ttl 72h

# Backup excluding volumes (metadata only)
velero backup create metadata-backup --snapshot-volumes=false

# Wait for backup completion
velero backup create production-backup --wait

Managing Backups

# List all backups
velero backup get

# Describe backup details
velero backup describe full-backup

# Describe with volume details
velero backup describe full-backup --details

# View backup logs
velero backup logs full-backup

# Download backup locally
velero backup download full-backup

# Delete backup
velero backup delete full-backup

# Delete backup without confirmation
velero backup delete full-backup --confirm

# Delete multiple backups
velero backup delete backup1 backup2 backup3 --confirm

Creating Restores

# Restore from backup (full restore)
velero restore create --from-backup full-backup

# Restore with custom name
velero restore create prod-restore --from-backup full-backup

# Restore specific namespaces
velero restore create --from-backup full-backup --include-namespaces production

# Restore to different namespace
velero restore create --from-backup full-backup \
  --namespace-mappings old-namespace:new-namespace

# Restore excluding resources
velero restore create --from-backup full-backup --exclude-resources persistentvolumeclaims

# Restore specific resources by label
velero restore create --from-backup full-backup --selector app=nginx

# Restore without volumes
velero restore create --from-backup full-backup --restore-volumes=false

# Restore with existing resource policy (skip/update)
velero restore create --from-backup full-backup --existing-resource-policy update

# Wait for restore completion
velero restore create --from-backup full-backup --wait

Managing Restores

# List all restores
velero restore get

# Describe restore details
velero restore describe prod-restore

# View restore logs
velero restore logs prod-restore

# Delete restore record
velero restore delete prod-restore --confirm

Examples

Full Cluster Backup

# Create comprehensive backup
velero backup create cluster-$(date +%Y%m%d) \
  --exclude-namespaces kube-system \
  --snapshot-volumes=true \
  --ttl 720h \
  --description "Weekly full cluster backup"

# Verify backup completed
velero backup describe cluster-$(date +%Y%m%d)

Namespace Migration

# Backup source namespace
velero backup create migration-backup --include-namespaces production

# Restore to new namespace on same or different cluster
velero restore create migration-restore \
  --from-backup migration-backup \
  --namespace-mappings production:production-v2

Scheduling Backups

Velero supports scheduled backups using cron expressions for automated backup strategies.

Key Concepts

  • Schedule: Defines recurring backup creation using cron syntax
  • Retention: Controlled via TTL (time to live) on backups
  • Schedule Status: Shows last backup time and next scheduled run
Create ScheduleCron TriggerBackup SuccessBackup ErrorWait for NextWait for NextScheduledRunningCompletedFailedCreate ScheduleCron TriggerBackup SuccessBackup ErrorWait for NextWait for NextScheduledRunningCompletedFailed

Creating Schedules

# Create hourly backup schedule
velero schedule create hourly-backup --schedule="0 * * * *"

# Create daily backup at 2 AM UTC
velero schedule create daily-backup --schedule="0 2 * * *"

# Create weekly backup (Sunday at midnight)
velero schedule create weekly-backup --schedule="0 0 * * 0"

# Create schedule with namespace filter
velero schedule create prod-backup \
  --schedule="0 */6 * * *" \
  --include-namespaces production

# Create schedule with TTL
velero schedule create daily-backup \
  --schedule="0 2 * * *" \
  --ttl 168h

# Create schedule excluding resources
velero schedule create app-backup \
  --schedule="0 3 * * *" \
  --include-namespaces app \
  --exclude-resources events,pods

# Create schedule with label selector
velero schedule create critical-apps \
  --schedule="0 * * * *" \
  --selector tier=critical

# Schedule with volume snapshots disabled
velero schedule create metadata-only \
  --schedule="0 4 * * *" \
  --snapshot-volumes=false

Managing Schedules

# List all schedules
velero schedule get

# Describe schedule details
velero schedule describe daily-backup

# Pause schedule
velero schedule pause daily-backup

# Unpause schedule
velero schedule unpause daily-backup

# Delete schedule (keeps existing backups)
velero schedule delete daily-backup --confirm

# Trigger immediate backup from schedule
velero backup create --from-schedule daily-backup

Cron Expression Reference

Expression Description
0 * * * * Every hour
0 */6 * * * Every 6 hours
0 0 * * * Daily at midnight
0 2 * * * Daily at 2 AM
0 0 * * 0 Weekly on Sunday
0 0 1 * * Monthly on 1st
*/15 * * * * Every 15 minutes

Examples

Multi-tier Backup Strategy

# Tier 1: Hourly backups of critical namespaces (24h retention)
velero schedule create hourly-critical \
  --schedule="0 * * * *" \
  --include-namespaces production,payments \
  --ttl 24h

# Tier 2: Daily backups of all apps (7 day retention)
velero schedule create daily-apps \
  --schedule="0 2 * * *" \
  --exclude-namespaces kube-system,velero \
  --ttl 168h

# Tier 3: Weekly full backups (30 day retention)
velero schedule create weekly-full \
  --schedule="0 3 * * 0" \
  --ttl 720h

Backup Rotation

# Example: Keep last 7 daily, 4 weekly, and 3 monthly backups
# Daily backups with 7-day TTL
velero schedule create daily \
  --schedule="0 1 * * *" \
  --ttl 168h

# Weekly backups with 28-day TTL
velero schedule create weekly \
  --schedule="0 2 * * 0" \
  --ttl 672h

# Monthly backups with 90-day TTL
velero schedule create monthly \
  --schedule="0 3 1 * *" \
  --ttl 2160h

File System Backup (Node Agent)

Velero's file system backup (FSB) copies volume contents at the file level via a node-agent DaemonSet. This is useful when cloud provider snapshots are unavailable or cross-provider restores are needed. Kopia has been the default uploader since v1.10; the Restic uploader was deprecated in v1.15 and removed in v1.17 (new Restic backups disabled, existing ones restorable through v1.18). Current installs are Kopia-only.

Key Concepts

  • Node Agent: DaemonSet that runs FSB on each node (replaces the old Restic DaemonSet)
  • Kopia: Default and current uploader engine (Restic path removed in v1.17)
  • File System Backup: Copies actual files rather than volume snapshots
  • Pod Annotations: Control which volumes to backup
Native SnapshotRestic/KopiaPod with PVCBackup TypeCloud Provider APIVelero FSBVolume SnapshotFile-level BackupSnapshot StorageObject StorageNative SnapshotRestic/KopiaPod with PVCBackup TypeCloud Provider APIVelero FSBVolume SnapshotFile-level BackupSnapshot StorageObject Storage

Installation with Node Agent (Kopia)

# Install Velero with node agent for file system backup (Kopia is the default uploader)
velero install \
  --provider aws \
  --bucket velero-backups \
  --secret-file ./credentials \
  --use-node-agent

# Install with both snapshots and file-level backup
velero install \
  --provider aws \
  --bucket velero-backups \
  --secret-file ./credentials \
  --use-node-agent \
  --default-volumes-to-fs-backup

Annotating Pods for Volume Backup

# Include specific volumes for file-level backup
apiVersion: v1
kind: Pod
metadata:
  name: app
  annotations:
    backup.velero.io/backup-volumes: data,config
spec:
  containers:
  - name: app
    image: myapp:v1
    volumeMounts:
    - name: data
      mountPath: /data
    - name: config
      mountPath: /etc/config
    - name: cache
      mountPath: /cache
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-pvc
  - name: config
    persistentVolumeClaim:
      claimName: config-pvc
  - name: cache
    emptyDir: {}
# Exclude specific volumes from backup
apiVersion: v1
kind: Pod
metadata:
  name: app
  annotations:
    backup.velero.io/backup-volumes-excludes: cache,tmp
spec:
  # ... pod spec

File System Backup Commands

# Create backup with default file-level backup for all volumes
velero backup create fs-backup --default-volumes-to-fs-backup

# Check node agent (formerly restic) pods
kubectl get pods -n velero -l name=node-agent

# View file system backup progress
velero backup describe fs-backup --details

# List pod volume backups
kubectl get podvolumebackups -n velero

# Describe specific pod volume backup
kubectl describe podvolumebackup <name> -n velero

# List pod volume restores
kubectl get podvolumerestores -n velero

Repository Management

# List backup repositories (BackupRepository objects)
velero repo get

# Inspect the Kopia repository status (runs in a node-agent pod; exec ds/ picks one)
kubectl -n velero exec daemonset/node-agent -- kopia repository status
# Repo maintenance (incl. clearing stale locks) runs as a scheduled job, not a manual unlock

# Generate a debug bundle with logs
velero debug --output ./velero-debug.tar.gz

Examples

StatefulSet with Volume Backup

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: database
spec:
  serviceName: database
  replicas: 3
  template:
    metadata:
      annotations:
        # Backup data volume with Restic/Kopia
        backup.velero.io/backup-volumes: data
    spec:
      containers:
      - name: postgres
        image: postgres:15
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

Backup with Mixed Volume Types

# Some volumes use snapshots, others use file-level backup
velero backup create mixed-backup \
  --include-namespaces production \
  --snapshot-volumes=true \
  --default-volumes-to-fs-backup=false

# Volumes with annotations will use file-level backup
# Volumes without annotations will use cloud snapshots

CSI Volume Snapshots

CSI snapshots back up volumes via the storage driver's own snapshot API (the snapshot.storage.k8s.io CRDs), rather than cloud-provider volume APIs or file-level copy. The CSI plugin is built into core Velero since v1.14, but the integration is gated behind a feature flag — EnableCSI must be set at install time (still required as of v1.18; it is not on by default).

Key Concepts

  • EnableCSI feature flag: Activates the CSI BackupItemAction; without it, PVCs backed by CSI drivers fall back to file-system backup or are skipped.
  • VolumeSnapshotClass label: Velero selects the VolumeSnapshotClass matching the PVC's CSI driver that carries the velero.io/csi-volumesnapshot-class: "true" label.
  • CSI snapshot data movement: by default a CSI snapshot stays in the cluster's snapshot storage (tied to that cluster/region). --snapshot-move-data copies the snapshot data out to the object-storage backup location via the node-agent (Kopia), making backups portable for cross-cluster/cross-region restore.

Enabling CSI Snapshots

# Install with the CSI feature flag (object-storage plugin still required)
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.1 \
  --bucket velero-backups \
  --backup-location-config region=eu-west-1 \
  --features=EnableCSI \
  --secret-file ./credentials-velero

# On an existing install, add the flag to the Velero deployment
kubectl -n velero patch deployment velero --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--features=EnableCSI"}]'

Labelling the VolumeSnapshotClass

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-hostpath-snapclass
  labels:
    velero.io/csi-volumesnapshot-class: "true"   # Velero picks this class for the matching driver
driver: hostpath.csi.k8s.io
deletionPolicy: Retain   # Retain so the snapshot survives VolumeSnapshot deletion

Backing Up and Restoring with CSI

# CSI snapshots are used automatically once EnableCSI is set and a labelled class exists
velero backup create csi-backup --include-namespaces production

# Move snapshot data to object storage (portable, cluster-independent)
velero backup create csi-portable \
  --include-namespaces production \
  --snapshot-move-data

# Inspect the CSI snapshots a backup produced
velero backup describe csi-backup --details

# Restore as usual — Velero recreates PVCs from the CSI snapshots
velero restore create --from-backup csi-backup

CSI snapshots and file-system backup are mutually exclusive per volume: a PVC selected for FS backup (via annotation or --default-volumes-to-fs-backup) is not also CSI-snapshotted. Choose snapshots for speed and storage-native efficiency, FS backup for drivers without snapshot support or when you need filesystem-level portability without the data mover.

Cloud Provider Configuration

Velero supports multiple cloud providers for storing backups and volume snapshots.

AWS Configuration

# Create credentials file
cat > credentials-velero <<EOF
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
EOF

# Install Velero for AWS
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.1 \
  --bucket velero-backups \
  --backup-location-config region=eu-west-1 \
  --snapshot-location-config region=eu-west-1 \
  --secret-file ./credentials-velero

AWS IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::velero-backups/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::velero-backups"
        }
    ]
}

GCP Configuration

# Create service account
gcloud iam service-accounts create velero \
  --display-name "Velero service account"

# Assign roles
PROJECT_ID=$(gcloud config get-value project)
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:velero@$PROJECT_ID.iam.gserviceaccount.com \
  --role roles/compute.storageAdmin

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:velero@$PROJECT_ID.iam.gserviceaccount.com \
  --role roles/storage.admin

# Create key file
gcloud iam service-accounts keys create credentials-velero \
  --iam-account velero@$PROJECT_ID.iam.gserviceaccount.com

# Install Velero for GCP
velero install \
  --provider gcp \
  --plugins velero/velero-plugin-for-gcp:v1.14.1 \
  --bucket velero-backups \
  --secret-file ./credentials-velero

Azure Configuration

# Set variables
AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
AZURE_RESOURCE_GROUP=velero-backups
AZURE_STORAGE_ACCOUNT=velerobackups

# Create resource group
az group create -n $AZURE_RESOURCE_GROUP --location uksouth

# Create storage account
az storage account create \
  --name $AZURE_STORAGE_ACCOUNT \
  --resource-group $AZURE_RESOURCE_GROUP \
  --sku Standard_GRS

# Create blob container
az storage container create \
  --name velero \
  --account-name $AZURE_STORAGE_ACCOUNT

# Create service principal
AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac \
  --name velero \
  --role Contributor \
  --query password -o tsv)
AZURE_CLIENT_ID=$(az ad sp show --id http://velero --query appId -o tsv)

# Create credentials file
cat > credentials-velero <<EOF
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

# Install Velero for Azure
velero install \
  --provider azure \
  --plugins velero/velero-plugin-for-microsoft-azure:v1.14.1 \
  --bucket velero \
  --secret-file ./credentials-velero \
  --backup-location-config \
    resourceGroup=$AZURE_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT \
  --snapshot-location-config \
    resourceGroup=$AZURE_RESOURCE_GROUP,subscriptionId=$AZURE_SUBSCRIPTION_ID

Multiple Backup Locations

# Create additional backup storage location
velero backup-location create secondary \
  --provider aws \
  --bucket velero-secondary \
  --config region=us-west-2 \
  --access-mode ReadWrite

# Create backup to specific location
velero backup create backup-west --storage-location secondary

# List backup locations
velero backup-location get

# Set backup location as default
velero backup-location set primary --default

Volume Snapshot Locations

# Create volume snapshot location
velero snapshot-location create aws-snapshots \
  --provider aws \
  --config region=eu-west-1

# Create backup with specific snapshot location
velero backup create backup-eu --volume-snapshot-locations aws-snapshots

# List snapshot locations
velero snapshot-location get

Common Use Cases

Disaster Recovery

Scheduled BackupsDisaster OccursSame ClusterNew ClusterProduction ClusterObject StorageRecovery OptionsRestore toProductionRestore to DRClusterVerify ServicesResume OperationsScheduled BackupsDisaster OccursSame ClusterNew ClusterProduction ClusterObject StorageRecovery OptionsRestore toProductionRestore to DRClusterVerify ServicesResume Operations

DR Backup Strategy

# Create DR backup schedule (every 4 hours)
velero schedule create disaster-recovery \
  --schedule="0 */4 * * *" \
  --ttl 168h \
  --exclude-namespaces kube-system,velero \
  --snapshot-volumes=true

# Pre-disaster validation
velero backup describe $(velero backup get -o json | jq -r '.items[-1].metadata.name')

# Restore to same cluster after disaster
velero restore create dr-restore \
  --from-backup disaster-recovery-20240115120000 \
  --wait

# Verify restoration
kubectl get deployments -A
kubectl get pvc -A

Cross-cluster DR

# On source cluster: create backup
velero backup create migration-backup \
  --exclude-namespaces kube-system \
  --wait

# On target cluster: configure same backup location
velero backup-location create source-backups \
  --provider aws \
  --bucket source-cluster-backups \
  --config region=eu-west-1 \
  --access-mode ReadOnly

# On target cluster: sync backups
velero backup get

# On target cluster: restore
velero restore create dr-restore \
  --from-backup migration-backup

Cluster Migration

Target ClusterObject StorageSource ClusterTarget ClusterObject StorageSource ClusterInclude all namespacesMap namespaces if neededvelero backup createConfigure backup locationvelero backup get (sync)velero restore createVerify workloadsUpdate DNS/IngressTarget ClusterObject StorageSource ClusterTarget ClusterObject StorageSource ClusterInclude all namespacesMap namespaces if neededvelero backup createConfigure backup locationvelero backup get (sync)velero restore createVerify workloadsUpdate DNS/Ingress

Migration Steps

# Step 1: On source cluster - create comprehensive backup
velero backup create cluster-migration \
  --exclude-namespaces kube-system,kube-public,velero \
  --include-cluster-resources=true \
  --default-volumes-to-fs-backup \
  --wait

# Step 2: Verify backup completeness
velero backup describe cluster-migration --details

# Step 3: On target cluster - install Velero with same storage
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.1 \
  --bucket source-cluster-backups \
  --backup-location-config region=eu-west-1 \
  --secret-file ./credentials-velero

# Step 4: Sync and verify backup is visible
velero backup get

# Step 5: Perform restore
velero restore create migration-restore \
  --from-backup cluster-migration \
  --include-cluster-resources=true \
  --wait

# Step 6: Verify resources
kubectl get namespaces
kubectl get deployments -A
kubectl get services -A
kubectl get pvc -A

Namespace Migration to New Name

# Backup specific namespace
velero backup create ns-migration --include-namespaces old-app

# Restore to new namespace name
velero restore create ns-restore \
  --from-backup ns-migration \
  --namespace-mappings old-app:new-app

Pre-upgrade Backup

# Before Kubernetes upgrade or major changes
velero backup create pre-upgrade-$(date +%Y%m%d-%H%M) \
  --ttl 720h \
  --description "Pre-upgrade backup before K8s 1.28 upgrade" \
  --wait

# After upgrade, if issues occur
velero restore create rollback \
  --from-backup pre-upgrade-20240115-1430

Application-level Backup

# Backup specific application stack
velero backup create myapp-backup \
  --include-namespaces myapp \
  --include-resources \
    deployments,services,configmaps,secrets,persistentvolumeclaims \
  --selector app.kubernetes.io/name=myapp \
  --snapshot-volumes=true

# Restore application
velero restore create myapp-restore \
  --from-backup myapp-backup \
  --include-namespaces myapp

Quick Reference

Backup Commands

velero backup create <name>                    # Create backup
velero backup create <name> --include-namespaces <ns>  # Backup namespace
velero backup create <name> --from-schedule <sched>    # From schedule
velero backup get                              # List backups
velero backup describe <name> --details        # Backup details
velero backup logs <name>                      # View logs
velero backup delete <name> --confirm          # Delete backup

Restore Commands

velero restore create --from-backup <backup>   # Create restore
velero restore create --from-backup <backup> --namespace-mappings old:new
velero restore get                             # List restores
velero restore describe <name>                 # Restore details
velero restore logs <name>                     # View logs

Schedule Commands

velero schedule create <name> --schedule="0 2 * * *"  # Create schedule
velero schedule get                            # List schedules
velero schedule describe <name>                # Schedule details
velero schedule pause <name>                   # Pause schedule
velero schedule unpause <name>                 # Resume schedule
velero schedule delete <name> --confirm        # Delete schedule

Storage Location Commands

velero backup-location get                     # List backup locations
velero backup-location create <name> --provider <p> --bucket <b>
velero snapshot-location get                   # List snapshot locations

Useful Flags

Flag Description
--include-namespaces Backup specific namespaces
--exclude-namespaces Exclude namespaces
--include-resources Backup specific resource types
--exclude-resources Exclude resource types
--selector Backup resources matching labels
--ttl Backup retention period
--snapshot-volumes Enable volume snapshots
--default-volumes-to-fs-backup Use file-level backup
--snapshot-move-data Move CSI snapshot data to object storage (portable backups)
--wait Wait for completion
--namespace-mappings Remap namespaces during restore

Common Issues and Solutions

Backup Stuck in Progress

Symptoms: Backup shows InProgress status for extended period

Solutions:

# Check Velero pod logs
kubectl logs -n velero deploy/velero

# Check for stuck volume snapshots
kubectl get volumesnapshots -A

# Cancel stuck backup
velero backup delete stuck-backup --confirm

# Check node agent (Restic) pods
kubectl get pods -n velero -l name=node-agent
kubectl logs -n velero -l name=node-agent

Volume Snapshot Failures

Symptoms: Backup completes but volumes show errors

Causes and solutions:

  1. Missing CSI driver - Install appropriate CSI snapshot controller

    kubectl get volumesnapshotclasses
    
  2. Wrong snapshot location - Verify VolumeSnapshotLocation configuration

    velero snapshot-location get
    
  3. Insufficient permissions - Check cloud provider IAM roles

Restore Partial Failures

Symptoms: Some resources fail to restore

Debugging steps:

# Check restore details
velero restore describe my-restore --details

# View restore logs
velero restore logs my-restore

# Check for specific errors
velero restore logs my-restore | grep -i error

# Common causes:
# - Resources already exist (use --existing-resource-policy update)
# - Missing CRDs (restore cluster resources first)
# - Namespace doesn't exist

Restic/Kopia Backup Failures

Symptoms: Pod volume backups fail

Solutions:

# Check node-agent pods are running
kubectl get pods -n velero -l name=node-agent

# Check node-agent logs
kubectl logs -n velero daemonset/node-agent

# Verify pod annotations
kubectl get pod <name> -o jsonpath='{.metadata.annotations}'

# Check for repository lock issues
velero repo get

Backup Storage Location Unavailable

Symptoms: BackupStorageLocation shows Unavailable

Solutions:

# Check BSL status
velero backup-location get

# Verify credentials
kubectl get secret -n velero cloud-credentials -o yaml

# Check connectivity to storage
kubectl exec -n velero deploy/velero -- \
  aws s3 ls s3://velero-backups/

# Recreate backup location
velero backup-location delete default --confirm
velero backup-location create default \
  --provider aws \
  --bucket velero-backups \
  --config region=eu-west-1

Cross-cluster Restore Issues

Symptoms: Resources fail during restore to different cluster

Solutions:

  1. StorageClass mismatch - Create matching StorageClasses or use ConfigMaps

    velero restore create --from-backup migration \
      --restore-volumes=false  # Restore without volumes first
    
  2. Cluster-scoped resources - Exclude or include carefully

    velero restore create --from-backup migration \
      --include-cluster-resources=false
    
  3. Service accounts - May need to be recreated with new tokens

Scheduled Backup Not Running

Symptoms: Schedule exists but no backups created

Solutions:

# Check schedule status
velero schedule describe my-schedule

# Verify cron expression
# Use https://crontab.guru to validate

# Check if schedule is paused
velero schedule get

# Unpause if needed
velero schedule unpause my-schedule

# Check Velero controller logs
kubectl logs -n velero deploy/velero | grep schedule

Related Topics

The following topics complement Velero knowledge and are commonly used together:

  • Kubernetes - Core platform for container orchestration; understanding cluster resources is essential for effective backups
  • Helm - Package manager often used for Velero installation; simplifies deployment and configuration
  • ArgoCD - GitOps tool for managing Velero configurations as code; automates backup policy deployment
  • AWS/GCP/Azure - Cloud providers for backup storage and volume snapshots; IAM and storage configuration is critical
  • Prometheus - Monitor Velero backup success/failure metrics; alerting on backup issues
  • Cert-Manager - Often backed up alongside applications; TLS certificates need protection