Velero
Kubernetes backup and restore tool for cluster resources and persistent volumes.
Velero
Kubernetes backup and restore tool for cluster resources and persistent volumes.
Overview
Velero is an open-source tool for backing up and restoring Kubernetes cluster resources and persistent volumes. It supports scheduled backups, on-demand backups, and migration of resources between clusters. Velero uses cloud provider object storage (S3, GCS, Azure Blob) as backup destinations and supports volume snapshots via native cloud APIs or Kopia-backed file-level backups via the node-agent DaemonSet. CSI snapshot support is built into core Velero since v1.14 (no separate plugin needed).
graph TB
subgraph Kubernetes Cluster
VS[Velero Server]
BSL[BackupStorageLocation]
VSL[VolumeSnapshotLocation]
subgraph Workloads
D[Deployments]
PVC[PVCs]
CM[ConfigMaps]
S[Secrets]
end
end
subgraph Cloud Storage
S3[(Object Storage)]
SNAP[(Volume Snapshots)]
end
VS --> BSL
VS --> VSL
VS --> Workloads
BSL --> S3
VSL --> SNAP
Backup and Restore Commands
Velero CLI provides commands for creating, managing, and restoring backups of cluster resources.
Key Concepts
- Backup: Point-in-time copy of cluster resources and volume data
- Restore: Process of recreating resources from a backup
- BackupStorageLocation: Defines where backups are stored (cloud object storage)
- VolumeSnapshotLocation: Defines where volume snapshots are stored
flowchart LR
A[velero backup create] --> B[Capture Resources]
B --> C[Snapshot Volumes]
C --> D[Upload to Storage]
D --> E[Backup Complete]
F[velero restore create] --> G[Download Backup]
G --> H[Restore Volumes]
H --> I[Create Resources]
I --> J[Restore Complete]
Creating Backups
# Create backup of entire cluster
velero backup create full-backup
# Create backup with description
velero backup create daily-backup --description "Daily production backup"
# Backup specific namespace
velero backup create app-backup --include-namespaces production
# Backup multiple namespaces
velero backup create multi-ns-backup --include-namespaces app1,app2,monitoring
# Backup excluding namespaces
velero backup create cluster-backup --exclude-namespaces kube-system,velero
# Backup specific resources by label
velero backup create labelled-backup --selector app=nginx
# Backup specific resource types
velero backup create deployments-backup --include-resources deployments,services,configmaps
# Backup excluding resource types
velero backup create no-secrets-backup --exclude-resources secrets
# Backup with TTL (time to live)
velero backup create temp-backup --ttl 72h
# Backup excluding volumes (metadata only)
velero backup create metadata-backup --snapshot-volumes=false
# Wait for backup completion
velero backup create production-backup --wait
Managing Backups
# List all backups
velero backup get
# Describe backup details
velero backup describe full-backup
# Describe with volume details
velero backup describe full-backup --details
# View backup logs
velero backup logs full-backup
# Download backup locally
velero backup download full-backup
# Delete backup
velero backup delete full-backup
# Delete backup without confirmation
velero backup delete full-backup --confirm
# Delete multiple backups
velero backup delete backup1 backup2 backup3 --confirm
Creating Restores
# Restore from backup (full restore)
velero restore create --from-backup full-backup
# Restore with custom name
velero restore create prod-restore --from-backup full-backup
# Restore specific namespaces
velero restore create --from-backup full-backup --include-namespaces production
# Restore to different namespace
velero restore create --from-backup full-backup \
--namespace-mappings old-namespace:new-namespace
# Restore excluding resources
velero restore create --from-backup full-backup --exclude-resources persistentvolumeclaims
# Restore specific resources by label
velero restore create --from-backup full-backup --selector app=nginx
# Restore without volumes
velero restore create --from-backup full-backup --restore-volumes=false
# Restore with existing resource policy (skip/update)
velero restore create --from-backup full-backup --existing-resource-policy update
# Wait for restore completion
velero restore create --from-backup full-backup --wait
Managing Restores
# List all restores
velero restore get
# Describe restore details
velero restore describe prod-restore
# View restore logs
velero restore logs prod-restore
# Delete restore record
velero restore delete prod-restore --confirm
Examples
Full Cluster Backup
# Create comprehensive backup
velero backup create cluster-$(date +%Y%m%d) \
--exclude-namespaces kube-system \
--snapshot-volumes=true \
--ttl 720h \
--description "Weekly full cluster backup"
# Verify backup completed
velero backup describe cluster-$(date +%Y%m%d)
Namespace Migration
# Backup source namespace
velero backup create migration-backup --include-namespaces production
# Restore to new namespace on same or different cluster
velero restore create migration-restore \
--from-backup migration-backup \
--namespace-mappings production:production-v2
Scheduling Backups
Velero supports scheduled backups using cron expressions for automated backup strategies.
Key Concepts
- Schedule: Defines recurring backup creation using cron syntax
- Retention: Controlled via TTL (time to live) on backups
- Schedule Status: Shows last backup time and next scheduled run
stateDiagram-v2
[*] --> Scheduled : Create Schedule
Scheduled --> Running : Cron Trigger
Running --> Completed : Backup Success
Running --> Failed : Backup Error
Completed --> Scheduled : Wait for Next
Failed --> Scheduled : Wait for Next
Creating Schedules
# Create hourly backup schedule
velero schedule create hourly-backup --schedule="0 * * * *"
# Create daily backup at 2 AM UTC
velero schedule create daily-backup --schedule="0 2 * * *"
# Create weekly backup (Sunday at midnight)
velero schedule create weekly-backup --schedule="0 0 * * 0"
# Create schedule with namespace filter
velero schedule create prod-backup \
--schedule="0 */6 * * *" \
--include-namespaces production
# Create schedule with TTL
velero schedule create daily-backup \
--schedule="0 2 * * *" \
--ttl 168h
# Create schedule excluding resources
velero schedule create app-backup \
--schedule="0 3 * * *" \
--include-namespaces app \
--exclude-resources events,pods
# Create schedule with label selector
velero schedule create critical-apps \
--schedule="0 * * * *" \
--selector tier=critical
# Schedule with volume snapshots disabled
velero schedule create metadata-only \
--schedule="0 4 * * *" \
--snapshot-volumes=false
Managing Schedules
# List all schedules
velero schedule get
# Describe schedule details
velero schedule describe daily-backup
# Pause schedule
velero schedule pause daily-backup
# Unpause schedule
velero schedule unpause daily-backup
# Delete schedule (keeps existing backups)
velero schedule delete daily-backup --confirm
# Trigger immediate backup from schedule
velero backup create --from-schedule daily-backup
Cron Expression Reference
| Expression | Description |
|---|---|
0 * * * * |
Every hour |
0 */6 * * * |
Every 6 hours |
0 0 * * * |
Daily at midnight |
0 2 * * * |
Daily at 2 AM |
0 0 * * 0 |
Weekly on Sunday |
0 0 1 * * |
Monthly on 1st |
*/15 * * * * |
Every 15 minutes |
Examples
Multi-tier Backup Strategy
# Tier 1: Hourly backups of critical namespaces (24h retention)
velero schedule create hourly-critical \
--schedule="0 * * * *" \
--include-namespaces production,payments \
--ttl 24h
# Tier 2: Daily backups of all apps (7 day retention)
velero schedule create daily-apps \
--schedule="0 2 * * *" \
--exclude-namespaces kube-system,velero \
--ttl 168h
# Tier 3: Weekly full backups (30 day retention)
velero schedule create weekly-full \
--schedule="0 3 * * 0" \
--ttl 720h
Backup Rotation
# Example: Keep last 7 daily, 4 weekly, and 3 monthly backups
# Daily backups with 7-day TTL
velero schedule create daily \
--schedule="0 1 * * *" \
--ttl 168h
# Weekly backups with 28-day TTL
velero schedule create weekly \
--schedule="0 2 * * 0" \
--ttl 672h
# Monthly backups with 90-day TTL
velero schedule create monthly \
--schedule="0 3 1 * *" \
--ttl 2160h
File System Backup (Node Agent)
Velero's file system backup (FSB) copies volume contents at the file level via a node-agent DaemonSet. This is useful when cloud provider snapshots are unavailable or cross-provider restores are needed. Kopia has been the default uploader since v1.10; the Restic uploader was deprecated in v1.15 and removed in v1.17 (new Restic backups disabled, existing ones restorable through v1.18). Current installs are Kopia-only.
Key Concepts
- Node Agent: DaemonSet that runs FSB on each node (replaces the old Restic DaemonSet)
- Kopia: Default and current uploader engine (Restic path removed in v1.17)
- File System Backup: Copies actual files rather than volume snapshots
- Pod Annotations: Control which volumes to backup
flowchart TD
A[Pod with PVC] --> B{Backup Type}
B -->|Native Snapshot| C[Cloud Provider API]
B -->|Restic/Kopia| D[Velero FSB]
C --> E[Volume Snapshot]
D --> F[File-level Backup]
E --> G[(Snapshot Storage)]
F --> H[(Object Storage)]
Installation with Node Agent (Kopia)
# Install Velero with node agent for file system backup (Kopia is the default uploader)
velero install \
--provider aws \
--bucket velero-backups \
--secret-file ./credentials \
--use-node-agent
# Install with both snapshots and file-level backup
velero install \
--provider aws \
--bucket velero-backups \
--secret-file ./credentials \
--use-node-agent \
--default-volumes-to-fs-backup
Annotating Pods for Volume Backup
# Include specific volumes for file-level backup
apiVersion: v1
kind: Pod
metadata:
name: app
annotations:
backup.velero.io/backup-volumes: data,config
spec:
containers:
- name: app
image: myapp:v1
volumeMounts:
- name: data
mountPath: /data
- name: config
mountPath: /etc/config
- name: cache
mountPath: /cache
volumes:
- name: data
persistentVolumeClaim:
claimName: data-pvc
- name: config
persistentVolumeClaim:
claimName: config-pvc
- name: cache
emptyDir: {}
# Exclude specific volumes from backup
apiVersion: v1
kind: Pod
metadata:
name: app
annotations:
backup.velero.io/backup-volumes-excludes: cache,tmp
spec:
# ... pod spec
File System Backup Commands
# Create backup with default file-level backup for all volumes
velero backup create fs-backup --default-volumes-to-fs-backup
# Check node agent (formerly restic) pods
kubectl get pods -n velero -l name=node-agent
# View file system backup progress
velero backup describe fs-backup --details
# List pod volume backups
kubectl get podvolumebackups -n velero
# Describe specific pod volume backup
kubectl describe podvolumebackup <name> -n velero
# List pod volume restores
kubectl get podvolumerestores -n velero
Repository Management
# List backup repositories (BackupRepository objects)
velero repo get
# Inspect the Kopia repository status (runs in a node-agent pod; exec ds/ picks one)
kubectl -n velero exec daemonset/node-agent -- kopia repository status
# Repo maintenance (incl. clearing stale locks) runs as a scheduled job, not a manual unlock
# Generate a debug bundle with logs
velero debug --output ./velero-debug.tar.gz
Examples
StatefulSet with Volume Backup
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: database
spec:
serviceName: database
replicas: 3
template:
metadata:
annotations:
# Backup data volume with Restic/Kopia
backup.velero.io/backup-volumes: data
spec:
containers:
- name: postgres
image: postgres:15
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
Backup with Mixed Volume Types
# Some volumes use snapshots, others use file-level backup
velero backup create mixed-backup \
--include-namespaces production \
--snapshot-volumes=true \
--default-volumes-to-fs-backup=false
# Volumes with annotations will use file-level backup
# Volumes without annotations will use cloud snapshots
CSI Volume Snapshots
CSI snapshots back up volumes via the storage driver's own snapshot API (the snapshot.storage.k8s.io CRDs), rather than cloud-provider volume APIs or file-level copy. The CSI plugin is built into core Velero since v1.14, but the integration is gated behind a feature flag — EnableCSI must be set at install time (still required as of v1.18; it is not on by default).
Key Concepts
EnableCSIfeature flag: Activates the CSI BackupItemAction; without it, PVCs backed by CSI drivers fall back to file-system backup or are skipped.- VolumeSnapshotClass label: Velero selects the
VolumeSnapshotClassmatching the PVC's CSI driver that carries thevelero.io/csi-volumesnapshot-class: "true"label. - CSI snapshot data movement: by default a CSI snapshot stays in the cluster's snapshot storage (tied to that cluster/region).
--snapshot-move-datacopies the snapshot data out to the object-storage backup location via the node-agent (Kopia), making backups portable for cross-cluster/cross-region restore.
Enabling CSI Snapshots
# Install with the CSI feature flag (object-storage plugin still required)
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.14.1 \
--bucket velero-backups \
--backup-location-config region=eu-west-1 \
--features=EnableCSI \
--secret-file ./credentials-velero
# On an existing install, add the flag to the Velero deployment
kubectl -n velero patch deployment velero --type=json \
-p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--features=EnableCSI"}]'
Labelling the VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-hostpath-snapclass
labels:
velero.io/csi-volumesnapshot-class: "true" # Velero picks this class for the matching driver
driver: hostpath.csi.k8s.io
deletionPolicy: Retain # Retain so the snapshot survives VolumeSnapshot deletion
Backing Up and Restoring with CSI
# CSI snapshots are used automatically once EnableCSI is set and a labelled class exists
velero backup create csi-backup --include-namespaces production
# Move snapshot data to object storage (portable, cluster-independent)
velero backup create csi-portable \
--include-namespaces production \
--snapshot-move-data
# Inspect the CSI snapshots a backup produced
velero backup describe csi-backup --details
# Restore as usual — Velero recreates PVCs from the CSI snapshots
velero restore create --from-backup csi-backup
CSI snapshots and file-system backup are mutually exclusive per volume: a PVC selected for FS backup (via annotation or
--default-volumes-to-fs-backup) is not also CSI-snapshotted. Choose snapshots for speed and storage-native efficiency, FS backup for drivers without snapshot support or when you need filesystem-level portability without the data mover.
Cloud Provider Configuration
Velero supports multiple cloud providers for storing backups and volume snapshots.
AWS Configuration
# Create credentials file
cat > credentials-velero <<EOF
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
EOF
# Install Velero for AWS
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.14.1 \
--bucket velero-backups \
--backup-location-config region=eu-west-1 \
--snapshot-location-config region=eu-west-1 \
--secret-file ./credentials-velero
AWS IAM Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeVolumes",
"ec2:DescribeSnapshots",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:CreateSnapshot",
"ec2:DeleteSnapshot"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::velero-backups/*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::velero-backups"
}
]
}
GCP Configuration
# Create service account
gcloud iam service-accounts create velero \
--display-name "Velero service account"
# Assign roles
PROJECT_ID=$(gcloud config get-value project)
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:velero@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/compute.storageAdmin
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:velero@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/storage.admin
# Create key file
gcloud iam service-accounts keys create credentials-velero \
--iam-account velero@$PROJECT_ID.iam.gserviceaccount.com
# Install Velero for GCP
velero install \
--provider gcp \
--plugins velero/velero-plugin-for-gcp:v1.14.1 \
--bucket velero-backups \
--secret-file ./credentials-velero
Azure Configuration
# Set variables
AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
AZURE_RESOURCE_GROUP=velero-backups
AZURE_STORAGE_ACCOUNT=velerobackups
# Create resource group
az group create -n $AZURE_RESOURCE_GROUP --location uksouth
# Create storage account
az storage account create \
--name $AZURE_STORAGE_ACCOUNT \
--resource-group $AZURE_RESOURCE_GROUP \
--sku Standard_GRS
# Create blob container
az storage container create \
--name velero \
--account-name $AZURE_STORAGE_ACCOUNT
# Create service principal
AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac \
--name velero \
--role Contributor \
--query password -o tsv)
AZURE_CLIENT_ID=$(az ad sp show --id http://velero --query appId -o tsv)
# Create credentials file
cat > credentials-velero <<EOF
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF
# Install Velero for Azure
velero install \
--provider azure \
--plugins velero/velero-plugin-for-microsoft-azure:v1.14.1 \
--bucket velero \
--secret-file ./credentials-velero \
--backup-location-config \
resourceGroup=$AZURE_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT \
--snapshot-location-config \
resourceGroup=$AZURE_RESOURCE_GROUP,subscriptionId=$AZURE_SUBSCRIPTION_ID
Multiple Backup Locations
# Create additional backup storage location
velero backup-location create secondary \
--provider aws \
--bucket velero-secondary \
--config region=us-west-2 \
--access-mode ReadWrite
# Create backup to specific location
velero backup create backup-west --storage-location secondary
# List backup locations
velero backup-location get
# Set backup location as default
velero backup-location set primary --default
Volume Snapshot Locations
# Create volume snapshot location
velero snapshot-location create aws-snapshots \
--provider aws \
--config region=eu-west-1
# Create backup with specific snapshot location
velero backup create backup-eu --volume-snapshot-locations aws-snapshots
# List snapshot locations
velero snapshot-location get
Common Use Cases
Disaster Recovery
flowchart TD
A[Production Cluster] -->|Scheduled Backups| B[(Object Storage)]
B -->|Disaster Occurs| C{Recovery Options}
C -->|Same Cluster| D[Restore to Production]
C -->|New Cluster| E[Restore to DR Cluster]
D --> F[Verify Services]
E --> F
F --> G[Resume Operations]
DR Backup Strategy
# Create DR backup schedule (every 4 hours)
velero schedule create disaster-recovery \
--schedule="0 */4 * * *" \
--ttl 168h \
--exclude-namespaces kube-system,velero \
--snapshot-volumes=true
# Pre-disaster validation
velero backup describe $(velero backup get -o json | jq -r '.items[-1].metadata.name')
# Restore to same cluster after disaster
velero restore create dr-restore \
--from-backup disaster-recovery-20240115120000 \
--wait
# Verify restoration
kubectl get deployments -A
kubectl get pvc -A
Cross-cluster DR
# On source cluster: create backup
velero backup create migration-backup \
--exclude-namespaces kube-system \
--wait
# On target cluster: configure same backup location
velero backup-location create source-backups \
--provider aws \
--bucket source-cluster-backups \
--config region=eu-west-1 \
--access-mode ReadOnly
# On target cluster: sync backups
velero backup get
# On target cluster: restore
velero restore create dr-restore \
--from-backup migration-backup
Cluster Migration
sequenceDiagram
participant Source as Source Cluster
participant Storage as Object Storage
participant Target as Target Cluster
Source->>Storage: velero backup create
Note over Source: Include all namespaces
Storage->>Target: Configure backup location
Target->>Storage: velero backup get (sync)
Target->>Target: velero restore create
Note over Target: Map namespaces if needed
Target->>Target: Verify workloads
Target->>Target: Update DNS/Ingress
Migration Steps
# Step 1: On source cluster - create comprehensive backup
velero backup create cluster-migration \
--exclude-namespaces kube-system,kube-public,velero \
--include-cluster-resources=true \
--default-volumes-to-fs-backup \
--wait
# Step 2: Verify backup completeness
velero backup describe cluster-migration --details
# Step 3: On target cluster - install Velero with same storage
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.14.1 \
--bucket source-cluster-backups \
--backup-location-config region=eu-west-1 \
--secret-file ./credentials-velero
# Step 4: Sync and verify backup is visible
velero backup get
# Step 5: Perform restore
velero restore create migration-restore \
--from-backup cluster-migration \
--include-cluster-resources=true \
--wait
# Step 6: Verify resources
kubectl get namespaces
kubectl get deployments -A
kubectl get services -A
kubectl get pvc -A
Namespace Migration to New Name
# Backup specific namespace
velero backup create ns-migration --include-namespaces old-app
# Restore to new namespace name
velero restore create ns-restore \
--from-backup ns-migration \
--namespace-mappings old-app:new-app
Pre-upgrade Backup
# Before Kubernetes upgrade or major changes
velero backup create pre-upgrade-$(date +%Y%m%d-%H%M) \
--ttl 720h \
--description "Pre-upgrade backup before K8s 1.28 upgrade" \
--wait
# After upgrade, if issues occur
velero restore create rollback \
--from-backup pre-upgrade-20240115-1430
Application-level Backup
# Backup specific application stack
velero backup create myapp-backup \
--include-namespaces myapp \
--include-resources \
deployments,services,configmaps,secrets,persistentvolumeclaims \
--selector app.kubernetes.io/name=myapp \
--snapshot-volumes=true
# Restore application
velero restore create myapp-restore \
--from-backup myapp-backup \
--include-namespaces myapp
Quick Reference
Backup Commands
velero backup create <name> # Create backup
velero backup create <name> --include-namespaces <ns> # Backup namespace
velero backup create <name> --from-schedule <sched> # From schedule
velero backup get # List backups
velero backup describe <name> --details # Backup details
velero backup logs <name> # View logs
velero backup delete <name> --confirm # Delete backup
Restore Commands
velero restore create --from-backup <backup> # Create restore
velero restore create --from-backup <backup> --namespace-mappings old:new
velero restore get # List restores
velero restore describe <name> # Restore details
velero restore logs <name> # View logs
Schedule Commands
velero schedule create <name> --schedule="0 2 * * *" # Create schedule
velero schedule get # List schedules
velero schedule describe <name> # Schedule details
velero schedule pause <name> # Pause schedule
velero schedule unpause <name> # Resume schedule
velero schedule delete <name> --confirm # Delete schedule
Storage Location Commands
velero backup-location get # List backup locations
velero backup-location create <name> --provider <p> --bucket <b>
velero snapshot-location get # List snapshot locations
Useful Flags
| Flag | Description |
|---|---|
--include-namespaces |
Backup specific namespaces |
--exclude-namespaces |
Exclude namespaces |
--include-resources |
Backup specific resource types |
--exclude-resources |
Exclude resource types |
--selector |
Backup resources matching labels |
--ttl |
Backup retention period |
--snapshot-volumes |
Enable volume snapshots |
--default-volumes-to-fs-backup |
Use file-level backup |
--snapshot-move-data |
Move CSI snapshot data to object storage (portable backups) |
--wait |
Wait for completion |
--namespace-mappings |
Remap namespaces during restore |
Common Issues and Solutions
Backup Stuck in Progress
Symptoms: Backup shows InProgress status for extended period
Solutions:
# Check Velero pod logs
kubectl logs -n velero deploy/velero
# Check for stuck volume snapshots
kubectl get volumesnapshots -A
# Cancel stuck backup
velero backup delete stuck-backup --confirm
# Check node agent (Restic) pods
kubectl get pods -n velero -l name=node-agent
kubectl logs -n velero -l name=node-agent
Volume Snapshot Failures
Symptoms: Backup completes but volumes show errors
Causes and solutions:
-
Missing CSI driver - Install appropriate CSI snapshot controller
kubectl get volumesnapshotclasses -
Wrong snapshot location - Verify VolumeSnapshotLocation configuration
velero snapshot-location get -
Insufficient permissions - Check cloud provider IAM roles
Restore Partial Failures
Symptoms: Some resources fail to restore
Debugging steps:
# Check restore details
velero restore describe my-restore --details
# View restore logs
velero restore logs my-restore
# Check for specific errors
velero restore logs my-restore | grep -i error
# Common causes:
# - Resources already exist (use --existing-resource-policy update)
# - Missing CRDs (restore cluster resources first)
# - Namespace doesn't exist
Restic/Kopia Backup Failures
Symptoms: Pod volume backups fail
Solutions:
# Check node-agent pods are running
kubectl get pods -n velero -l name=node-agent
# Check node-agent logs
kubectl logs -n velero daemonset/node-agent
# Verify pod annotations
kubectl get pod <name> -o jsonpath='{.metadata.annotations}'
# Check for repository lock issues
velero repo get
Backup Storage Location Unavailable
Symptoms: BackupStorageLocation shows Unavailable
Solutions:
# Check BSL status
velero backup-location get
# Verify credentials
kubectl get secret -n velero cloud-credentials -o yaml
# Check connectivity to storage
kubectl exec -n velero deploy/velero -- \
aws s3 ls s3://velero-backups/
# Recreate backup location
velero backup-location delete default --confirm
velero backup-location create default \
--provider aws \
--bucket velero-backups \
--config region=eu-west-1
Cross-cluster Restore Issues
Symptoms: Resources fail during restore to different cluster
Solutions:
-
StorageClass mismatch - Create matching StorageClasses or use ConfigMaps
velero restore create --from-backup migration \ --restore-volumes=false # Restore without volumes first -
Cluster-scoped resources - Exclude or include carefully
velero restore create --from-backup migration \ --include-cluster-resources=false -
Service accounts - May need to be recreated with new tokens
Scheduled Backup Not Running
Symptoms: Schedule exists but no backups created
Solutions:
# Check schedule status
velero schedule describe my-schedule
# Verify cron expression
# Use https://crontab.guru to validate
# Check if schedule is paused
velero schedule get
# Unpause if needed
velero schedule unpause my-schedule
# Check Velero controller logs
kubectl logs -n velero deploy/velero | grep schedule
Related Topics
The following topics complement Velero knowledge and are commonly used together:
- Kubernetes - Core platform for container orchestration; understanding cluster resources is essential for effective backups
- Helm - Package manager often used for Velero installation; simplifies deployment and configuration
- ArgoCD - GitOps tool for managing Velero configurations as code; automates backup policy deployment
- AWS/GCP/Azure - Cloud providers for backup storage and volume snapshots; IAM and storage configuration is critical
- Prometheus - Monitor Velero backup success/failure metrics; alerting on backup issues
- Cert-Manager - Often backed up alongside applications; TLS certificates need protection