K3s
A lightweight, certified Kubernetes distribution designed for resource-constrained environments, edge computing, and IoT deployments.
K3s Cheatsheet
A lightweight, certified Kubernetes distribution designed for resource-constrained environments, edge computing, and IoT deployments.
Overview
K3s is a fully compliant Kubernetes distribution packaged as a single binary under 100MB. It simplifies Kubernetes operations by bundling essential components and removing unnecessary features, making it ideal for edge computing, IoT devices, CI/CD pipelines, and development environments.
graph TB
subgraph "K3s Architecture"
subgraph "Server Node"
API[API Server]
SCHED[Scheduler]
CCM[Controller Manager]
SQLITE[(SQLite/etcd)]
TUNNEL[Tunnel Proxy]
end
subgraph "Built-in Components"
TRAEFIK[Traefik Ingress]
FLANNEL[Flannel CNI]
COREDNS[CoreDNS]
LB[ServiceLB]
HELM[Helm Controller]
end
subgraph "Agent Node"
KUBELET[Kubelet]
KPROXY[Kube-proxy]
CONTAINERD[containerd]
end
API --> SQLITE
API --> TUNNEL
TUNNEL --> KUBELET
KUBELET --> CONTAINERD
TRAEFIK --> API
FLANNEL --> KUBELET
end
style API fill:#326ce5
style TRAEFIK fill:#24a1c1
style FLANNEL fill:#4aa1d8
Lightweight Kubernetes Setup
Key Concepts
- Single Binary: K3s packages all Kubernetes components into a single binary (~75MB, under 100MB)
- Minimal Dependencies: Requires only the Linux kernel, cgroups, and iptables
- Embedded Components: Includes containerd, Flannel, CoreDNS, Traefik, and local storage provisioner
- Optimised for ARM: Native support for ARM64 and ARMv7 architectures
- Reduced Memory Footprint: Runs with as little as 512MB RAM for server nodes
Common Commands
# Check K3s version
k3s --version
# View K3s server configuration
k3s server --help
# View K3s agent configuration
k3s agent --help
# Check system requirements
k3s check-config
# Access kubectl through K3s
k3s kubectl get nodes
k3s kubectl get pods -A
# Create kubectl alias
alias kubectl='k3s kubectl'
# Export kubeconfig for external tools
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
Examples
# Verify K3s is running
sudo systemctl status k3s
# Check K3s logs
sudo journalctl -u k3s -f
# View K3s configuration file
cat /etc/rancher/k3s/k3s.yaml
# Check cluster info
k3s kubectl cluster-info
# View node resources
k3s kubectl top nodes
Simplified Installation and Operation
Key Concepts
- Automatic TLS: K3s automatically generates TLS certificates for cluster communication
- Token-based Joining: Nodes join the cluster using a shared token
- Automatic Updates: K3s can be configured for automatic updates via system-upgrade-controller
- Uninstall Scripts: Clean uninstallation scripts are automatically created
Common Commands
# Install K3s server (single node)
curl -sfL https://get.k3s.io | sh -
# Install K3s with specific version
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.28.4+k3s1 sh -
# Install without Traefik
curl -sfL https://get.k3s.io | sh -s - --disable traefik
# Install without ServiceLB
curl -sfL https://get.k3s.io | sh -s - --disable servicelb
# Get node token for joining agents
sudo cat /var/lib/rancher/k3s/server/node-token
# Install K3s agent on worker node
curl -sfL https://get.k3s.io | K3S_URL=https://server-ip:6443 K3S_TOKEN=<token> sh -
# Uninstall K3s server
/usr/local/bin/k3s-uninstall.sh
# Uninstall K3s agent
/usr/local/bin/k3s-agent-uninstall.sh
Examples
# Install with custom cluster CIDR
curl -sfL https://get.k3s.io | sh -s - \
--cluster-cidr=10.42.0.0/16 \
--service-cidr=10.43.0.0/16
# Install with external database
curl -sfL https://get.k3s.io | sh -s - \
--datastore-endpoint="mysql://user:pass@tcp(hostname:3306)/k3s"
# Install with custom kubeconfig permissions
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
# Configure K3s via config file
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml << EOF
write-kubeconfig-mode: "0644"
tls-san:
- "k3s.example.com"
- "192.168.1.100"
disable:
- traefik
EOF
# Restart K3s to apply configuration
sudo systemctl restart k3s
Built-in Components
Key Concepts
graph LR
subgraph "K3s Built-in Stack"
direction TB
subgraph "Networking"
FLANNEL[Flannel CNI<br/>VXLAN overlay]
COREDNS[CoreDNS<br/>DNS resolution]
TRAEFIK[Traefik v3<br/>Ingress controller]
SLB[ServiceLB<br/>Load balancer]
end
subgraph "Storage"
LOCAL[Local-path<br/>Provisioner]
end
subgraph "Runtime"
CTD[containerd<br/>Container runtime]
end
subgraph "Management"
HELM[Helm Controller<br/>Chart deployment]
MC[Metrics Server<br/>Resource metrics]
end
end
style FLANNEL fill:#4aa1d8
style TRAEFIK fill:#24a1c1
style CTD fill:#575757
- Traefik: Default ingress controller (v3) with automatic HTTPS and Let's Encrypt support
- Flannel: Simple overlay network using VXLAN by default
- CoreDNS: Cluster DNS for service discovery
- Local-path Provisioner: Dynamic PersistentVolume provisioning on local storage
- ServiceLB: Simple load balancer for bare metal deployments
- Metrics Server: Resource usage metrics for HPA and kubectl top
Common Commands
# Check built-in component status
k3s kubectl get pods -n kube-system
# View Traefik configuration
k3s kubectl get ingressroute -A
# Check Flannel network
k3s kubectl get pods -n kube-system -l app=flannel
# View CoreDNS configuration
k3s kubectl get configmap -n kube-system coredns -o yaml
# Check local-path provisioner
k3s kubectl get storageclass
# Disable specific components during installation
curl -sfL https://get.k3s.io | sh -s - \
--disable traefik \
--disable servicelb \
--disable local-storage
# Replace Flannel with Calico
curl -sfL https://get.k3s.io | sh -s - --flannel-backend=none
Examples
# Deploy application with Traefik ingress
cat << EOF | k3s kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
rules:
- host: myapp.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: myapp
port:
number: 80
EOF
# Create PVC with local-path provisioner
cat << EOF | k3s kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: local-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
storage: 1Gi
EOF
# Configure Traefik dashboard (Traefik v3 — uses traefik.io/v1alpha1, not traefik.containo.us/v1alpha1)
cat << EOF | k3s kubectl apply -f -
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard
namespace: kube-system
spec:
entryPoints:
- web
routes:
- match: Host(\`traefik.local\`)
kind: Rule
services:
- name: api@internal
kind: TraefikService
EOF
Resource Efficiency
Key Concepts
- Memory Usage: Server ~512MB, Agent ~256MB minimum
- CPU Usage: Minimal CPU overhead compared to full Kubernetes
- Disk Space: Binary ~75MB (under 100MB), total installation ~200MB
- SQLite Backend: Eliminates etcd for single-node deployments
- Optimised Binaries: Stripped and compressed Kubernetes components
Common Commands
# Monitor K3s resource usage
k3s kubectl top nodes
k3s kubectl top pods -A
# Check K3s process memory
ps aux | grep k3s
# View detailed node resources
k3s kubectl describe node <node-name>
# Set resource limits for system pods
k3s kubectl set resources deployment coredns -n kube-system \
--limits=cpu=100m,memory=128Mi \
--requests=cpu=50m,memory=64Mi
# Configure kubelet resource reservation
curl -sfL https://get.k3s.io | sh -s - \
--kubelet-arg=system-reserved=cpu=100m,memory=100Mi \
--kubelet-arg=kube-reserved=cpu=100m,memory=100Mi
Examples
# Optimise K3s for minimal resources
sudo tee /etc/rancher/k3s/config.yaml << EOF
disable:
- traefik
- servicelb
- metrics-server
kubelet-arg:
- "max-pods=50"
- "system-reserved=cpu=100m,memory=100Mi"
kube-controller-manager-arg:
- "node-monitor-period=60s"
- "node-monitor-grace-period=180s"
EOF
# Deploy lightweight monitoring
cat << EOF | k3s kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: resource-monitor
spec:
containers:
- name: monitor
image: alpine
command: ['sh', '-c', 'while true; do free -m; sleep 60; done']
resources:
limits:
memory: "32Mi"
cpu: "50m"
EOF
# Check etcd/SQLite database size
sudo ls -lh /var/lib/rancher/k3s/server/db/
# Reset etcd to single-member (note: --cluster-reset affects etcd, not SQLite;
# for SQLite there is no compact operation — delete state.db to wipe and restart)
sudo k3s server --cluster-reset
Multi-node Cluster Setup
Key Concepts
graph TB
subgraph "Multi-node K3s Cluster"
subgraph "Server Node"
SERVER[K3s Server<br/>Control Plane]
TOKEN[Node Token<br/>/var/lib/rancher/k3s/server/node-token]
end
subgraph "Agent Nodes"
AGENT1[K3s Agent 1<br/>Worker]
AGENT2[K3s Agent 2<br/>Worker]
AGENT3[K3s Agent 3<br/>Worker]
end
SERVER --> TOKEN
TOKEN -.->|Join Token| AGENT1
TOKEN -.->|Join Token| AGENT2
TOKEN -.->|Join Token| AGENT3
SERVER <-->|6443| AGENT1
SERVER <-->|6443| AGENT2
SERVER <-->|6443| AGENT3
end
style SERVER fill:#326ce5
style AGENT1 fill:#4aa1d8
style AGENT2 fill:#4aa1d8
style AGENT3 fill:#4aa1d8
- Server Nodes: Run the control plane components
- Agent Nodes: Run workloads only (no control plane)
- Node Token: Shared secret for cluster authentication
- TLS SAN: Additional hostnames/IPs for API server certificate
Common Commands
# On server node - install and get token
curl -sfL https://get.k3s.io | sh -
sudo cat /var/lib/rancher/k3s/server/node-token
# On agent nodes - join cluster
curl -sfL https://get.k3s.io | K3S_URL=https://server-ip:6443 \
K3S_TOKEN=<node-token> sh -
# Label worker nodes
k3s kubectl label node worker1 node-role.kubernetes.io/worker=worker
# Taint server node to prevent workloads
k3s kubectl taint nodes server node-role.kubernetes.io/master:NoSchedule
# Cordon node for maintenance
k3s kubectl cordon <node-name>
# Drain node before removal
k3s kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
# Remove node from cluster
k3s kubectl delete node <node-name>
Examples
# Server installation with TLS SAN
curl -sfL https://get.k3s.io | sh -s - \
--tls-san k3s.example.com \
--tls-san 192.168.1.100 \
--node-name server1
# Agent installation with labels
curl -sfL https://get.k3s.io | K3S_URL=https://server-ip:6443 \
K3S_TOKEN=<token> sh -s - \
--node-name worker1 \
--node-label role=compute \
--node-label zone=east
# Deploy workload with node affinity
cat << EOF | k3s kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: compute-app
spec:
replicas: 3
selector:
matchLabels:
app: compute
template:
metadata:
labels:
app: compute
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: role
operator: In
values:
- compute
containers:
- name: app
image: nginx
EOF
# Set up cluster with configuration file
# On server
sudo tee /etc/rancher/k3s/config.yaml << EOF
token: my-shared-secret
tls-san:
- k3s.example.com
- 192.168.1.100
node-label:
- "nodeType=server"
EOF
curl -sfL https://get.k3s.io | sh -
# On agents
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml << EOF
server: https://192.168.1.100:6443
token: my-shared-secret
node-label:
- "nodeType=agent"
EOF
curl -sfL https://get.k3s.io | sh -s - agent
High Availability Configuration
Key Concepts
graph TB
subgraph "K3s HA Architecture"
LB[Load Balancer<br/>TCP 6443]
subgraph "Server Nodes"
S1[Server 1<br/>Control Plane]
S2[Server 2<br/>Control Plane]
S3[Server 3<br/>Control Plane]
end
subgraph "External Datastore"
direction LR
DB[(PostgreSQL/MySQL<br/>or etcd cluster)]
end
subgraph "Agent Nodes"
A1[Agent 1]
A2[Agent 2]
end
LB --> S1
LB --> S2
LB --> S3
S1 --> DB
S2 --> DB
S3 --> DB
A1 --> LB
A2 --> LB
end
style LB fill:#f0ad4e
style S1 fill:#326ce5
style S2 fill:#326ce5
style S3 fill:#326ce5
style DB fill:#336791
- Embedded etcd: Built-in HA using embedded etcd (3+ server nodes)
- External Datastore: PostgreSQL, MySQL, or external etcd cluster
- Load Balancer: Required for HA to distribute API requests
- Odd Number Servers: Use 3, 5, or 7 servers for quorum
Common Commands
# Initialise first server with embedded etcd
curl -sfL https://get.k3s.io | sh -s - server \
--cluster-init \
--tls-san <load-balancer-ip>
# Join additional server nodes
curl -sfL https://get.k3s.io | sh -s - server \
--server https://<first-server-ip>:6443 \
--token <node-token> \
--tls-san <load-balancer-ip>
# Check etcd cluster health
k3s kubectl get endpoints -n kube-system
k3s etcd-snapshot ls
# Create etcd snapshot
k3s etcd-snapshot save --name my-snapshot
# Restore from etcd snapshot
k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot
Examples
# HA with external PostgreSQL
# First server
curl -sfL https://get.k3s.io | sh -s - server \
--datastore-endpoint="postgres://user:pass@postgres-host:5432/k3s" \
--tls-san k3s-lb.example.com
# Additional servers
curl -sfL https://get.k3s.io | sh -s - server \
--datastore-endpoint="postgres://user:pass@postgres-host:5432/k3s" \
--tls-san k3s-lb.example.com \
--token <node-token>
# HA with embedded etcd
# First server
sudo tee /etc/rancher/k3s/config.yaml << EOF
cluster-init: true
tls-san:
- k3s-lb.example.com
- 192.168.1.10
token: my-cluster-token
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 5
EOF
curl -sfL https://get.k3s.io | sh -
# Second and third servers
sudo tee /etc/rancher/k3s/config.yaml << EOF
server: https://192.168.1.100:6443
token: my-cluster-token
tls-san:
- k3s-lb.example.com
- 192.168.1.10
EOF
curl -sfL https://get.k3s.io | sh -
# HAProxy configuration for load balancing
cat << EOF > /etc/haproxy/haproxy.cfg
frontend k3s-api
bind *:6443
mode tcp
default_backend k3s-servers
backend k3s-servers
mode tcp
balance roundrobin
option tcp-check
server server1 192.168.1.101:6443 check
server server2 192.168.1.102:6443 check
server server3 192.168.1.103:6443 check
EOF
# Configure agents to use load balancer
curl -sfL https://get.k3s.io | K3S_URL=https://k3s-lb.example.com:6443 \
K3S_TOKEN=<token> sh -
Common Use Cases
Key Concepts
- Edge Computing: Deploy Kubernetes at network edge with minimal resources
- IoT Gateways: Run containerised workloads on IoT devices
- Development: Lightweight local Kubernetes for development and testing
- CI/CD: Ephemeral clusters for testing in pipelines
- ARM Devices: Native support for Raspberry Pi and ARM servers
Examples
# Edge deployment with offline installation
# Download binary on connected machine
wget https://github.com/k3s-io/k3s/releases/download/v1.28.4+k3s1/k3s
wget https://github.com/k3s-io/k3s/releases/download/v1.28.4+k3s1/k3s-airgap-images-amd64.tar
# Transfer to air-gapped machine and install
sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/
sudo cp k3s /usr/local/bin/
sudo chmod +x /usr/local/bin/k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_DOWNLOAD=true sh -
# Raspberry Pi cluster setup
# On master Pi
curl -sfL https://get.k3s.io | sh -s - \
--write-kubeconfig-mode 644 \
--disable servicelb \
--disable traefik
# On worker Pis
curl -sfL https://get.k3s.io | K3S_URL=https://master-pi:6443 \
K3S_TOKEN=<token> sh -
# Development environment with k3d (K3s in Docker)
# Install k3d
curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
# Create development cluster
k3d cluster create dev --servers 1 --agents 2 \
--port "8080:80@loadbalancer" \
--port "8443:443@loadbalancer"
# Delete cluster when done
k3d cluster delete dev
# CI/CD pipeline example (GitLab CI)
# .gitlab-ci.yml
stages:
- test
integration-test:
stage: test
image: rancher/k3s:v1.28.4-k3s1
services:
- docker:dind
script:
- k3s server &
- sleep 30
- k3s kubectl apply -f manifests/
- k3s kubectl wait --for=condition=ready pod -l app=myapp --timeout=60s
- k3s kubectl run test --image=curlimages/curl --rm -it --restart=Never -- curl myapp-svc
# IoT gateway deployment
cat << EOF | k3s kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: iot-collector
spec:
selector:
matchLabels:
app: iot-collector
template:
metadata:
labels:
app: iot-collector
spec:
hostNetwork: true
containers:
- name: collector
image: myregistry/iot-collector:latest
resources:
limits:
memory: "64Mi"
cpu: "100m"
volumeMounts:
- name: device
mountPath: /dev/ttyUSB0
volumes:
- name: device
hostPath:
path: /dev/ttyUSB0
EOF
Quick Reference
| Task | Command |
|---|---|
| Install K3s (single node) | curl -sfL https://get.k3s.io | sh - |
| Get node token | sudo cat /var/lib/rancher/k3s/server/node-token |
| Join agent node | curl -sfL https://get.k3s.io | K3S_URL=https://server:6443 K3S_TOKEN=token sh - |
| Use kubectl | k3s kubectl get nodes |
| Check K3s status | sudo systemctl status k3s |
| View K3s logs | sudo journalctl -u k3s -f |
| Uninstall server | /usr/local/bin/k3s-uninstall.sh |
| Uninstall agent | /usr/local/bin/k3s-agent-uninstall.sh |
| Create etcd snapshot | k3s etcd-snapshot save --name backup |
| List etcd snapshots | k3s etcd-snapshot ls |
| Disable component | --disable traefik |
| Add TLS SAN | --tls-san hostname |
| Set cluster CIDR | --cluster-cidr 10.42.0.0/16 |
| External database | --datastore-endpoint "postgres://..." |
| Initialise HA cluster | --cluster-init |
| Config file location | /etc/rancher/k3s/config.yaml |
| Kubeconfig location | /etc/rancher/k3s/k3s.yaml |
| Data directory | /var/lib/rancher/k3s/ |
Common Issues and Solutions
Installation Failures
Problem: Installation script fails with network errors
# Solution: Use offline installation
# Download on connected machine
wget https://github.com/k3s-io/k3s/releases/download/v1.28.4+k3s1/k3s
wget https://get.k3s.io -O install.sh
# Transfer and install
chmod +x k3s && sudo mv k3s /usr/local/bin/
chmod +x install.sh && INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh
Node Not Joining Cluster
Problem: Agent cannot connect to server
# Check firewall rules (required ports: 6443, 8472 UDP for Flannel)
sudo ufw allow 6443/tcp
sudo ufw allow 8472/udp
sudo ufw allow 10250/tcp
# Verify server is accessible
curl -k https://server-ip:6443
# Check token is correct
sudo cat /var/lib/rancher/k3s/server/node-token
# Verify DNS resolution
nslookup server-hostname
Traefik Ingress Not Working
Problem: Ingress routes not responding
# Check Traefik pod status
k3s kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik
# View Traefik logs
k3s kubectl logs -n kube-system -l app.kubernetes.io/name=traefik
# Verify ingress configuration
k3s kubectl describe ingress <ingress-name>
# Check service endpoints
k3s kubectl get endpoints <service-name>
CoreDNS Resolution Failures
Problem: Pods cannot resolve DNS names
# Check CoreDNS pods
k3s kubectl get pods -n kube-system -l k8s-app=kube-dns
# Test DNS resolution
k3s kubectl run test --image=busybox --rm -it --restart=Never -- nslookup kubernetes
# Check CoreDNS configuration
k3s kubectl get configmap -n kube-system coredns -o yaml
# Restart CoreDNS
k3s kubectl rollout restart deployment coredns -n kube-system
High Memory Usage
Problem: K3s consuming too much memory
# Disable unused components
curl -sfL https://get.k3s.io | sh -s - \
--disable traefik \
--disable metrics-server \
--disable servicelb
# Set kubelet memory limits
sudo tee -a /etc/rancher/k3s/config.yaml << EOF
kubelet-arg:
- "eviction-hard=memory.available<100Mi"
- "system-reserved=memory=100Mi"
EOF
sudo systemctl restart k3s
# Check what's consuming memory
k3s kubectl top pods -A --sort-by=memory
Etcd/Database Issues
Problem: Cluster loses quorum or data corruption
# List available snapshots
k3s etcd-snapshot ls
# Restore from snapshot (stops cluster)
sudo systemctl stop k3s
k3s server --cluster-reset \
--cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<snapshot>
# For SQLite corruption
sudo systemctl stop k3s
sudo rm /var/lib/rancher/k3s/server/db/state.db
sudo systemctl start k3s
Certificate Errors
Problem: TLS certificate errors when accessing API
# Add additional SANs
sudo tee -a /etc/rancher/k3s/config.yaml << EOF
tls-san:
- "new-hostname.example.com"
- "10.0.0.100"
EOF
# Regenerate certificates
sudo systemctl stop k3s
sudo rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json
sudo systemctl start k3s
# Update kubeconfig with correct server address
sed -i 's/127.0.0.1/actual-server-ip/' /etc/rancher/k3s/k3s.yaml
Storage Issues
Problem: PersistentVolumeClaims stuck in Pending
# Check storage class
k3s kubectl get storageclass
# Verify local-path provisioner is running
k3s kubectl get pods -n kube-system -l app=local-path-provisioner
# Check provisioner logs
k3s kubectl logs -n kube-system -l app=local-path-provisioner
# Manual PV creation if needed
cat << EOF | k3s kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
name: manual-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /data/manual-pv
storageClassName: local-path
EOF
Related Topics
- Kubernetes Fundamentals - Core Kubernetes concepts, architecture, and API resources
- Helm - Package manager for deploying applications on K3s clusters
- Traefik - Advanced ingress configuration and routing options
- containerd - Container runtime configuration and troubleshooting
- Rancher - Multi-cluster management platform that integrates with K3s
- GitOps with Flux/ArgoCD - Continuous deployment patterns for K3s clusters