Available for day contractsFrom 21st September I have availability for day and half day contracts. Please contact for more information.

Contact →
mikepreston.org

K3s

A lightweight, certified Kubernetes distribution designed for resource-constrained environments, edge computing, and IoT deployments.

K3s Cheatsheet

A lightweight, certified Kubernetes distribution designed for resource-constrained environments, edge computing, and IoT deployments.

Overview

K3s is a fully compliant Kubernetes distribution packaged as a single binary under 100MB. It simplifies Kubernetes operations by bundling essential components and removing unnecessary features, making it ideal for edge computing, IoT devices, CI/CD pipelines, and development environments.

K3s ArchitectureAgent NodeBuilt-in ComponentsServer NodeKubeletAPI ServerSQLite/etcdTunnel ProxycontainerdTraefik IngressFlannel CNIKube-proxyCoreDNSServiceLBHelm ControllerSchedulerController ManagerK3s ArchitectureAgent NodeBuilt-in ComponentsServer NodeKubeletAPI ServerSQLite/etcdTunnel ProxycontainerdTraefik IngressFlannel CNIKube-proxyCoreDNSServiceLBHelm ControllerSchedulerController Manager

Lightweight Kubernetes Setup

Key Concepts

  • Single Binary: K3s packages all Kubernetes components into a single binary (~75MB, under 100MB)
  • Minimal Dependencies: Requires only the Linux kernel, cgroups, and iptables
  • Embedded Components: Includes containerd, Flannel, CoreDNS, Traefik, and local storage provisioner
  • Optimised for ARM: Native support for ARM64 and ARMv7 architectures
  • Reduced Memory Footprint: Runs with as little as 512MB RAM for server nodes

Common Commands

# Check K3s version
k3s --version

# View K3s server configuration
k3s server --help

# View K3s agent configuration
k3s agent --help

# Check system requirements
k3s check-config

# Access kubectl through K3s
k3s kubectl get nodes
k3s kubectl get pods -A

# Create kubectl alias
alias kubectl='k3s kubectl'

# Export kubeconfig for external tools
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Examples

# Verify K3s is running
sudo systemctl status k3s

# Check K3s logs
sudo journalctl -u k3s -f

# View K3s configuration file
cat /etc/rancher/k3s/k3s.yaml

# Check cluster info
k3s kubectl cluster-info

# View node resources
k3s kubectl top nodes

Simplified Installation and Operation

Key Concepts

  • Automatic TLS: K3s automatically generates TLS certificates for cluster communication
  • Token-based Joining: Nodes join the cluster using a shared token
  • Automatic Updates: K3s can be configured for automatic updates via system-upgrade-controller
  • Uninstall Scripts: Clean uninstallation scripts are automatically created

Common Commands

# Install K3s server (single node)
curl -sfL https://get.k3s.io | sh -

# Install K3s with specific version
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.28.4+k3s1 sh -

# Install without Traefik
curl -sfL https://get.k3s.io | sh -s - --disable traefik

# Install without ServiceLB
curl -sfL https://get.k3s.io | sh -s - --disable servicelb

# Get node token for joining agents
sudo cat /var/lib/rancher/k3s/server/node-token

# Install K3s agent on worker node
curl -sfL https://get.k3s.io | K3S_URL=https://server-ip:6443 K3S_TOKEN=<token> sh -

# Uninstall K3s server
/usr/local/bin/k3s-uninstall.sh

# Uninstall K3s agent
/usr/local/bin/k3s-agent-uninstall.sh

Examples

# Install with custom cluster CIDR
curl -sfL https://get.k3s.io | sh -s - \
  --cluster-cidr=10.42.0.0/16 \
  --service-cidr=10.43.0.0/16

# Install with external database
curl -sfL https://get.k3s.io | sh -s - \
  --datastore-endpoint="mysql://user:pass@tcp(hostname:3306)/k3s"

# Install with custom kubeconfig permissions
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

# Configure K3s via config file
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml << EOF
write-kubeconfig-mode: "0644"
tls-san:
  - "k3s.example.com"
  - "192.168.1.100"
disable:
  - traefik
EOF

# Restart K3s to apply configuration
sudo systemctl restart k3s

Built-in Components

Key Concepts

K3s Built-in StackManagementHelm ControllerChart deploymentMetrics ServerResource metricsRuntimecontainerdContainer runtimeStorageLocal-pathProvisionerNetworkingFlannel CNIVXLAN overlayCoreDNSDNS resolutionTraefik v3Ingress controllerServiceLBLoad balancerK3s Built-in StackManagementHelm ControllerChart deploymentMetrics ServerResource metricsRuntimecontainerdContainer runtimeStorageLocal-pathProvisionerNetworkingFlannel CNIVXLAN overlayCoreDNSDNS resolutionTraefik v3Ingress controllerServiceLBLoad balancer
  • Traefik: Default ingress controller (v3) with automatic HTTPS and Let's Encrypt support
  • Flannel: Simple overlay network using VXLAN by default
  • CoreDNS: Cluster DNS for service discovery
  • Local-path Provisioner: Dynamic PersistentVolume provisioning on local storage
  • ServiceLB: Simple load balancer for bare metal deployments
  • Metrics Server: Resource usage metrics for HPA and kubectl top

Common Commands

# Check built-in component status
k3s kubectl get pods -n kube-system

# View Traefik configuration
k3s kubectl get ingressroute -A

# Check Flannel network
k3s kubectl get pods -n kube-system -l app=flannel

# View CoreDNS configuration
k3s kubectl get configmap -n kube-system coredns -o yaml

# Check local-path provisioner
k3s kubectl get storageclass

# Disable specific components during installation
curl -sfL https://get.k3s.io | sh -s - \
  --disable traefik \
  --disable servicelb \
  --disable local-storage

# Replace Flannel with Calico
curl -sfL https://get.k3s.io | sh -s - --flannel-backend=none

Examples

# Deploy application with Traefik ingress
cat << EOF | k3s kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
  - host: myapp.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80
EOF

# Create PVC with local-path provisioner
cat << EOF | k3s kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: local-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 1Gi
EOF

# Configure Traefik dashboard (Traefik v3 — uses traefik.io/v1alpha1, not traefik.containo.us/v1alpha1)
cat << EOF | k3s kubectl apply -f -
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: kube-system
spec:
  entryPoints:
    - web
  routes:
    - match: Host(\`traefik.local\`)
      kind: Rule
      services:
        - name: api@internal
          kind: TraefikService
EOF

Resource Efficiency

Key Concepts

  • Memory Usage: Server ~512MB, Agent ~256MB minimum
  • CPU Usage: Minimal CPU overhead compared to full Kubernetes
  • Disk Space: Binary ~75MB (under 100MB), total installation ~200MB
  • SQLite Backend: Eliminates etcd for single-node deployments
  • Optimised Binaries: Stripped and compressed Kubernetes components

Common Commands

# Monitor K3s resource usage
k3s kubectl top nodes
k3s kubectl top pods -A

# Check K3s process memory
ps aux | grep k3s

# View detailed node resources
k3s kubectl describe node <node-name>

# Set resource limits for system pods
k3s kubectl set resources deployment coredns -n kube-system \
  --limits=cpu=100m,memory=128Mi \
  --requests=cpu=50m,memory=64Mi

# Configure kubelet resource reservation
curl -sfL https://get.k3s.io | sh -s - \
  --kubelet-arg=system-reserved=cpu=100m,memory=100Mi \
  --kubelet-arg=kube-reserved=cpu=100m,memory=100Mi

Examples

# Optimise K3s for minimal resources
sudo tee /etc/rancher/k3s/config.yaml << EOF
disable:
  - traefik
  - servicelb
  - metrics-server
kubelet-arg:
  - "max-pods=50"
  - "system-reserved=cpu=100m,memory=100Mi"
kube-controller-manager-arg:
  - "node-monitor-period=60s"
  - "node-monitor-grace-period=180s"
EOF

# Deploy lightweight monitoring
cat << EOF | k3s kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: resource-monitor
spec:
  containers:
  - name: monitor
    image: alpine
    command: ['sh', '-c', 'while true; do free -m; sleep 60; done']
    resources:
      limits:
        memory: "32Mi"
        cpu: "50m"
EOF

# Check etcd/SQLite database size
sudo ls -lh /var/lib/rancher/k3s/server/db/

# Reset etcd to single-member (note: --cluster-reset affects etcd, not SQLite;
# for SQLite there is no compact operation — delete state.db to wipe and restart)
sudo k3s server --cluster-reset

Multi-node Cluster Setup

Key Concepts

Multi-node K3s ClusterAgent NodesServer NodeJoin TokenJoin TokenJoin Token644364436443K3s Agent 1WorkerK3s ServerControl PlaneNode Token/var/lib/rancher/k3s/server/node-tokenK3s Agent 2WorkerK3s Agent 3WorkerMulti-node K3s ClusterAgent NodesServer NodeJoin TokenJoin TokenJoin Token644364436443K3s Agent 1WorkerK3s ServerControl PlaneNode Token/var/lib/rancher/k3s/server/node-tokenK3s Agent 2WorkerK3s Agent 3Worker
  • Server Nodes: Run the control plane components
  • Agent Nodes: Run workloads only (no control plane)
  • Node Token: Shared secret for cluster authentication
  • TLS SAN: Additional hostnames/IPs for API server certificate

Common Commands

# On server node - install and get token
curl -sfL https://get.k3s.io | sh -
sudo cat /var/lib/rancher/k3s/server/node-token

# On agent nodes - join cluster
curl -sfL https://get.k3s.io | K3S_URL=https://server-ip:6443 \
  K3S_TOKEN=<node-token> sh -

# Label worker nodes
k3s kubectl label node worker1 node-role.kubernetes.io/worker=worker

# Taint server node to prevent workloads
k3s kubectl taint nodes server node-role.kubernetes.io/master:NoSchedule

# Cordon node for maintenance
k3s kubectl cordon <node-name>

# Drain node before removal
k3s kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Remove node from cluster
k3s kubectl delete node <node-name>

Examples

# Server installation with TLS SAN
curl -sfL https://get.k3s.io | sh -s - \
  --tls-san k3s.example.com \
  --tls-san 192.168.1.100 \
  --node-name server1

# Agent installation with labels
curl -sfL https://get.k3s.io | K3S_URL=https://server-ip:6443 \
  K3S_TOKEN=<token> sh -s - \
  --node-name worker1 \
  --node-label role=compute \
  --node-label zone=east

# Deploy workload with node affinity
cat << EOF | k3s kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: compute-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: compute
  template:
    metadata:
      labels:
        app: compute
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: role
                operator: In
                values:
                - compute
      containers:
      - name: app
        image: nginx
EOF

# Set up cluster with configuration file
# On server
sudo tee /etc/rancher/k3s/config.yaml << EOF
token: my-shared-secret
tls-san:
  - k3s.example.com
  - 192.168.1.100
node-label:
  - "nodeType=server"
EOF
curl -sfL https://get.k3s.io | sh -

# On agents
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml << EOF
server: https://192.168.1.100:6443
token: my-shared-secret
node-label:
  - "nodeType=agent"
EOF
curl -sfL https://get.k3s.io | sh -s - agent

High Availability Configuration

Key Concepts

K3s HA ArchitectureAgent NodesExternal DatastoreServer NodesAgent 1Load BalancerTCP 6443Server 1Control PlaneServer 2Control PlaneServer 3Control PlanePostgreSQL/MySQLor etcd clusterAgent 2K3s HA ArchitectureAgent NodesExternal DatastoreServer NodesAgent 1Load BalancerTCP 6443Server 1Control PlaneServer 2Control PlaneServer 3Control PlanePostgreSQL/MySQLor etcd clusterAgent 2
  • Embedded etcd: Built-in HA using embedded etcd (3+ server nodes)
  • External Datastore: PostgreSQL, MySQL, or external etcd cluster
  • Load Balancer: Required for HA to distribute API requests
  • Odd Number Servers: Use 3, 5, or 7 servers for quorum

Common Commands

# Initialise first server with embedded etcd
curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --tls-san <load-balancer-ip>

# Join additional server nodes
curl -sfL https://get.k3s.io | sh -s - server \
  --server https://<first-server-ip>:6443 \
  --token <node-token> \
  --tls-san <load-balancer-ip>

# Check etcd cluster health
k3s kubectl get endpoints -n kube-system
k3s etcd-snapshot ls

# Create etcd snapshot
k3s etcd-snapshot save --name my-snapshot

# Restore from etcd snapshot
k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot

Examples

# HA with external PostgreSQL
# First server
curl -sfL https://get.k3s.io | sh -s - server \
  --datastore-endpoint="postgres://user:pass@postgres-host:5432/k3s" \
  --tls-san k3s-lb.example.com

# Additional servers
curl -sfL https://get.k3s.io | sh -s - server \
  --datastore-endpoint="postgres://user:pass@postgres-host:5432/k3s" \
  --tls-san k3s-lb.example.com \
  --token <node-token>

# HA with embedded etcd
# First server
sudo tee /etc/rancher/k3s/config.yaml << EOF
cluster-init: true
tls-san:
  - k3s-lb.example.com
  - 192.168.1.10
token: my-cluster-token
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 5
EOF
curl -sfL https://get.k3s.io | sh -

# Second and third servers
sudo tee /etc/rancher/k3s/config.yaml << EOF
server: https://192.168.1.100:6443
token: my-cluster-token
tls-san:
  - k3s-lb.example.com
  - 192.168.1.10
EOF
curl -sfL https://get.k3s.io | sh -

# HAProxy configuration for load balancing
cat << EOF > /etc/haproxy/haproxy.cfg
frontend k3s-api
    bind *:6443
    mode tcp
    default_backend k3s-servers

backend k3s-servers
    mode tcp
    balance roundrobin
    option tcp-check
    server server1 192.168.1.101:6443 check
    server server2 192.168.1.102:6443 check
    server server3 192.168.1.103:6443 check
EOF

# Configure agents to use load balancer
curl -sfL https://get.k3s.io | K3S_URL=https://k3s-lb.example.com:6443 \
  K3S_TOKEN=<token> sh -

Common Use Cases

Key Concepts

  • Edge Computing: Deploy Kubernetes at network edge with minimal resources
  • IoT Gateways: Run containerised workloads on IoT devices
  • Development: Lightweight local Kubernetes for development and testing
  • CI/CD: Ephemeral clusters for testing in pipelines
  • ARM Devices: Native support for Raspberry Pi and ARM servers

Examples

# Edge deployment with offline installation
# Download binary on connected machine
wget https://github.com/k3s-io/k3s/releases/download/v1.28.4+k3s1/k3s
wget https://github.com/k3s-io/k3s/releases/download/v1.28.4+k3s1/k3s-airgap-images-amd64.tar

# Transfer to air-gapped machine and install
sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/
sudo cp k3s /usr/local/bin/
sudo chmod +x /usr/local/bin/k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_DOWNLOAD=true sh -

# Raspberry Pi cluster setup
# On master Pi
curl -sfL https://get.k3s.io | sh -s - \
  --write-kubeconfig-mode 644 \
  --disable servicelb \
  --disable traefik

# On worker Pis
curl -sfL https://get.k3s.io | K3S_URL=https://master-pi:6443 \
  K3S_TOKEN=<token> sh -

# Development environment with k3d (K3s in Docker)
# Install k3d
curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

# Create development cluster
k3d cluster create dev --servers 1 --agents 2 \
  --port "8080:80@loadbalancer" \
  --port "8443:443@loadbalancer"

# Delete cluster when done
k3d cluster delete dev

# CI/CD pipeline example (GitLab CI)
# .gitlab-ci.yml
stages:
  - test

integration-test:
  stage: test
  image: rancher/k3s:v1.28.4-k3s1
  services:
    - docker:dind
  script:
    - k3s server &
    - sleep 30
    - k3s kubectl apply -f manifests/
    - k3s kubectl wait --for=condition=ready pod -l app=myapp --timeout=60s
    - k3s kubectl run test --image=curlimages/curl --rm -it --restart=Never -- curl myapp-svc

# IoT gateway deployment
cat << EOF | k3s kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: iot-collector
spec:
  selector:
    matchLabels:
      app: iot-collector
  template:
    metadata:
      labels:
        app: iot-collector
    spec:
      hostNetwork: true
      containers:
      - name: collector
        image: myregistry/iot-collector:latest
        resources:
          limits:
            memory: "64Mi"
            cpu: "100m"
        volumeMounts:
        - name: device
          mountPath: /dev/ttyUSB0
      volumes:
      - name: device
        hostPath:
          path: /dev/ttyUSB0
EOF

Quick Reference

Task Command
Install K3s (single node) curl -sfL https://get.k3s.io | sh -
Get node token sudo cat /var/lib/rancher/k3s/server/node-token
Join agent node curl -sfL https://get.k3s.io | K3S_URL=https://server:6443 K3S_TOKEN=token sh -
Use kubectl k3s kubectl get nodes
Check K3s status sudo systemctl status k3s
View K3s logs sudo journalctl -u k3s -f
Uninstall server /usr/local/bin/k3s-uninstall.sh
Uninstall agent /usr/local/bin/k3s-agent-uninstall.sh
Create etcd snapshot k3s etcd-snapshot save --name backup
List etcd snapshots k3s etcd-snapshot ls
Disable component --disable traefik
Add TLS SAN --tls-san hostname
Set cluster CIDR --cluster-cidr 10.42.0.0/16
External database --datastore-endpoint "postgres://..."
Initialise HA cluster --cluster-init
Config file location /etc/rancher/k3s/config.yaml
Kubeconfig location /etc/rancher/k3s/k3s.yaml
Data directory /var/lib/rancher/k3s/

Common Issues and Solutions

Installation Failures

Problem: Installation script fails with network errors

# Solution: Use offline installation
# Download on connected machine
wget https://github.com/k3s-io/k3s/releases/download/v1.28.4+k3s1/k3s
wget https://get.k3s.io -O install.sh

# Transfer and install
chmod +x k3s && sudo mv k3s /usr/local/bin/
chmod +x install.sh && INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh

Node Not Joining Cluster

Problem: Agent cannot connect to server

# Check firewall rules (required ports: 6443, 8472 UDP for Flannel)
sudo ufw allow 6443/tcp
sudo ufw allow 8472/udp
sudo ufw allow 10250/tcp

# Verify server is accessible
curl -k https://server-ip:6443

# Check token is correct
sudo cat /var/lib/rancher/k3s/server/node-token

# Verify DNS resolution
nslookup server-hostname

Traefik Ingress Not Working

Problem: Ingress routes not responding

# Check Traefik pod status
k3s kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik

# View Traefik logs
k3s kubectl logs -n kube-system -l app.kubernetes.io/name=traefik

# Verify ingress configuration
k3s kubectl describe ingress <ingress-name>

# Check service endpoints
k3s kubectl get endpoints <service-name>

CoreDNS Resolution Failures

Problem: Pods cannot resolve DNS names

# Check CoreDNS pods
k3s kubectl get pods -n kube-system -l k8s-app=kube-dns

# Test DNS resolution
k3s kubectl run test --image=busybox --rm -it --restart=Never -- nslookup kubernetes

# Check CoreDNS configuration
k3s kubectl get configmap -n kube-system coredns -o yaml

# Restart CoreDNS
k3s kubectl rollout restart deployment coredns -n kube-system

High Memory Usage

Problem: K3s consuming too much memory

# Disable unused components
curl -sfL https://get.k3s.io | sh -s - \
  --disable traefik \
  --disable metrics-server \
  --disable servicelb

# Set kubelet memory limits
sudo tee -a /etc/rancher/k3s/config.yaml << EOF
kubelet-arg:
  - "eviction-hard=memory.available<100Mi"
  - "system-reserved=memory=100Mi"
EOF
sudo systemctl restart k3s

# Check what's consuming memory
k3s kubectl top pods -A --sort-by=memory

Etcd/Database Issues

Problem: Cluster loses quorum or data corruption

# List available snapshots
k3s etcd-snapshot ls

# Restore from snapshot (stops cluster)
sudo systemctl stop k3s
k3s server --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<snapshot>

# For SQLite corruption
sudo systemctl stop k3s
sudo rm /var/lib/rancher/k3s/server/db/state.db
sudo systemctl start k3s

Certificate Errors

Problem: TLS certificate errors when accessing API

# Add additional SANs
sudo tee -a /etc/rancher/k3s/config.yaml << EOF
tls-san:
  - "new-hostname.example.com"
  - "10.0.0.100"
EOF

# Regenerate certificates
sudo systemctl stop k3s
sudo rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json
sudo systemctl start k3s

# Update kubeconfig with correct server address
sed -i 's/127.0.0.1/actual-server-ip/' /etc/rancher/k3s/k3s.yaml

Storage Issues

Problem: PersistentVolumeClaims stuck in Pending

# Check storage class
k3s kubectl get storageclass

# Verify local-path provisioner is running
k3s kubectl get pods -n kube-system -l app=local-path-provisioner

# Check provisioner logs
k3s kubectl logs -n kube-system -l app=local-path-provisioner

# Manual PV creation if needed
cat << EOF | k3s kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: manual-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/manual-pv
  storageClassName: local-path
EOF

Related Topics

  • Kubernetes Fundamentals - Core Kubernetes concepts, architecture, and API resources
  • Helm - Package manager for deploying applications on K3s clusters
  • Traefik - Advanced ingress configuration and routing options
  • containerd - Container runtime configuration and troubleshooting
  • Rancher - Multi-cluster management platform that integrates with K3s
  • GitOps with Flux/ArgoCD - Continuous deployment patterns for K3s clusters