Consul

Service mesh and service discovery platform providing service registration, health checking, and distributed key/value storage.

Overview

Consul is a distributed, highly available system designed for service discovery, configuration, and orchestration. It provides a full-featured control plane with service mesh functionality, health checking, key/value storage, and multi-datacenter support. Consul agents run on every node in the infrastructure, communicating with servers via the gossip protocol (Serf) and Raft consensus for state replication.

⤢ zoom

Service Registration and Discovery

Consul's core functionality enables services to register themselves and discover other services in the network.

Key Concepts

• Service Definition - JSON/HCL configuration describing a service and its metadata
• Service Catalog - Central registry of all services across the cluster
• Service Tags - Labels for filtering and routing (e.g., primary, v2)
• Service Mesh - Automatic sidecar proxy injection for secure service-to-service communication
• Anti-entropy - Agents periodically sync local state with servers

⤢ zoom

Common Commands/Patterns

# Register a service via CLI
consul services register /etc/consul.d/web.json

# Deregister a service
consul services deregister -id=web-1

# List all registered services
consul catalog services

# Get detailed info about a service
consul catalog nodes -service=web

# List service instances with health status
consul catalog services -tags

# Query service via HTTP API
curl http://localhost:8500/v1/catalog/service/web

# Query healthy instances only
curl http://localhost:8500/v1/health/service/web?passing

Examples

Service Definition File

{
  "service": {
    "id": "web-1",
    "name": "web",
    "tags": ["primary", "v1.2.3"],
    "address": "192.168.1.10",
    "port": 8080,
    "meta": {
      "version": "1.2.3",
      "environment": "production"
    },
    "weights": {
      "passing": 10,
      "warning": 1
    },
    "enable_tag_override": false,
    "check": {
      "http": "http://localhost:8080/health",
      "interval": "10s",
      "timeout": "5s"
    }
  }
}

HCL Service Definition

# /etc/consul.d/web.hcl
service {
  id      = "web-1"
  name    = "web"
  tags    = ["primary", "v1.2.3"]
  address = "192.168.1.10"
  port    = 8080

  meta {
    version     = "1.2.3"
    environment = "production"
  }

  check {
    http     = "http://localhost:8080/health"
    interval = "10s"
    timeout  = "5s"
  }

  # Connect sidecar proxy configuration
  connect {
    sidecar_service {
      proxy {
        upstreams {
          destination_name = "database"
          local_bind_port  = 5432
        }
      }
    }
  }
}

Multiple Services Registration

# /etc/consul.d/services.hcl
services {
  id      = "api-1"
  name    = "api"
  port    = 3000
  tags    = ["production"]

  check {
    http     = "http://localhost:3000/healthz"
    interval = "10s"
  }
}

services {
  id      = "worker-1"
  name    = "worker"
  port    = 4000
  tags    = ["production"]

  check {
    tcp      = "localhost:4000"
    interval = "15s"
  }
}

Health Checks

Consul provides multiple health check types to monitor service availability and report status to the cluster.

Key Concepts

• Check Types - HTTP, TCP, gRPC, Script, TTL, Docker, Alias
• Check Status - passing, warning, critical
• Deregister Critical Service - Automatically remove unhealthy services after timeout
• Local Checks - Run on the agent where the service is registered
• Session-based Checks - Tied to Consul sessions for distributed locking

⤢ zoom

Common Commands/Patterns

# View all health checks
consul catalog services

# Check specific service health
curl http://localhost:8500/v1/health/service/web

# List all checks on a node
curl http://localhost:8500/v1/agent/checks

# Update TTL check status
curl -X PUT http://localhost:8500/v1/agent/check/pass/service:web-1

# Set check to warning
curl -X PUT http://localhost:8500/v1/agent/check/warn/service:web-1

# Set check to critical
curl -X PUT http://localhost:8500/v1/agent/check/fail/service:web-1

# Register a service via CLI flags (checks require a definition file, not flags)
consul services register -name=web -port=8080

# To register with a health check, use a definition file (see Examples below)
consul services register /etc/consul.d/web.json

Examples

HTTP Health Check

{
  "check": {
    "id": "web-http-check",
    "name": "HTTP Health Check",
    "http": "http://localhost:8080/health",
    "method": "GET",
    "header": {
      "Authorization": ["Bearer token123"]
    },
    "interval": "10s",
    "timeout": "5s",
    "deregister_critical_service_after": "1m"
  }
}

TCP Health Check

check {
  id       = "db-tcp"
  name     = "PostgreSQL TCP Check"
  tcp      = "localhost:5432"
  interval = "10s"
  timeout  = "3s"
}

gRPC Health Check

check {
  id          = "api-grpc"
  name        = "gRPC Health"
  grpc        = "localhost:50051"
  grpc_use_tls = true
  interval    = "10s"
  timeout     = "5s"
}

Script Health Check

Script (args) and Docker checks are disabled by default for security. Start the agent with -enable-local-script-checks (config-file checks only) or -enable-script-checks (also allows API-registered scripts; less safe), or set enable_local_script_checks = true in config — otherwise the agent refuses to start or rejects the registration.

check {
  id       = "memory-check"
  name     = "Memory Usage"
  args     = ["/usr/local/bin/check_memory.sh", "--warn=80", "--crit=90"]
  interval = "30s"
  timeout  = "10s"
}

TTL Health Check

# Service must call API to update status
check {
  id   = "worker-ttl"
  name = "Worker Heartbeat"
  ttl  = "30s"
  notes = "Worker must send heartbeat every 30 seconds"
}

# Application updates TTL check (output is set via the ?note= query param)
curl -X PUT "http://localhost:8500/v1/agent/check/pass/worker-ttl?note=Worker+processed+150+jobs"

Docker Health Check

check {
  id                     = "redis-docker"
  name                   = "Redis Docker Check"
  docker_container_id    = "redis-container"
  args                   = ["redis-cli", "ping"]
  interval               = "10s"
  shell                  = "/bin/bash"
}

Multiple Checks per Service

service {
  id   = "api-1"
  name = "api"
  port = 8080

  checks = [
    {
      http     = "http://localhost:8080/health"
      interval = "10s"
      name     = "HTTP Health"
    },
    {
      http     = "http://localhost:8080/ready"
      interval = "5s"
      name     = "Readiness Check"
    },
    {
      tcp      = "localhost:8080"
      interval = "30s"
      name     = "TCP Port Check"
    }
  ]
}

Key/Value Store Operations

Consul's distributed key/value store provides dynamic configuration, feature flags, and coordination primitives.

Key Concepts

• Hierarchical Keys - Organised using / as path separator
• Consistency Modes - default, consistent, stale
• Blocking Queries - Long-poll for changes using index
• CAS (Check-And-Set) - Atomic conditional updates
• Sessions - TTL-based locks for distributed coordination
• Watches - React to KV changes automatically

⤢ zoom

Common Commands/Patterns

# Set a key
consul kv put config/database/host "db.example.com"

# Set key with flags
consul kv put -flags=42 config/api/timeout "30"

# Get a key value
consul kv get config/database/host

# Get key with metadata
consul kv get -detailed config/database/host

# Get all keys with prefix
consul kv get -recurse config/

# List keys only (no values)
consul kv get -keys config/

# Delete a key
consul kv delete config/database/host

# Delete all keys with prefix
consul kv delete -recurse config/database/

# Export KV data
consul kv export config/ > backup.json

# Import KV data
consul kv import @backup.json

# Atomic check-and-set
consul kv put -cas -modify-index=123 config/key "new-value"

Examples

HTTP API Operations

# Create/Update key
curl -X PUT -d 'postgresql://db.example.com:5432/mydb' \
  http://localhost:8500/v1/kv/config/database/connection_string

# Get key (base64 encoded value)
curl http://localhost:8500/v1/kv/config/database/connection_string

# Get key with raw value
curl http://localhost:8500/v1/kv/config/database/connection_string?raw

# Delete key
curl -X DELETE http://localhost:8500/v1/kv/config/database/connection_string

# List all keys with prefix
curl http://localhost:8500/v1/kv/config/?keys

# Check-and-set (atomic update)
curl -X PUT -d 'new-value' \
  "http://localhost:8500/v1/kv/config/key?cas=123"

# Acquire lock with session
curl -X PUT -d 'lock-holder-id' \
  "http://localhost:8500/v1/kv/locks/mylock?acquire=session-id"

# Release lock
curl -X PUT \
  "http://localhost:8500/v1/kv/locks/mylock?release=session-id"

Blocking Queries for Changes

# Get current index
INDEX=$(curl -s http://localhost:8500/v1/kv/config/key | jq -r '.[0].ModifyIndex')

# Block until change (long-poll)
curl "http://localhost:8500/v1/kv/config/key?index=${INDEX}&wait=5m"

Session-based Locking

# Create session
SESSION=$(curl -X PUT -d '{"Name": "my-lock", "TTL": "15s"}' \
  http://localhost:8500/v1/session/create | jq -r '.ID')

# Acquire lock
curl -X PUT -d "$(hostname)" \
  "http://localhost:8500/v1/kv/locks/critical-section?acquire=${SESSION}"

# Renew session
curl -X PUT "http://localhost:8500/v1/session/renew/${SESSION}"

# Release lock and destroy session
curl -X PUT "http://localhost:8500/v1/session/destroy/${SESSION}"

Watch for Changes

# /etc/consul.d/watches.hcl
watches = [
  {
    type       = "key"
    key        = "config/database/host"
    handler    = "/usr/local/bin/reload-config.sh"
  },
  {
    type       = "keyprefix"
    prefix     = "config/api/"
    handler_type = "http"
    http_handler_config {
      path    = "http://localhost:8080/config-changed"
      method  = "POST"
    }
  }
]

Hierarchical Configuration Example

# Set up configuration hierarchy
consul kv put config/global/log_level "info"
consul kv put config/global/timeout "30s"
consul kv put config/services/api/port "8080"
consul kv put config/services/api/replicas "3"
consul kv put config/services/worker/concurrency "10"

# Retrieve all configuration
consul kv get -recurse config/

# Application reads specific config
consul kv get config/services/api/port

Service Mesh Capabilities

Consul Connect provides service mesh functionality with automatic mTLS, traffic management, and service segmentation.

Key Concepts

• Connect - Native service mesh built into Consul
• Sidecar Proxy - Envoy proxy handling service-to-service traffic
• Intentions - Service-level access control (allow/deny)
• mTLS - Automatic mutual TLS between services
• Service Segmentation - Isolate services by namespace and partition
• Traffic Management - Routing, splitting, and failover

⤢ zoom

Common Commands/Patterns

# Enable Connect via config (on by default for server agents); no CLI flag exists.
# In a config file: connect { enabled = true }
consul agent -config-dir=/etc/consul.d

# Register service with sidecar
consul services register /etc/consul.d/web-connect.hcl

# List all intentions
consul intention list

# Create allow intention
consul intention create web database

# Create deny intention
consul intention create -deny web secrets

# Delete intention
consul intention delete web database

# Check intention between services
consul intention check web database

# Get intention details
consul intention get web database

# List all sidecar proxies (registered as <service>-sidecar-proxy; filter by Kind)
curl -s -G http://localhost:8500/v1/agent/services \
  --data-urlencode 'filter=Kind=="connect-proxy"'

Examples

Service with Sidecar Proxy

# /etc/consul.d/web-connect.hcl
service {
  name = "web"
  port = 8080

  connect {
    sidecar_service {
      port = 21000

      proxy {
        # Upstream service dependencies
        upstreams {
          destination_name = "api"
          local_bind_port  = 9001
        }

        upstreams {
          destination_name = "database"
          local_bind_port  = 5432

          # Optional datacenter
          datacenter = "dc2"
        }

        # Proxy configuration
        config {
          protocol                  = "http"
          local_connect_timeout_ms  = 5000
          handshake_timeout_ms      = 10000
        }
      }
    }
  }
}

Intention Configuration (L4)

# /etc/consul.d/intentions.hcl
Kind = "service-intentions"
Name = "database"

Sources = [
  {
    Name   = "api"
    Action = "allow"
  },
  {
    Name   = "web"
    Action = "deny"
  },
  {
    # Default deny for everything else
    Name   = "*"
    Action = "deny"
  }
]

L7 Traffic Management

# Service router for path-based routing
Kind = "service-router"
Name = "api"

Routes = [
  {
    Match {
      HTTP {
        PathPrefix = "/v2/"
      }
    }
    Destination {
      Service = "api-v2"
    }
  },
  {
    Match {
      HTTP {
        Header = [
          {
            Name  = "x-canary"
            Exact = "true"
          }
        ]
      }
    }
    Destination {
      Service       = "api-canary"
      ServiceSubset = "canary"
    }
  }
]

Traffic Splitting (Canary Deployment)

# Service splitter for weighted routing
Kind = "service-splitter"
Name = "api"

Splits = [
  {
    Weight  = 90
    Service = "api"
    ServiceSubset = "stable"
  },
  {
    Weight  = 10
    Service = "api"
    ServiceSubset = "canary"
  }
]

Service Resolver

# Define service subsets and failover
Kind = "service-resolver"
Name = "api"

# Define subsets based on metadata
Subsets = {
  stable = {
    Filter = "Service.Meta.version == v1"
  }
  canary = {
    Filter = "Service.Meta.version == v2"
  }
}

# Failover configuration
Failover = {
  "*" = {
    Datacenters = ["dc2", "dc3"]
  }
}

# Load balancing policy
LoadBalancer = {
  Policy = "least_request"
  LeastRequestConfig = {
    ChoiceCount = 2
  }
}

Service Defaults

Kind = "service-defaults"
Name = "api"

Protocol = "http"

# Mesh gateway mode
MeshGateway = {
  Mode = "local"
}

# Circuit breaker settings
UpstreamConfig = {
  Defaults = {
    Limits = {
      MaxConnections        = 100
      MaxPendingRequests    = 100
      MaxConcurrentRequests = 100
    }
    PassiveHealthCheck = {
      Interval     = "10s"
      MaxFailures  = 5
    }
  }
}

DNS Interface

Consul provides a built-in DNS server for service discovery, allowing applications to find services using standard DNS queries.

Key Concepts

• Service DNS - Query services via <service>.service.consul
• Node DNS - Query nodes via <node>.node.consul
• Tag Filtering - Filter by tag: <tag>.<service>.service.consul
• Datacenter Queries - Query other DCs: <service>.service.<dc>.consul
• Prepared Queries - Complex queries accessible via DNS
• DNS Forwarding - Forward .consul queries from system resolver

⤢ zoom

Common Commands/Patterns

# Query service A record
dig @127.0.0.1 -p 8600 web.service.consul

# Query service SRV record (includes port)
dig @127.0.0.1 -p 8600 web.service.consul SRV

# Query with tag filter
dig @127.0.0.1 -p 8600 primary.web.service.consul

# Query specific datacenter
dig @127.0.0.1 -p 8600 web.service.dc2.consul

# Query node
dig @127.0.0.1 -p 8600 node1.node.consul

# Query prepared query
dig @127.0.0.1 -p 8600 my-query.query.consul

# Query Connect-enabled service
dig @127.0.0.1 -p 8600 web.connect.consul

# List all nodes
dig @127.0.0.1 -p 8600 consul.service.consul

# Use nslookup
nslookup -port=8600 web.service.consul 127.0.0.1

Examples

DNS Configuration

# /etc/consul.d/consul.hcl
dns_config {
  # Allow stale reads for better performance
  allow_stale = true
  max_stale   = "5s"

  # Cache settings
  node_ttl         = "30s"
  service_ttl {
    "*" = "10s"
  }

  # Only return passing services
  only_passing = true

  # Enable recursors for non-Consul domains
  recursors = ["8.8.8.8", "8.8.4.4"]

  # DNS port configuration
  enable_truncate = true
}

# Bind DNS to all interfaces
addresses {
  dns = "0.0.0.0"
}

ports {
  dns = 8600
}

systemd-resolved Integration

# /etc/systemd/resolved.conf.d/consul.conf
[Resolve]
DNS=127.0.0.1:8600
Domains=~consul

# Restart resolved
sudo systemctl restart systemd-resolved

# Verify
resolvectl status

dnsmasq Integration

# /etc/dnsmasq.d/10-consul
# Forward .consul queries to Consul
server=/consul/127.0.0.1#8600

# Restart dnsmasq
sudo systemctl restart dnsmasq

Prepared Query for Geo-Failover

# Create prepared query with failover
curl -X POST -d '{
  "Name": "geo-api",
  "Service": {
    "Service": "api",
    "Failover": {
      "NearestN": 3,
      "Datacenters": ["dc2", "dc3"]
    },
    "OnlyPassing": true,
    "Tags": ["production"]
  },
  "DNS": {
    "TTL": "10s"
  }
}' http://localhost:8500/v1/query

# Query via DNS
dig @127.0.0.1 -p 8600 geo-api.query.consul

Application DNS Usage

# Python example using DNS for service discovery
import socket
import dns.resolver

def get_service_endpoints(service_name, consul_dns="127.0.0.1", port=8600):
    """Resolve service endpoints via Consul DNS."""
    resolver = dns.resolver.Resolver()
    resolver.nameservers = [consul_dns]
    resolver.port = port

    # Get SRV records for service with port info
    srv_records = resolver.resolve(
        f"{service_name}.service.consul",
        "SRV"
    )

    endpoints = []
    for record in srv_records:
        host = str(record.target).rstrip('.')
        port = record.port
        endpoints.append((host, port))

    return endpoints

# Usage
api_endpoints = get_service_endpoints("api")
for host, port in api_endpoints:
    print(f"API endpoint: {host}:{port}")

ACLs and Security

Consul's ACL system controls access to data and APIs, providing authentication and authorisation for all operations.

Key Concepts

• Tokens - Bearer tokens for authentication
• Policies - Rules defining permitted operations
• Roles - Collections of policies
• Auth Methods - External authentication (JWT, OIDC, Kubernetes)
• Namespaces - Multi-tenant isolation (Enterprise)
• Bootstrap Token - Initial management token

⤢ zoom

Common Commands/Patterns

# Bootstrap ACL system (first time only)
consul acl bootstrap

# Create ACL policy
consul acl policy create -name "web-policy" -rules @web-policy.hcl

# List policies
consul acl policy list

# Read policy
consul acl policy read -name "web-policy"

# Create token with policy
consul acl token create -description "Web service token" \
  -policy-name "web-policy"

# List tokens
consul acl token list

# Create role
consul acl role create -name "web-role" -policy-name "web-policy"

# Set token for CLI
export CONSUL_HTTP_TOKEN="your-token-here"

# Validate token
consul acl token read -self

# Update token
consul acl token update -id "token-id" -policy-name "new-policy"

# Delete token
consul acl token delete -id "token-id"

Examples

Enable ACLs

# /etc/consul.d/acl.hcl
acl {
  enabled                  = true
  default_policy           = "deny"
  enable_token_persistence = true

  tokens {
    # Agent token for internal operations
    agent = "agent-token-here"

    # Default token for requests without explicit token
    default = "default-token-here"
  }
}

Policy Examples

# web-policy.hcl - Policy for web service
# Read/write own service
service "web" {
  policy = "write"
}

service "web-sidecar-proxy" {
  policy = "write"
}

# Read other services for discovery
service_prefix "" {
  policy = "read"
}

# Read nodes for health checks
node_prefix "" {
  policy = "read"
}

# Read/write own KV prefix
key_prefix "config/web/" {
  policy = "write"
}

# Read shared configuration
key_prefix "config/shared/" {
  policy = "read"
}

# operator-policy.hcl - Operator/admin policy
# Full access to ACL system
acl = "write"

# Manage all services
service_prefix "" {
  policy = "write"
}

# Manage all KV data
key_prefix "" {
  policy = "write"
}

# Manage all nodes
node_prefix "" {
  policy = "write"
}

# Operator actions
operator = "write"

# Prepared queries
query_prefix "" {
  policy = "write"
}

# read-only-policy.hcl - Read-only for monitoring
service_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

key_prefix "" {
  policy = "read"
}

# Access agent info
agent_prefix "" {
  policy = "read"
}

Kubernetes Auth Method

# Configure Kubernetes auth method
consul acl auth-method create \
  -name "kubernetes" \
  -type "kubernetes" \
  -config '{
    "Host": "https://kubernetes.default.svc",
    "CACert": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
    "ServiceAccountJWT": "eyJ..."
  }'

# Create binding rule
consul acl binding-rule create \
  -method="kubernetes" \
  -bind-type="service" \
  -bind-name="${serviceaccount.name}" \
  -selector="serviceaccount.namespace==default"

TLS Configuration

# /etc/consul.d/tls.hcl
# Enable TLS for all communications
tls {
  defaults {
    ca_file         = "/etc/consul.d/certs/consul-ca.pem"
    cert_file       = "/etc/consul.d/certs/server.pem"
    key_file        = "/etc/consul.d/certs/server-key.pem"
    verify_incoming = true
    verify_outgoing = true
  }

  internal_rpc {
    verify_server_hostname = true
  }

  https {
    verify_incoming = false  # Allow unauthenticated UI access
  }
}

# HTTPS API
ports {
  https = 8501
}

Agent Token Setup

# Create agent policy
cat > agent-policy.hcl << 'EOF'
node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}
EOF

consul acl policy create -name "agent-policy" -rules @agent-policy.hcl

# Create agent token
consul acl token create -description "Agent token" \
  -policy-name "agent-policy" \
  -secret "your-agent-token-secret"

# Set agent token
consul acl set-agent-token agent "your-agent-token-secret"

Common CLI Commands

Essential commands for managing Consul clusters, services, and configuration.

Key Concepts

• consul agent - Runs the Consul agent
• consul members - List cluster members
• consul operator - Cluster operator commands
• consul snapshot - Backup and restore
• consul monitor - Stream logs from agent

Agent and Cluster Management

# Start agent in dev mode (testing only)
consul agent -dev

# Start agent as server
consul agent -server -bootstrap-expect=3 \
  -data-dir=/var/consul \
  -config-dir=/etc/consul.d \
  -bind=192.168.1.10

# Start agent as client
consul agent \
  -data-dir=/var/consul \
  -config-dir=/etc/consul.d \
  -join=192.168.1.10

# Join cluster
consul join 192.168.1.10

# Leave cluster gracefully
consul leave

# Force remove failed node
consul force-leave node-name

# List cluster members
consul members

# List members with detailed info
consul members -detailed

# Check cluster status
consul operator raft list-peers

# View agent info
consul info

# Reload configuration
consul reload

# Stream agent logs
consul monitor -log-level=debug

# Validate configuration
consul validate /etc/consul.d/

Snapshot and Backup

# Create snapshot
consul snapshot save backup.snap

# Restore snapshot
consul snapshot restore backup.snap

# Inspect snapshot
consul snapshot inspect backup.snap

# Automated backup script
#!/bin/bash
BACKUP_DIR="/var/backups/consul"
DATE=$(date +%Y%m%d_%H%M%S)
consul snapshot save "${BACKUP_DIR}/consul_${DATE}.snap"
find "${BACKUP_DIR}" -name "*.snap" -mtime +7 -delete

Debug and Troubleshooting

# Generate debug bundle
consul debug -duration=30s -interval=5s

# View leader
consul operator raft list-peers | grep leader

# Check autopilot status
consul operator autopilot state

# View licence (Enterprise)
consul license get

# Keyring management (gossip encryption)
consul keyring -list
consul keyring -install="new-key"
consul keyring -use="new-key"
consul keyring -remove="old-key"

Event and Watch Commands

# Fire custom event (payload is a positional argument, not a flag)
consul event -name=deploy "v1.2.3"

# Watch for events
consul watch -type=event -name=deploy /usr/local/bin/deploy.sh

# Watch service changes
consul watch -type=service -service=web /usr/local/bin/update-lb.sh

# Watch KV prefix
consul watch -type=keyprefix -prefix=config/ /usr/local/bin/reload.sh

Namespace Commands (Enterprise)

# Create namespace
consul namespace create -name production

# List namespaces
consul namespace list

# Set namespace context
export CONSUL_NAMESPACE=production

# Use namespace with command
consul catalog services -namespace=production

Quick Reference

Operation	Command
Start dev agent	consul agent -dev
Join cluster	consul join <address>
List members	consul members
Register service	consul services register <file>
List services	consul catalog services
Query service DNS	dig @127.0.0.1 -p 8600 <service>.service.consul
Get KV value	consul kv get <key>
Set KV value	consul kv put <key> <value>
Delete KV	consul kv delete <key>
Create ACL policy	consul acl policy create -name <name> -rules @<file>
Create ACL token	consul acl token create -policy-name <policy>
Create intention	consul intention create <source> <destination>
Check service health	curl http://localhost:8500/v1/health/service/<name>
Create snapshot	consul snapshot save <file>
Reload config	consul reload
Stream logs	consul monitor -log-level=debug

Common Issues and Solutions

Issue	Cause	Solution
No cluster leader	Servers unable to elect leader	Ensure majority of servers are up; check network connectivity between servers
No known Consul servers	Client cannot find servers	Verify -join address; check firewall allows ports 8300-8302
ACL not found	Token doesn't exist or expired	Create new token with consul acl token create; check token persistence
Permission denied	Insufficient ACL permissions	Review policy rules; check token has required permissions
Service not found in DNS	Service not registered or unhealthy	Verify registration with consul catalog services; check health checks
No route to host for Connect	Sidecar proxy not running	Ensure Envoy sidecar is running; check proxy configuration
TLS handshake error	Certificate mismatch	Verify CA, cert, and key match; check verify_server_hostname
Raft timeout	Network latency too high	Check network between servers; consider increasing timeouts
Too many open files	ulimit too low	Increase file descriptor limits: ulimit -n 65535
Checkpoint failure	Disk full or permissions	Check disk space; verify data directory permissions
RPC error: rpc error making call	Server overloaded	Scale server resources; review query patterns
Coordinate update blocked	Network coordinates failing	Check UDP port 8301 between agents
Session invalidated	TTL expired	Increase session TTL; ensure client renews session
Intention denied	No intention allowing traffic	Create allow intention: consul intention create <src> <dst>
Connect CA not initialized	Connect not bootstrapped	Ensure servers have connect { enabled = true } (on by default for servers)