Istio Service Mesh

Comprehensive guide to Istio service mesh for traffic management, security, and observability.

Overview

Istio is an open-source service mesh that provides a uniform way to connect, secure, control, and observe microservices. It intercepts network traffic between services using sidecar proxies (Envoy) deployed alongside each application pod, enabling sophisticated traffic management, security policies, and observability without modifying application code.

⤢ zoom

Installation

Install Istio

# Download Istio
curl -L https://istio.io/downloadIstio | sh -
cd istio-1.30.1   # cd into the version the script just fetched
export PATH=$PWD/bin:$PATH

# Install with default profile
istioctl install --set profile=default -y

# Install with demo profile (for testing)
istioctl install --set profile=demo -y

# Install minimal profile
istioctl install --set profile=minimal -y

# Verify installation (control plane reachable, versions match)
istioctl version

# Check control plane status
kubectl get pods -n istio-system

Enable Sidecar Injection

# Label namespace for automatic sidecar injection
kubectl label namespace default istio-injection=enabled

# Verify label
kubectl get namespace -L istio-injection

# Manual injection for a deployment
istioctl kube-inject -f deployment.yaml | kubectl apply -f -

# Check if pod has sidecar
kubectl get pods
kubectl describe pod <pod-name> | grep envoy

Traffic Management

VirtualService

Controls how requests are routed to services within the mesh.

# Basic routing
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1

# Weighted routing for canary deployment
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews-canary
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

# URI-based routing
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
  - bookinfo.com
  http:
  - match:
    - uri:
        prefix: "/api/v1"
    route:
    - destination:
        host: api-v1
  - match:
    - uri:
        prefix: "/api/v2"
    route:
    - destination:
        host: api-v2
  - route:
    - destination:
        host: frontend

# Header-based routing
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: header-routing
spec:
  hosts:
  - myapp
  http:
  - match:
    - headers:
        x-canary:
          exact: "true"
    route:
    - destination:
        host: myapp
        subset: canary
  - route:
    - destination:
        host: myapp
        subset: stable

DestinationRule

Defines policies for traffic after routing has occurred, including load balancing, connection pool settings, and subsets.

# Define service subsets
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
  - name: v3
    labels:
      version: v3

# Circuit breaker configuration
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 2
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 40

# TLS settings for external service
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: external-api
spec:
  host: api.external.com
  trafficPolicy:
    tls:
      mode: SIMPLE
      sni: api.external.com

Retries and Timeouts

# Retry configuration
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - route:
    - destination:
        host: ratings
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: gateway-error,connect-failure,refused-stream

# Timeout configuration
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
    timeout: 10s

# Fault injection for testing
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: ratings-fault
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        percentage:
          value: 10
        fixedDelay: 5s
      abort:
        percentage:
          value: 5
        httpStatus: 500
    route:
    - destination:
        host: ratings

Traffic Mirroring

# Mirror traffic to test version
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: httpbin-mirror
spec:
  hosts:
  - httpbin
  http:
  - route:
    - destination:
        host: httpbin
        subset: v1
      weight: 100
    mirror:
      host: httpbin
      subset: v2
    mirrorPercentage:
      value: 100

Gateway Configuration

Istio Gateways manage ingress and egress traffic for the mesh.

Ingress Gateway

# Basic ingress gateway
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "bookinfo.example.com"
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
  - "bookinfo.example.com"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    route:
    - destination:
        host: productpage
        port:
          number: 9080

# HTTPS gateway with TLS
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: https-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: bookinfo-credential
    hosts:
    - "bookinfo.example.com"
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "bookinfo.example.com"
    tls:
      httpsRedirect: true

# Multi-host gateway
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: multi-host-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https-api
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: api-credential
    hosts:
    - "api.example.com"
  - port:
      number: 443
      name: https-web
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: web-credential
    hosts:
    - "www.example.com"

Egress Gateway

# External service access via egress
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-api
spec:
  hosts:
  - api.external.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: egress-gateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - api.external.com
    tls:
      mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: external-api-through-egress
spec:
  hosts:
  - api.external.com
  gateways:
  - mesh
  - egress-gateway
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        port:
          number: 443
  - match:
    - gateways:
      - egress-gateway
      port: 443
    route:
    - destination:
        host: api.external.com
        port:
          number: 443

Get Gateway Information

# Get ingress gateway IP/hostname
kubectl get svc istio-ingressgateway -n istio-system

# Get gateway configuration
kubectl get gateway -A

# Describe gateway
kubectl describe gateway <gateway-name>

# Check gateway logs
kubectl logs -n istio-system -l istio=ingressgateway

Security

Mutual TLS (mTLS)

# Enforce mTLS for entire mesh
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

# Namespace-specific mTLS
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: foo
spec:
  mtls:
    mode: STRICT

# Workload-specific mTLS with per-port configuration
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: httpbin
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  mtls:
    mode: STRICT
  portLevelMtls:
    8080:
      mode: PERMISSIVE

# Permissive mode (allow both mTLS and plaintext)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: PERMISSIVE

AuthorizationPolicy

Fine-grained access control for services in the mesh.

# Deny all by default
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: default
spec:
  {}

# Allow specific sources
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend
  namespace: default
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
  - from:
    - source:
        principals:
        - "cluster.local/ns/default/sa/frontend"

# HTTP method and path-based authorization
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: httpbin-authz
spec:
  selector:
    matchLabels:
      app: httpbin
  action: ALLOW
  rules:
  - to:
    - operation:
        methods: ["GET"]
        paths: ["/info*"]
  - to:
    - operation:
        methods: ["POST"]
        paths: ["/data"]
    from:
    - source:
        namespaces: ["trusted"]

# JWT-based authorization
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
spec:
  selector:
    matchLabels:
      app: api
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["*"]
    when:
    - key: request.auth.claims[iss]
      values: ["https://auth.example.com"]

# IP-based access control
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-ips
spec:
  selector:
    matchLabels:
      app: admin
  action: ALLOW
  rules:
  - from:
    - source:
        ipBlocks: ["10.0.0.0/8", "172.16.0.0/12"]

# Deny policy example
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-path
spec:
  selector:
    matchLabels:
      app: httpbin
  action: DENY
  rules:
  - to:
    - operation:
        paths: ["/admin*"]

RequestAuthentication

# JWT token validation
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  jwtRules:
  - issuer: "https://auth.example.com"
    jwksUri: "https://auth.example.com/.well-known/jwks.json"
    audiences:
    - "api.example.com"

Certificate Management

# Check certificate expiry
istioctl proxy-config secret <pod-name> -o json | jq '[.dynamicActiveSecrets[] | select(.name == "default")] | .[0].secret.tlsCertificate.certificateChain.inlineBytes' -r | base64 -d | openssl x509 -noout -text

# Verify effective mTLS mode for a workload
istioctl experimental describe pod <pod-name>

# Check certificate rotation
kubectl get secret -n istio-system istio-ca-secret -o yaml

Observability

Metrics and Telemetry

# Enable telemetry v2
# NOTE: IstioOperator manifests are applied with `istioctl install -f <file>`
# (or `istioctl manifest generate -f`), NOT `kubectl apply` — istioctl no
# longer installs the IstioOperator CRD into the cluster.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    telemetry:
      v2:
        enabled: true
        prometheus:
          enabled: true

# Access Prometheus
kubectl port-forward -n istio-system svc/prometheus 9090:9090

# Access Grafana
kubectl port-forward -n istio-system svc/grafana 3000:3000

# View metrics for a service
kubectl exec -it <pod-name> -c istio-proxy -- curl localhost:15000/stats/prometheus

Distributed Tracing

# Configure Jaeger tracing
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        sampling: 100
        zipkin:
          address: jaeger-collector.istio-system.svc.cluster.local:9411

# Access Jaeger UI
kubectl port-forward -n istio-system svc/jaeger-query 16686:16686

# Access Kiali (service mesh visualisation)
kubectl port-forward -n istio-system svc/kiali 20001:20001

Access Logs

# Enable access logging
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
  - providers:
    - name: envoy

# View Envoy access logs
kubectl logs <pod-name> -c istio-proxy

# View Envoy stats
kubectl exec <pod-name> -c istio-proxy -- curl localhost:15000/stats

# View Envoy config
kubectl exec <pod-name> -c istio-proxy -- curl localhost:15000/config_dump

Service Mesh Visualisation

⤢ zoom

Deployment Patterns

Canary Deployment

⤢ zoom

# Step 1: Deploy v2 alongside v1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      version: v2
  template:
    metadata:
      labels:
        app: myapp
        version: v2
    spec:
      containers:
      - name: myapp
        image: myapp:v2
---
# Step 2: Define subsets in DestinationRule
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: myapp
spec:
  host: myapp
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---
# Step 3: Route 10% traffic to v2
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp-canary
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 90
    - destination:
        host: myapp
        subset: v2
      weight: 10

Blue-Green Deployment

⤢ zoom

# Initially route to blue
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp-bluegreen
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: blue
      weight: 100
---
# After testing, switch to green
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp-bluegreen
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: green
      weight: 100

A/B Testing

# Route based on user cohort
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp-ab
spec:
  hosts:
  - myapp
  http:
  - match:
    - headers:
        x-user-cohort:
          exact: "test-group"
    route:
    - destination:
        host: myapp
        subset: v2
  - route:
    - destination:
        host: myapp
        subset: v1

Dark Launch

# Shadow traffic to new version without impacting users
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp-dark
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 100
    mirror:
      host: myapp
      subset: v2
    mirrorPercentage:
      value: 100

Circuit Breaking

Connection Pool Settings

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: circuit-breaker
spec:
  host: myapp
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 2
        idleTimeout: 30s

Outlier Detection

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: outlier-detection
spec:
  host: myapp
  trafficPolicy:
    outlierDetection:
      consecutiveGatewayErrors: 5
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 40

Testing Circuit Breaker

# Generate load to trigger circuit breaker
kubectl exec -it loadgen -- /bin/bash
for i in {1..100}; do
  curl -s -o /dev/null -w "%{http_code}\n" http://myapp:8080/
done

# Check circuit breaker stats
kubectl exec <pod-name> -c istio-proxy -- curl localhost:15000/stats | grep myapp | grep pending

Advanced Configuration

Service Entries

# Access external service
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-database
spec:
  hosts:
  - db.example.com
  ports:
  - number: 5432
    name: postgres
    protocol: TCP
  location: MESH_EXTERNAL
  resolution: DNS

# Static endpoints for external service
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-api
spec:
  hosts:
  - api.partner.com
  addresses:
  - 203.0.113.0/24
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: STATIC
  endpoints:
  - address: 203.0.113.10
    ports:
      https: 443
  - address: 203.0.113.20
    ports:
      https: 443

Sidecar Resource

# Limit sidecar scope for performance
apiVersion: networking.istio.io/v1
kind: Sidecar
metadata:
  name: default
  namespace: prod
spec:
  outboundTrafficPolicy:
    mode: REGISTRY_ONLY
  egress:
  - hosts:
    - "./*"
    - "istio-system/*"

Workload Entry

# Register VM workload in mesh
apiVersion: networking.istio.io/v1
kind: WorkloadEntry
metadata:
  name: vm-workload
  namespace: default
spec:
  address: 192.168.1.10
  labels:
    app: legacy-app
    version: v1
  serviceAccount: vm-sa

EnvoyFilter

# Custom Envoy configuration
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: custom-header
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inline_code: |
            function envoy_on_request(request_handle)
              request_handle:headers():add("x-custom-header", "istio")
            end

Troubleshooting Commands

# Check Istio installation (client/control-plane versions)
istioctl version

# Analyze configuration for issues
istioctl analyze

# Analyze specific namespace
istioctl analyze -n production

# Validate configuration before applying
istioctl validate -f virtualservice.yaml

# Check proxy status
istioctl proxy-status

# Get proxy configuration
istioctl proxy-config cluster <pod-name>
istioctl proxy-config listener <pod-name>
istioctl proxy-config route <pod-name>
istioctl proxy-config endpoint <pod-name>

# Debug specific service routing
istioctl proxy-config routes <pod-name> --name <route-name> -o json

# Experimental commands for debugging (also shows effective mTLS / PeerAuthentication)
istioctl experimental describe pod <pod-name>
istioctl experimental describe service <service-name>

# Dashboard for config analysis
istioctl dashboard controlz <istiod-pod>

# Get Envoy logs
kubectl logs <pod-name> -c istio-proxy -f

# Enable debug logging for Envoy
istioctl proxy-config log <pod-name> --level debug

# Check why traffic is not routing
istioctl experimental metrics <pod-name>

Performance Tuning

Resource Limits

# Configure sidecar resources
apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/proxyCPU: "100m"
    sidecar.istio.io/proxyCPULimit: "2000m"
    sidecar.istio.io/proxyMemory: "128Mi"
    sidecar.istio.io/proxyMemoryLimit: "1Gi"
spec:
  containers:
  - name: myapp
    image: myapp:latest

Sidecar Injection Control

# Disable sidecar injection for specific pod
apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  containers:
  - name: myapp
    image: myapp:latest

Mesh Configuration

# Global mesh config optimisation
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
    accessLogEncoding: JSON
    defaultConfig:
      concurrency: 2
      holdApplicationUntilProxyStarts: true
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY

Quick Reference

Common Commands

Task	Command
Install Istio	istioctl install --set profile=default -y
Enable sidecar injection	kubectl label namespace default istio-injection=enabled
Check proxy status	istioctl proxy-status
Analyse configuration	istioctl analyze
Validate config file	istioctl validate -f <file>
Check mTLS status	istioctl experimental describe pod <pod>
Get proxy config	istioctl proxy-config <cluster|listener|route|endpoint> <pod>
Enable debug logging	istioctl proxy-config log <pod> --level debug
Dashboard	istioctl dashboard <kiali|grafana|jaeger|prometheus>
Describe service	istioctl experimental describe service <name>

Resource Types

Resource	Purpose
VirtualService	Route traffic to services
DestinationRule	Configure service subsets and traffic policies
Gateway	Configure ingress/egress
ServiceEntry	Add external services to mesh
PeerAuthentication	Configure mTLS
AuthorizationPolicy	Access control policies
RequestAuthentication	JWT validation
Sidecar	Configure sidecar scope
EnvoyFilter	Custom Envoy configuration
Telemetry	Configure observability

Traffic Policy Options

Policy	Description
ROUND_ROBIN	Distribute requests evenly
LEAST_REQUEST	Send to least busy instance
RANDOM	Random distribution
PASSTHROUGH	Forward without load balancing
CONSISTENT_HASH	Hash-based routing (session affinity)

mTLS Modes

Mode	Behaviour
STRICT	Only accept mTLS traffic
PERMISSIVE	Accept both mTLS and plaintext
DISABLE	Only accept plaintext

Common Issues and Solutions

Sidecar Not Injected

Symptom: Pod has only one container, no Envoy sidecar

# Check namespace label
kubectl get namespace -L istio-injection

# Add label if missing
kubectl label namespace default istio-injection=enabled

# Restart pods to inject sidecar
kubectl rollout restart deployment/<name>

# Check for pod annotation override
kubectl get pod <name> -o yaml | grep sidecar.istio.io/inject

503 Service Unavailable

Symptom: Services returning 503 errors

# Check if destination rule subsets match pod labels
istioctl proxy-config endpoints <pod-name> | grep <service-name>

# Verify mTLS configuration (effective mode shown in output)
istioctl experimental describe pod <pod-name>

# Check for circuit breaker triggering
kubectl logs <pod-name> -c istio-proxy | grep -i "upstream_rq_pending_overflow"

# Analyse routing rules
istioctl analyze -n <namespace>

mTLS Connection Failures

Symptom: Connection refused or TLS errors

# Check peer authentication policy
kubectl get peerauthentication -A

# Verify certificates are valid
istioctl proxy-config secret <pod-name> -o json

# Check if mTLS mode conflicts exist
istioctl experimental describe pod <source-pod>

# Ensure both sides have sidecars
istioctl proxy-status | grep <pod-name>

Configuration Not Applied

Symptom: Changes to VirtualService/DestinationRule not taking effect

# Check for configuration errors
istioctl analyze

# Verify configuration was accepted
kubectl get virtualservice,destinationrule -A

# Check proxy sync status
istioctl proxy-status

# Force configuration sync
kubectl delete pod <pod-name>

# Check Envoy has received config
istioctl proxy-config routes <pod-name> -o json

High Latency

Symptom: Increased request latency after enabling Istio

# Check sidecar resource usage
kubectl top pod <pod-name> --containers

# Increase sidecar resources
kubectl annotate pod <pod-name> sidecar.istio.io/proxyCPU="500m"

# Reduce telemetry overhead
# Decrease sampling rate in mesh config

# Check for retry storms
kubectl logs <pod-name> -c istio-proxy | grep retry

# Limit sidecar scope
# Apply Sidecar resource with REGISTRY_ONLY

Certificate Rotation Issues

Symptom: Services fail after certificate expiry

# Check certificate expiry time
istioctl proxy-config secret <pod-name> -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 -d | openssl x509 -noout -dates

# Restart istiod for cert refresh
kubectl rollout restart deployment/istiod -n istio-system

# Restart workload pods
kubectl rollout restart deployment/<name>

# Check citadel logs
kubectl logs -n istio-system -l app=istiod

Gateway Not Accessible

Symptom: Cannot access services through ingress gateway

# Check gateway pod is running
kubectl get pods -n istio-system -l istio=ingressgateway

# Get gateway external IP
kubectl get svc istio-ingressgateway -n istio-system

# Verify gateway configuration
kubectl get gateway -A
istioctl analyze

# Check gateway logs
kubectl logs -n istio-system -l istio=ingressgateway

# Verify VirtualService is bound to gateway
kubectl get virtualservice <name> -o yaml | grep gateways

# Test internal connectivity
kubectl exec -it <pod-name> -- curl http://istio-ingressgateway.istio-system

Memory/CPU Issues

Symptom: Pods consuming excessive resources

# Check current resource usage
kubectl top pods -A | grep istio-proxy

# Set resource limits
kubectl annotate pod <pod-name> \
  sidecar.istio.io/proxyCPULimit="1000m" \
  sidecar.istio.io/proxyMemoryLimit="512Mi"

# Reduce connection pool size in DestinationRule
# Limit sidecar scope with Sidecar resource
# Use REGISTRY_ONLY for outbound traffic

# Check for config bloat
istioctl proxy-config all <pod-name> -o json | jq '. | length'

DNS Resolution Failures

Symptom: Services cannot resolve DNS names

# Check if service entry exists for external services
kubectl get serviceentry -A

# Test DNS from pod
kubectl exec -it <pod-name> -c istio-proxy -- nslookup <service-name>

# Check Envoy cluster configuration
istioctl proxy-config cluster <pod-name> --fqdn <service-name>

# Verify outbound traffic policy
kubectl get configmap istio -n istio-system -o yaml | grep outboundTrafficPolicy

Debugging Workflow

⤢ zoom

Best Practices

1. Start with PERMISSIVE mTLS then move to STRICT after validation
2. Use namespace-level policies for consistency, override at workload level only when needed
3. Implement circuit breakers on all external service calls
4. Set appropriate timeouts and retries to prevent cascading failures
5. Use REGISTRY_ONLY outbound traffic policy to reduce configuration size
6. Monitor resource usage of sidecars and adjust limits accordingly
7. Test configuration changes with istioctl analyze before applying
8. Use canary deployments for risky changes with gradual traffic shifting
9. Implement observability early - enable metrics, traces, and logs
10. Regularly update Istio to get security fixes and performance improvements
11. Use Gateway for ingress rather than exposing services directly
12. Document AuthorizationPolicies clearly as they can block legitimate traffic
13. Test circuit breakers under load to ensure correct thresholds
14. Backup configuration before major changes
15. Use specific host matching in VirtualServices to avoid routing conflicts

Related Technologies

• Kubernetes: Container orchestration platform that Istio runs on
• Envoy Proxy: Data plane proxy used by Istio
• Prometheus: Metrics collection and storage
• Grafana: Metrics visualisation and dashboards
• Jaeger: Distributed tracing system
• Kiali: Service mesh observability and management console
• Cert-Manager: Certificate management for Kubernetes
• Linkerd: Alternative service mesh implementation
• Consul: Service mesh and service discovery platform