Load Testing (k6/Locust/JMeter)

Performance testing tools for simulating user load, measuring system behaviour, and identifying bottlenecks under stress.

Overview

Load testing validates system performance by simulating concurrent users and measuring response times, throughput, and resource utilisation under various load conditions. Modern tools like k6 (JavaScript), Locust (Python), and JMeter (Java/GUI) offer different approaches to test design, from code-first frameworks to GUI-based test builders, each suited for different team workflows and infrastructure requirements.

graph TB
    subgraph "Load Testing Architecture"
        A[Test Script] --> B[Load Generator]
        B --> C[Multiple Workers/VUs]
        C -->|HTTP/gRPC/WebSocket| D[Target System]
        D --> E[Response Data]
        E --> F[Metrics Collector]
        F --> G[Time Series DB]
        G --> H[Dashboard/Report]
        F --> I[Real-time Streaming]
        I --> J[Grafana/InfluxDB]
    end

    subgraph "Distributed Setup"
        K[Controller] --> L[Worker 1]
        K --> M[Worker 2]
        K --> N[Worker N]
        L --> D
        M --> D
        N --> D
    end

Tool Comparison:

Tool	Language	Best For	Strengths	CI Integration
k6	JavaScript	Modern APIs, DevOps teams	Built-in metrics, cloud integration, scriptability	Excellent (native CLI)
Locust	Python	Python developers, complex scenarios	Programmable, distributed, real-time UI	Good (Python ecosystem)
JMeter	Java/GUI	Enterprise, GUI users, protocols	Protocol support, plugins, mature ecosystem	Moderate (requires wrapper)

Test Plan Design and Parameterisation

Test plans define user scenarios, load profiles, and validation criteria to simulate realistic application usage patterns.

Key Concepts

flowchart TB
    subgraph "Test Plan Components"
        A[Scenario Definition] --> B[Load Profile]
        B --> C[Ramp-up/down]
        C --> D[Virtual Users]
        D --> E[Think Time]
        E --> F[Assertions/Checks]
        F --> G[Thresholds]
    end

    subgraph "Load Patterns"
        H[Constant Load] --> I[Staged Load]
        I --> J[Spike Testing]
        J --> K[Stress Testing]
        K --> L[Soak Testing]
    end

Load profile types:

Constant: Fixed number of users for duration
Ramping: Gradual increase/decrease in users
Spike: Sudden load increase to test resilience
Stress: Push beyond capacity to find breaking point
Soak: Extended duration to detect memory leaks

Common Patterns

k6 Test Plan:

// test.js - k6 script with stages and thresholds
import http from 'k6/http';
import { check, sleep } from 'k6';
import { SharedArray } from 'k6/data';
import { Rate, Trend, Counter } from 'k6/metrics';

// Custom metrics
const errorRate = new Rate('error_rate');
const checkoutDuration = new Trend('checkout_duration');
const failedCheckouts = new Counter('failed_checkouts');

// Load test data from CSV/JSON
const users = new SharedArray('users', function() {
    return JSON.parse(open('./users.json'));
});

// Configuration and thresholds
export const options = {
    stages: [
        { duration: '2m', target: 10 },    // Ramp-up to 10 users
        { duration: '5m', target: 10 },    // Stay at 10 users
        { duration: '2m', target: 50 },    // Ramp-up to 50 users
        { duration: '5m', target: 50 },    // Stay at 50 users
        { duration: '2m', target: 100 },   // Spike to 100 users
        { duration: '5m', target: 0 },     // Ramp-down to 0
    ],
    thresholds: {
        'http_req_duration': ['p(95)<500', 'p(99)<1000'],  // 95% < 500ms, 99% < 1s
        'http_req_failed': ['rate<0.01'],                   // Error rate < 1%
        'error_rate': ['rate<0.05'],                        // Custom error rate < 5%
        'checkout_duration': ['p(95)<2000'],                // Checkout p95 < 2s
    },
    // Grafana Cloud k6 options (formerly ext.loadimpact)
    cloud: {
        stackID: 123456,    // Your Grafana Cloud stack ID (required for Grafana Cloud k6)
        projectID: 123456,  // Your k6 Cloud project ID
        name: 'API Load Test'
    }
};

// Setup function - runs once
export function setup() {
    const res = http.post('https://api.example.com/auth', {
        username: 'test',
        password: 'secret'
    });
    return { token: res.json('token') };
}

// Main test function
export default function(data) {
    const user = users[Math.floor(Math.random() * users.length)];

    // Parameterised requests
    const params = {
        headers: {
            'Authorization': `Bearer ${data.token}`,
            'Content-Type': 'application/json',
        },
        tags: { name: 'ListProducts' }
    };

    // Request with checks
    const response = http.get(
        `https://api.example.com/products?user=${user.id}`,
        params
    );

    const success = check(response, {
        'status is 200': (r) => r.status === 200,
        'response time < 500ms': (r) => r.timings.duration < 500,
        'has products': (r) => r.json('products').length > 0,
    });

    errorRate.add(!success);

    // Simulate checkout flow
    if (success && Math.random() < 0.3) {  // 30% of users checkout
        const startTime = new Date();

        const checkoutRes = http.post(
            'https://api.example.com/checkout',
            JSON.stringify({
                userId: user.id,
                items: [1, 2, 3]
            }),
            params
        );

        const checkoutSuccess = check(checkoutRes, {
            'checkout succeeded': (r) => r.status === 201,
        });

        checkoutDuration.add(new Date() - startTime);

        if (!checkoutSuccess) {
            failedCheckouts.add(1);
        }
    }

    // Think time between requests (randomised)
    sleep(Math.random() * 3 + 1);  // 1-4 seconds
}

// Teardown function - runs once at end
export function teardown(data) {
    // Cleanup resources
    http.post('https://api.example.com/logout', null, {
        headers: { 'Authorization': `Bearer ${data.token}` }
    });
}

Locust Test Plan:

# locustfile.py - Python-based load test
from locust import HttpUser, task, between, events, constant_pacing
from locust.contrib.fasthttp import FastHttpUser  # Faster for high load
import random
import json
import csv

# Load test data
with open('users.csv', 'r') as f:
    users_data = list(csv.DictReader(f))

class APIUser(HttpUser):
    """Simulated user behaviour for API testing"""

    # Wait time between tasks (simulates think time)
    wait_time = between(1, 3)  # Random wait 1-3 seconds
    # wait_time = constant_pacing(1)  # For consistent RPS

    # Host is set in command line or here
    host = "https://api.example.com"

    def on_start(self):
        """Called once per user at start - login/setup"""
        response = self.client.post("/auth", json={
            "username": "test",
            "password": "secret"
        })
        self.token = response.json()["token"]
        self.headers = {
            "Authorization": f"Bearer {self.token}",
            "Content-Type": "application/json"
        }
        self.user_data = random.choice(users_data)

    @task(3)  # Weight: 3x more likely than other tasks
    def list_products(self):
        """List products - most common action"""
        with self.client.get(
            f"/products?user={self.user_data['id']}",
            headers=self.headers,
            catch_response=True,  # Manual response handling
            name="/products?user=[id]"  # Group in metrics
        ) as response:
            if response.status_code == 200:
                products = response.json()
                if len(products['products']) > 0:
                    response.success()
                else:
                    response.failure("No products returned")
            else:
                response.failure(f"Got status {response.status_code}")

    @task(1)
    def checkout(self):
        """Checkout flow - less common, more critical"""
        # Multi-step transaction
        items = [1, 2, 3]

        # Add to cart
        with self.client.post(
            "/cart/add",
            json={"userId": self.user_data['id'], "items": items},
            headers=self.headers,
            name="/cart/add"
        ) as response:
            if response.status_code != 200:
                return  # Abort checkout if cart fails

        # Complete checkout
        with self.client.post(
            "/checkout",
            json={"userId": self.user_data['id']},
            headers=self.headers,
            name="/checkout"
        ) as response:
            if response.status_code == 201:
                # Record custom metric
                events.request.fire(
                    request_type="CHECKOUT",
                    name="checkout_flow",
                    response_time=response.elapsed.total_seconds() * 1000,
                    response_length=len(response.content),
                    exception=None,
                    context={}
                )

    @task(2)
    def search(self):
        """Search products"""
        keywords = ["laptop", "phone", "tablet", "monitor"]
        keyword = random.choice(keywords)

        self.client.get(
            f"/search?q={keyword}",
            headers=self.headers,
            name="/search?q=[keyword]"
        )

    def on_stop(self):
        """Called once per user at end - cleanup"""
        self.client.post("/logout", headers=self.headers)

class AdminUser(HttpUser):
    """Different user type with different behaviour"""
    wait_time = constant_pacing(5)  # One action every 5 seconds
    host = "https://api.example.com"
    weight = 1  # Only 1 admin per 10 regular users (if APIUser weight=10)

    @task
    def view_analytics(self):
        """Admin-only expensive query"""
        self.client.get("/admin/analytics")

# Custom shape class for complex load patterns
from locust import LoadTestShape

class StagesShape(LoadTestShape):
    """Custom load shape matching k6 stages"""

    stages = [
        {"duration": 120, "users": 10, "spawn_rate": 1},
        {"duration": 420, "users": 10, "spawn_rate": 1},
        {"duration": 540, "users": 50, "spawn_rate": 2},
        {"duration": 840, "users": 50, "spawn_rate": 2},
        {"duration": 960, "users": 100, "spawn_rate": 5},
        {"duration": 1260, "users": 0, "spawn_rate": 5},
    ]

    def tick(self):
        run_time = self.get_run_time()

        for stage in self.stages:
            if run_time < stage["duration"]:
                return (stage["users"], stage["spawn_rate"])

        return None  # Stop test

JMeter Test Plan (XML configuration):

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="API Load Test">
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments">
        <collectionProp name="Arguments.arguments">
          <elementProp name="BASE_URL" elementType="Argument">
            <stringProp name="Argument.name">BASE_URL</stringProp>
            <stringProp name="Argument.value">https://api.example.com</stringProp>
          </elementProp>
          <elementProp name="THREADS" elementType="Argument">
            <stringProp name="Argument.name">THREADS</stringProp>
            <stringProp name="Argument.value">${__P(threads,10)}</stringProp>
          </elementProp>
        </collectionProp>
      </elementProp>
    </TestPlan>
    <hashTree>
      <!-- Thread Group with stepping pattern -->
      <com.blazemeter.jmeter.threads.concurrency.ConcurrencyThreadGroup guiclass="ConcurrencyThreadGroupGui" testclass="ConcurrencyThreadGroup" testname="Concurrent Users">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <stringProp name="TargetLevel">${THREADS}</stringProp>
        <stringProp name="RampUp">60</stringProp>
        <stringProp name="Steps">5</stringProp>
        <stringProp name="Hold">300</stringProp>
        <stringProp name="LogFilename"></stringProp>
        <stringProp name="Unit">S</stringProp>
      </com.blazemeter.jmeter.threads.concurrency.ConcurrencyThreadGroup>
      <hashTree>
        <!-- CSV Data Set for parameterisation -->
        <CSVDataSet guiclass="TestBeanGUI" testclass="CSVDataSet" testname="User Data">
          <stringProp name="filename">users.csv</stringProp>
          <stringProp name="fileEncoding">UTF-8</stringProp>
          <stringProp name="variableNames">userId,email,token</stringProp>
          <boolProp name="recycle">true</boolProp>
          <boolProp name="stopThread">false</boolProp>
          <stringProp name="shareMode">shareMode.all</stringProp>
        </CSVDataSet>
        <hashTree/>

        <!-- HTTP Request Defaults -->
        <ConfigTestElement guiclass="HttpDefaultsGui" testclass="ConfigTestElement" testname="HTTP Defaults">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">${BASE_URL}</stringProp>
          <stringProp name="HTTPSampler.protocol">https</stringProp>
          <stringProp name="HTTPSampler.contentEncoding">UTF-8</stringProp>
        </ConfigTestElement>
        <hashTree/>

        <!-- HTTP Header Manager -->
        <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Headers">
          <collectionProp name="HeaderManager.headers">
            <elementProp name="Authorization" elementType="Header">
              <stringProp name="Header.name">Authorization</stringProp>
              <stringProp name="Header.value">Bearer ${token}</stringProp>
            </elementProp>
            <elementProp name="Content-Type" elementType="Header">
              <stringProp name="Header.name">Content-Type</stringProp>
              <stringProp name="Header.value">application/json</stringProp>
            </elementProp>
          </collectionProp>
        </HeaderManager>
        <hashTree/>

        <!-- Transaction Controller for grouping -->
        <TransactionController guiclass="TransactionControllerGui" testclass="TransactionController" testname="Product Browsing">
          <boolProp name="TransactionController.includeTimers">false</boolProp>
        </TransactionController>
        <hashTree>
          <!-- HTTP Request -->
          <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="List Products">
            <stringProp name="HTTPSampler.path">/products?user=${userId}</stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
          </HTTPSamplerProxy>
          <hashTree>
            <!-- Response Assertions -->
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Status 200">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="49586">200</stringProp>
              </collectionProp>
              <stringProp name="Assertion.test_field">Assertion.response_code</stringProp>
              <intProp name="Assertion.test_type">8</intProp>
            </ResponseAssertion>
            <hashTree/>

            <!-- JSON Extractor for next request -->
            <JSONPostProcessor guiclass="JSONPostProcessorGui" testclass="JSONPostProcessor" testname="Extract Product IDs">
              <stringProp name="JSONPostProcessor.referenceNames">productIds</stringProp>
              <stringProp name="JSONPostProcessor.jsonPathExprs">$.products[*].id</stringProp>
              <stringProp name="JSONPostProcessor.match_numbers">-1</stringProp>
            </JSONPostProcessor>
            <hashTree/>
          </hashTree>

          <!-- Think Time -->
          <UniformRandomTimer guiclass="UniformRandomTimerGui" testclass="UniformRandomTimer" testname="Think Time">
            <stringProp name="ConstantTimer.delay">1000</stringProp>
            <stringProp name="RandomTimer.range">2000</stringProp>
          </UniformRandomTimer>
          <hashTree/>
        </hashTree>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Examples

k6 advanced scenarios:

// scenarios.js - Multiple concurrent scenarios
export const options = {
    scenarios: {
        // Constant VUs for baseline load
        smoke_test: {
            executor: 'constant-vus',
            vus: 10,
            duration: '5m',
            exec: 'smokeTest',
            tags: { test_type: 'smoke' },
        },
        // Ramping arrival rate (RPS-based)
        load_test: {
            executor: 'ramping-arrival-rate',
            startRate: 10,
            timeUnit: '1s',
            preAllocatedVUs: 50,
            maxVUs: 200,
            stages: [
                { duration: '5m', target: 50 },   // Ramp to 50 RPS
                { duration: '10m', target: 50 },  // Hold at 50 RPS
                { duration: '5m', target: 100 },  // Ramp to 100 RPS
                { duration: '10m', target: 0 },   // Ramp down
            ],
            exec: 'loadTest',
            startTime: '5m',  // Start after smoke test
        },
        // Shared iterations across VUs
        stress_test: {
            executor: 'shared-iterations',
            vus: 100,
            iterations: 10000,
            maxDuration: '30m',
            exec: 'stressTest',
            startTime: '35m',
        },
    },
};

export function smokeTest() {
    // Minimal validation
    http.get('https://api.example.com/health');
}

export function loadTest() {
    // Normal user flow
    const res = http.get('https://api.example.com/products');
    check(res, { 'status is 200': (r) => r.status === 200 });
}

export function stressTest() {
    // Push limits with batch requests
    const requests = [
        { method: 'GET', url: 'https://api.example.com/products' },
        { method: 'GET', url: 'https://api.example.com/users' },
        { method: 'GET', url: 'https://api.example.com/orders' },
    ];
    http.batch(requests);
}

Locust parameterisation with Faker:

# Advanced data generation
from locust import HttpUser, task, between
from faker import Faker
import random

fake = Faker()

class RealisticUser(HttpUser):
    wait_time = between(2, 5)

    def on_start(self):
        # Generate realistic user data
        self.user = {
            'email': fake.email(),
            'name': fake.name(),
            'address': fake.address(),
            'phone': fake.phone_number(),
        }

        # Register user
        self.client.post("/register", json=self.user)

    @task
    def create_order(self):
        """Create order with realistic data"""
        order = {
            'userId': self.user['email'],
            'items': [
                {
                    'productId': random.randint(1, 1000),
                    'quantity': random.randint(1, 5),
                    'price': round(random.uniform(10, 500), 2)
                }
                for _ in range(random.randint(1, 10))
            ],
            'shippingAddress': self.user['address'],
            'paymentMethod': random.choice(['credit_card', 'paypal', 'bank_transfer'])
        }

        self.client.post("/orders", json=order)

JMeter CLI for CI/CD:

#!/bin/bash
# run-jmeter.sh - JMeter CLI wrapper for CI/CD

# Run test with parameters
jmeter -n \
  -t load-test.jmx \
  -Jthreads=50 \
  -Jrampup=60 \
  -Jduration=300 \
  -JBASE_URL=https://staging.example.com \
  -l results.jtl \
  -e -o ./report

# Generate HTML report from existing results
jmeter -g results.jtl -o ./html-report

# Using JMeter plugins for better CLI reporting
# Install: https://jmeter-plugins.org/wiki/PluginsManager/
jmeter -n -t test.jmx \
  -l results.jtl \
  -JjmeterPlugin.sts.loadAndRunOnStartup=true \
  -JjmeterPlugin.sts.port=9191 \
  -JjmeterPlugin.sts.dataDir=./sts-data

Distributed Load Generation

Distributed testing spreads load generation across multiple machines to achieve higher throughput and avoid bottlenecks in load generators.

Key Concepts

graph TB
    subgraph "Distributed Architecture"
        A[Controller/Master] --> B[Worker 1]
        A --> C[Worker 2]
        A --> D[Worker N]

        B --> E[Target System]
        C --> E
        D --> E

        E --> F[Metrics Aggregator]
        F --> A
        A --> G[Results Dashboard]
    end

    subgraph "Cloud-based Distribution"
        H[Local Trigger] --> I[Cloud Provider]
        I --> J[Auto-scaled Workers]
        J --> K[Target]
        J --> L[Time Series DB]
        L --> M[Grafana]
    end

When to distribute:

Single machine CPU/network limits reached
Need >10,000 concurrent users
Testing from multiple geographical regions
Simulating distributed attack patterns

Challenges:

Clock synchronisation across workers
Network overhead for coordination
Results aggregation and deduplication
Coordinated ramp-up timing

Common Patterns

k6 distributed with k6 Cloud or custom orchestration:

// k6 can run distributed via k6 Cloud service
// Or orchestrate manually with k6-operator on Kubernetes

// test.js - same script runs on all nodes
export const options = {
    // Grafana Cloud k6 options (formerly ext.loadimpact)
    cloud: {
        stackID: 123456,    // Your Grafana Cloud stack ID (required for Grafana Cloud k6)
        projectID: 123456,  // Your k6 Cloud project ID
        distribution: {
            distributionLabel1: { loadZone: 'amazon:us:ashburn', percent: 50 },
            distributionLabel2: { loadZone: 'amazon:ie:dublin', percent: 25 },
            distributionLabel3: { loadZone: 'amazon:sg:singapore', percent: 25 },
        }
    }
};

// Run distributed with k6 cloud
// k6 cloud run test.js

// Or use k6-operator on Kubernetes

k6-operator Kubernetes manifest:

# k6-distributed.yaml - Run k6 test on Kubernetes
apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-distributed-test
  namespace: load-testing
spec:
  parallelism: 4  # Number of worker pods
  script:
    configMap:
      name: k6-test-script
      file: test.js
  arguments: --out statsd
  runner:
    image: grafana/k6:latest
    env:
      - name: K6_STATSD_ADDR
        value: "statsd-service:8125"
      - name: K6_STATSD_ENABLE_TAGS
        value: "true"
    resources:
      requests:
        memory: "512Mi"
        cpu: "500m"
      limits:
        memory: "1Gi"
        cpu: "1000m"

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: k6-test-script
  namespace: load-testing
data:
  test.js: |
    import http from 'k6/http';
    import { check } from 'k6';

    export const options = {
        stages: [
            { duration: '2m', target: 100 },
            { duration: '5m', target: 100 },
            { duration: '2m', target: 0 },
        ],
    };

    export default function() {
        const res = http.get('https://api.example.com/test');
        check(res, { 'status is 200': (r) => r.status === 200 });
    }

Locust distributed mode:

# Start Locust master node
locust -f locustfile.py \
  --master \
  --master-bind-host=0.0.0.0 \
  --master-bind-port=5557 \
  --expect-workers=4 \
  --web-host=0.0.0.0 \
  --web-port=8089

# Start Locust worker nodes (on separate machines)
locust -f locustfile.py \
  --worker \
  --master-host=192.168.1.10 \
  --master-port=5557

# Or use environment variables
export LOCUST_MODE=worker
export LOCUST_MASTER_NODE_HOST=master.example.com
export LOCUST_MASTER_NODE_PORT=5557
locust -f locustfile.py

# Headless mode for CI/CD
locust -f locustfile.py \
  --master \
  --expect-workers=4 \
  --headless \
  --users=1000 \
  --spawn-rate=100 \
  --run-time=10m \
  --html=report.html \
  --csv=results

Locust on Kubernetes:

# locust-distributed.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: locust-script
  namespace: load-testing
data:
  locustfile.py: |
    from locust import HttpUser, task, between

    class QuickstartUser(HttpUser):
        wait_time = between(1, 2)

        @task
        def hello_world(self):
            self.client.get("/")

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: locust-master
  namespace: load-testing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: locust-master
  template:
    metadata:
      labels:
        app: locust-master
    spec:
      containers:
      - name: locust
        image: locustio/locust:latest
        args:
          - "--master"
          - "--expect-workers=4"
          - "--web-host=0.0.0.0"
        ports:
        - containerPort: 8089
          name: web
        - containerPort: 5557
          name: master
        volumeMounts:
        - name: locust-script
          mountPath: /home/locust
      volumes:
      - name: locust-script
        configMap:
          name: locust-script

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: locust-worker
  namespace: load-testing
spec:
  replicas: 4
  selector:
    matchLabels:
      app: locust-worker
  template:
    metadata:
      labels:
        app: locust-worker
    spec:
      containers:
      - name: locust
        image: locustio/locust:latest
        args:
          - "--worker"
          - "--master-host=locust-master"
        volumeMounts:
        - name: locust-script
          mountPath: /home/locust
      volumes:
      - name: locust-script
        configMap:
          name: locust-script

---
apiVersion: v1
kind: Service
metadata:
  name: locust-master
  namespace: load-testing
spec:
  type: LoadBalancer
  selector:
    app: locust-master
  ports:
  - name: web
    port: 8089
    targetPort: 8089
  - name: master
    port: 5557
    targetPort: 5557

JMeter distributed mode:

# Configure JMeter distributed testing
# Edit jmeter.properties on controller:
# remote_hosts=worker1:1099,worker2:1099,worker3:1099
# server_port=1099
# server.rmi.ssl.disable=true

# Start JMeter server on worker nodes
jmeter-server \
  -Dserver.rmi.localport=1099 \
  -Dserver_port=1099 \
  -Jserver.rmi.ssl.disable=true

# Run test from controller to all workers
jmeter -n -t test.jmx \
  -R worker1,worker2,worker3 \
  -l results.jtl \
  -Gthreads=100 \  # Global property sent to all workers
  -X  # Exit workers when test completes

# Run on specific workers
jmeter -n -t test.jmx -r  # Uses remote_hosts from config

# Using Docker for distributed JMeter
docker run --rm -v $(pwd):/tests \
  justb4/jmeter:latest \
  -n -t /tests/test.jmx \
  -R worker1,worker2 \
  -l /tests/results.jtl

Examples

AWS deployment with Terraform:

# terraform/load-testing.tf
# Deploy distributed load testing infrastructure

resource "aws_instance" "locust_master" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  key_name      = var.key_name

  vpc_security_group_ids = [aws_security_group.locust.id]

  user_data = <<-EOF
              #!/bin/bash
              apt-get update
              # Install uv (house preference), then Locust system-wide
              curl -LsSf https://astral.sh/uv/install.sh | sh
              export PATH="$HOME/.local/bin:$PATH"
              uv pip install --system locust

              # Start master
              locust -f /opt/locustfile.py --master --web-host=0.0.0.0
              EOF

  tags = {
    Name = "locust-master"
    Role = "load-test-master"
  }
}

resource "aws_instance" "locust_worker" {
  count         = 4
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "c5.xlarge"  # CPU-optimised for load generation
  key_name      = var.key_name

  vpc_security_group_ids = [aws_security_group.locust.id]

  user_data = <<-EOF
              #!/bin/bash
              apt-get update
              # Install uv (house preference), then Locust system-wide
              curl -LsSf https://astral.sh/uv/install.sh | sh
              export PATH="$HOME/.local/bin:$PATH"
              uv pip install --system locust

              # Start worker
              locust -f /opt/locustfile.py \
                --worker \
                --master-host=${aws_instance.locust_master.private_ip}
              EOF

  tags = {
    Name = "locust-worker-${count.index}"
    Role = "load-test-worker"
  }
}

resource "aws_security_group" "locust" {
  name = "locust-sg"

  ingress {
    from_port   = 8089
    to_port     = 8089
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # Restrict in production
  }

  ingress {
    from_port   = 5557
    to_port     = 5558
    protocol    = "tcp"
    cidr_blocks = [var.vpc_cidr]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Custom k6 orchestration with Redis coordination:

// distributed-k6.js - Coordinated start across workers
import http from 'k6/http';
import { check } from 'k6';
import redis from 'k6/x/redis';

const redisClient = new redis.Client('redis://coordinator:6379');

export const options = {
    scenarios: {
        coordinated_load: {
            executor: 'ramping-vus',
            startVUs: 0,
            stages: [
                { duration: '5m', target: 100 },
            ],
        },
    },
};

export function setup() {
    // Coordinator waits for all workers to check in
    const workerId = `worker-${__ENV.HOSTNAME}`;
    redisClient.sadd('workers', workerId);

    // Wait for all workers (polling)
    while (redisClient.scard('workers') < __ENV.EXPECTED_WORKERS) {
        console.log('Waiting for all workers...');
        k6.sleep(1);
    }

    // Synchronise start time
    const startTime = Date.now() + 10000;  // Start in 10 seconds
    redisClient.set('start_time', startTime);

    return { startTime };
}

export default function(data) {
    // Wait until coordinated start time
    while (Date.now() < data.startTime) {
        k6.sleep(0.1);
    }

    // Execute test
    const res = http.get('https://api.example.com/test');
    check(res, { 'status is 200': (r) => r.status === 200 });
}

Metrics Capture and Dashboards

Collecting, storing, and visualising performance metrics enables real-time monitoring and historical analysis of load test results.

Key Concepts

flowchart TB
    subgraph "Metrics Pipeline"
        A[Load Generator] -->|StatsD/InfluxDB| B[Time Series DB]
        A -->|CSV/JSON| C[File Storage]
        A -->|HTTP| D[Backend API]

        B --> E[Grafana]
        C --> F[CI Report]
        D --> E

        E --> G[Real-time Dashboard]
        E --> H[Historical Analysis]
        F --> I[Test Summary]
    end

    subgraph "Key Metrics"
        J[Response Time] --> K[P50/P95/P99]
        L[Throughput] --> M[RPS]
        N[Errors] --> O[Error Rate %]
        P[Active VUs] --> Q[Concurrency]
    end

Standard metrics:

Response time: p50, p90, p95, p99, max
Throughput: Requests per second (RPS)
Error rate: Failed requests percentage
Active users: Current virtual users
Data transfer: Bytes sent/received

Output formats:

Real-time streaming (StatsD, InfluxDB, Kafka)
File-based (CSV, JSON, JTL)
Cloud platforms (k6 Cloud, BlazeMeter)
Custom webhooks and APIs

Common Patterns

k6 output to InfluxDB + Grafana:

Note: InfluxDB 1.x is legacy (on the EOL track). The examples below still work, but for new setups prefer InfluxDB v2/Cloud or Prometheus remote-write (k6 run -o experimental-prometheus-rw test.js).

// test.js with InfluxDB output
export const options = {
    stages: [
        { duration: '5m', target: 100 },
    ],
};

// Run with InfluxDB output
// k6 run --out influxdb=http://localhost:8086/k6 test.js

// Or configure in script
export const options = {
    // Grafana Cloud k6 options (formerly ext.loadimpact)
    cloud: {
        stackID: 123,       // Your Grafana Cloud stack ID (required for Grafana Cloud k6)
        projectID: 123,     // Your k6 Cloud project ID
        name: 'Performance Test'
    }
};

// Custom metrics for business KPIs
import { Trend, Counter, Rate, Gauge } from 'k6/metrics';

const checkoutTime = new Trend('checkout_time', true);  // Time trend
const checkoutErrors = new Counter('checkout_errors');
const checkoutSuccessRate = new Rate('checkout_success_rate');
const activeCheckouts = new Gauge('active_checkouts');

export default function() {
    activeCheckouts.add(1);

    const start = Date.now();
    const res = http.post('https://api.example.com/checkout', payload);
    const duration = Date.now() - start;

    checkoutTime.add(duration);

    const success = check(res, {
        'checkout succeeded': (r) => r.status === 201,
    });

    checkoutSuccessRate.add(success);
    if (!success) {
        checkoutErrors.add(1);
    }

    activeCheckouts.add(-1);
}

InfluxDB configuration:

# Start InfluxDB
docker run -d -p 8086:8086 \
  -v influxdb-data:/var/lib/influxdb \
  --name influxdb \
  influxdb:1.8

# Create database
curl -XPOST 'http://localhost:8086/query' \
  --data-urlencode "q=CREATE DATABASE k6"

# Run k6 with InfluxDB output
k6 run \
  --out influxdb=http://localhost:8086/k6 \
  --tag testid=run-001 \
  --tag environment=staging \
  test.js

Grafana dashboard for k6:

{
  "dashboard": {
    "title": "k6 Load Test Dashboard",
    "panels": [
      {
        "title": "Virtual Users",
        "targets": [
          {
            "measurement": "vus",
            "select": [[{"type": "field", "params": ["value"]}, {"type": "mean"}]],
            "groupBy": [{"type": "time", "params": ["$__interval"]}, {"type": "fill", "params": ["linear"]}]
          }
        ],
        "type": "graph"
      },
      {
        "title": "Response Time (95th Percentile)",
        "targets": [
          {
            "measurement": "http_req_duration",
            "select": [[{"type": "field", "params": ["value"]}, {"type": "percentile", "params": [95]}]],
            "groupBy": [{"type": "time", "params": ["$__interval"]}, {"type": "tag", "params": ["name"]}]
          }
        ],
        "type": "graph"
      },
      {
        "title": "Request Rate",
        "targets": [
          {
            "measurement": "http_reqs",
            "select": [[{"type": "field", "params": ["value"]}, {"type": "derivative", "params": ["1s"]}]],
            "groupBy": [{"type": "time", "params": ["$__interval"]}]
          }
        ],
        "type": "graph"
      },
      {
        "title": "Error Rate",
        "targets": [
          {
            "measurement": "http_req_failed",
            "select": [[{"type": "field", "params": ["value"]}, {"type": "mean"}]],
            "groupBy": [{"type": "time", "params": ["$__interval"]}]
          }
        ],
        "type": "graph"
      }
    ]
  }
}

Locust with Prometheus exporter:

# locustfile.py with Prometheus metrics
from locust import HttpUser, task, between, events
from prometheus_client import start_http_server, Counter, Histogram, Gauge
import time

# Prometheus metrics
requests_total = Counter(
    'locust_requests_total',
    'Total requests',
    ['method', 'endpoint', 'status']
)

request_duration = Histogram(
    'locust_request_duration_seconds',
    'Request duration',
    ['method', 'endpoint']
)

active_users = Gauge(
    'locust_users_active',
    'Currently active users'
)

# Start Prometheus metrics server
@events.init.add_listener
def on_locust_init(environment, **kwargs):
    if environment.web_ui:
        start_http_server(9090)  # Expose metrics on :9090/metrics

# Hook into request events
@events.request.add_listener
def on_request(request_type, name, response_time, response_length, exception, **kwargs):
    status = "failed" if exception else "success"
    requests_total.labels(
        method=request_type,
        endpoint=name,
        status=status
    ).inc()

    request_duration.labels(
        method=request_type,
        endpoint=name
    ).observe(response_time / 1000)  # Convert to seconds

@events.user_add.add_listener
def on_user_add(**kwargs):
    active_users.inc()

@events.user_remove.add_listener
def on_user_remove(**kwargs):
    active_users.dec()

class APIUser(HttpUser):
    wait_time = between(1, 3)

    @task
    def test_endpoint(self):
        self.client.get("/api/test")

JMeter Backend Listener for InfluxDB:

<!-- Add to JMeter test plan -->
<BackendListener guiclass="BackendListenerGui" testclass="BackendListener" testname="InfluxDB Backend">
  <elementProp name="arguments" elementType="Arguments">
    <collectionProp name="Arguments.arguments">
      <elementProp name="influxdbMetricsSender" elementType="Argument">
        <stringProp name="Argument.name">influxdbMetricsSender</stringProp>
        <stringProp name="Argument.value">org.apache.jmeter.visualizers.backend.influxdb.HttpMetricsSender</stringProp>
      </elementProp>
      <elementProp name="influxdbUrl" elementType="Argument">
        <stringProp name="Argument.name">influxdbUrl</stringProp>
        <stringProp name="Argument.value">http://localhost:8086/write?db=jmeter</stringProp>
      </elementProp>
      <elementProp name="application" elementType="Argument">
        <stringProp name="Argument.name">application</stringProp>
        <stringProp name="Argument.value">my-app</stringProp>
      </elementProp>
      <elementProp name="measurement" elementType="Argument">
        <stringProp name="Argument.name">measurement</stringProp>
        <stringProp name="Argument.value">jmeter</stringProp>
      </elementProp>
      <elementProp name="summaryOnly" elementType="Argument">
        <stringProp name="Argument.name">summaryOnly</stringProp>
        <stringProp name="Argument.value">false</stringProp>
      </elementProp>
      <elementProp name="samplersRegex" elementType="Argument">
        <stringProp name="Argument.name">samplersRegex</stringProp>
        <stringProp name="Argument.value">.*</stringProp>
      </elementProp>
      <elementProp name="percentiles" elementType="Argument">
        <stringProp name="Argument.name">percentiles</stringProp>
        <stringProp name="Argument.value">50;90;95;99</stringProp>
      </elementProp>
      <elementProp name="testTitle" elementType="Argument">
        <stringProp name="Argument.name">testTitle</stringProp>
        <stringProp name="Argument.value">Load Test</stringProp>
      </elementProp>
    </collectionProp>
  </elementProp>
  <stringProp name="classname">org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient</stringProp>
</BackendListener>

Examples

Complete monitoring stack with Docker Compose:

# docker-compose.yml - Full monitoring stack
version: '3.8'

services:
  influxdb:
    image: influxdb:1.8  # Legacy/EOL-track; prefer InfluxDB v2/Cloud or Prometheus remote-write for new setups
    container_name: influxdb
    ports:
      - "8086:8086"
    environment:
      - INFLUXDB_DB=k6
      - INFLUXDB_HTTP_AUTH_ENABLED=false
    volumes:
      - influxdb-data:/var/lib/influxdb
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_INSTALL_PLUGINS=grafana-clock-panel
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/datasources:/etc/grafana/provisioning/datasources
    depends_on:
      - influxdb
    networks:
      - monitoring

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9091:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    networks:
      - monitoring

  k6:
    image: grafana/k6:latest
    container_name: k6
    volumes:
      - ./tests:/scripts
    environment:
      - K6_OUT=influxdb=http://influxdb:8086/k6
    command: run /scripts/test.js
    depends_on:
      - influxdb
    networks:
      - monitoring

volumes:
  influxdb-data:
  grafana-data:
  prometheus-data:

networks:
  monitoring:
    driver: bridge

Grafana datasource provisioning:

# grafana/datasources/influxdb.yml
apiVersion: 1

datasources:
  - name: InfluxDB-k6
    type: influxdb
    access: proxy
    url: http://influxdb:8086
    database: k6
    isDefault: true
    editable: true

  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: false
    editable: true

Custom HTML report generation:

# generate_report.py - Create custom HTML report from Locust CSV
import pandas as pd
import matplotlib.pyplot as plt
from jinja2 import Template

# Load results
stats = pd.read_csv('results_stats.csv')
failures = pd.read_csv('results_failures.csv')

# Calculate summary metrics
total_requests = stats['Request Count'].sum()
total_failures = stats['Failure Count'].sum()
error_rate = (total_failures / total_requests * 100) if total_requests > 0 else 0
avg_response_time = stats['Average Response Time'].mean()
p95_response_time = stats['95%'].mean()

# Generate charts
fig, axes = plt.subplots(2, 2, figsize=(15, 10))

# Response time distribution
axes[0, 0].hist(stats['Average Response Time'], bins=20)
axes[0, 0].set_title('Response Time Distribution')
axes[0, 0].set_xlabel('Response Time (ms)')

# RPS over time
axes[0, 1].plot(stats['Requests/s'])
axes[0, 1].set_title('Throughput Over Time')
axes[0, 1].set_ylabel('Requests/s')

# Error rate
axes[1, 0].bar(['Success', 'Failure'], [total_requests - total_failures, total_failures])
axes[1, 0].set_title('Request Success/Failure')

# Percentiles
percentiles = ['50%', '66%', '75%', '80%', '90%', '95%', '98%', '99%']
values = [stats[p].mean() for p in percentiles]
axes[1, 1].plot(percentiles, values, marker='o')
axes[1, 1].set_title('Response Time Percentiles')
axes[1, 1].set_ylabel('Response Time (ms)')

plt.tight_layout()
plt.savefig('report_charts.png', dpi=150)

# HTML template
html_template = """
<!DOCTYPE html>
<html>
<head>
    <title>Load Test Report</title>
    <style>
        body { font-family: Arial, sans-serif; margin: 20px; }
        .metric { display: inline-block; margin: 20px; padding: 20px; background: #f0f0f0; border-radius: 5px; }
        .metric-value { font-size: 36px; font-weight: bold; color: #333; }
        .metric-label { font-size: 14px; color: #666; }
        table { border-collapse: collapse; width: 100%; margin: 20px 0; }
        th, td { border: 1px solid #ddd; padding: 12px; text-align: left; }
        th { background-color: #4CAF50; color: white; }
        tr:nth-child(even) { background-color: #f2f2f2; }
        .pass { color: green; } .fail { color: red; }
    </style>
</head>
<body>
    <h1>Load Test Report</h1>
    <div>
        <div class="metric">
            <div class="metric-value">{{ total_requests }}</div>
            <div class="metric-label">Total Requests</div>
        </div>
        <div class="metric">
            <div class="metric-value {{ 'pass' if error_rate < 1 else 'fail' }}">{{ "%.2f"|format(error_rate) }}%</div>
            <div class="metric-label">Error Rate</div>
        </div>
        <div class="metric">
            <div class="metric-value">{{ "%.0f"|format(avg_response_time) }}ms</div>
            <div class="metric-label">Avg Response Time</div>
        </div>
        <div class="metric">
            <div class="metric-value">{{ "%.0f"|format(p95_response_time) }}ms</div>
            <div class="metric-label">P95 Response Time</div>
        </div>
    </div>

    <h2>Statistics by Endpoint</h2>
    <table>
        <tr>
            <th>Endpoint</th>
            <th>Requests</th>
            <th>Failures</th>
            <th>Avg (ms)</th>
            <th>P95 (ms)</th>
            <th>RPS</th>
        </tr>
        {% for row in stats_rows %}
        <tr>
            <td>{{ row['Name'] }}</td>
            <td>{{ row['Request Count'] }}</td>
            <td>{{ row['Failure Count'] }}</td>
            <td>{{ "%.0f"|format(row['Average Response Time']) }}</td>
            <td>{{ "%.0f"|format(row['95%']) }}</td>
            <td>{{ "%.1f"|format(row['Requests/s']) }}</td>
        </tr>
        {% endfor %}
    </table>

    <h2>Charts</h2>
    <img src="report_charts.png" style="max-width: 100%;">
</body>
</html>
"""

# Render report
template = Template(html_template)
html = template.render(
    total_requests=total_requests,
    error_rate=error_rate,
    avg_response_time=avg_response_time,
    p95_response_time=p95_response_time,
    stats_rows=stats.to_dict('records')
)

with open('report.html', 'w') as f:
    f.write(html)

print("Report generated: report.html")

CI Integration for Performance Gates

Integrating load tests into CI/CD pipelines enables automated performance regression detection and enforcement of performance SLOs.

Key Concepts

flowchart LR
    A[Code Commit] --> B[CI Pipeline]
    B --> C[Build & Unit Tests]
    C --> D[Deploy to Test Env]
    D --> E[Run Load Test]
    E --> F{Performance Gate}
    F -->|Pass| G[Deploy to Staging]
    F -->|Fail| H[Block Deployment]
    H --> I[Alert Team]
    E --> J[Store Metrics]
    J --> K[Trend Analysis]
    K --> L[Performance Dashboard]

Performance gate criteria:

Response time thresholds (p95 < 500ms)
Error rate limits (< 1%)
Throughput minimums (> 1000 RPS)
Resource utilisation (CPU < 80%)
Comparison to baseline (regression > 10%)

CI/CD integration patterns:

Pre-deployment: Run before production deploy
Synthetic monitoring: Continuous background load
Scheduled: Nightly comprehensive tests
On-demand: Manual trigger for capacity planning

Common Patterns

GitHub Actions with k6:

# .github/workflows/load-test.yml
name: Load Test

on:
  pull_request:
    branches: [ main ]
  push:
    branches: [ main ]
  schedule:
    - cron: '0 2 * * *'  # Nightly at 2 AM
  workflow_dispatch:  # Manual trigger

env:
  K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }}

jobs:
  load-test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v6

      # Official Grafana action installs k6 (replaces the manual gpg/apt block)
      - name: Setup k6
        uses: grafana/setup-k6-action@v1

      - name: Run k6 test
        uses: grafana/run-k6-action@v1
        with:
          path: tests/load-test.js
          flags: --out json=results.json --summary-export=summary.json
        env:
          TEST_ENV: staging
          BASE_URL: https://staging.example.com

      - name: Check thresholds
        run: |
          # Parse summary for threshold failures
          failures=$(jq -r '.metrics | to_entries[] | select(.value.thresholds | length > 0) | .value.thresholds | to_entries[] | select(.value.ok == false) | .key' summary.json)

          if [ -n "$failures" ]; then
            echo "❌ Performance thresholds failed:"
            echo "$failures"
            exit 1
          else
            echo "✅ All performance thresholds passed"
          fi

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: k6-results
          path: |
            results.json
            summary.json

      - name: Compare with baseline
        run: |
          # Download previous baseline
          wget -O baseline.json https://example.com/baselines/latest.json || echo "{}" > baseline.json

          # Compare metrics
          python3 scripts/compare-performance.py baseline.json summary.json

      - name: Send metrics to InfluxDB
        if: always()
        run: |
          # Send results for historical tracking
          curl -XPOST 'http://influxdb.example.com:8086/write?db=k6' \
            --data-binary @results.influx

      - name: Comment PR with results
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v9
        with:
          script: |
            const fs = require('fs');
            const summary = JSON.parse(fs.readFileSync('summary.json'));

            const p95 = summary.metrics.http_req_duration.values['p(95)'];
            const errorRate = summary.metrics.http_req_failed.values.rate * 100;
            const rps = summary.metrics.http_reqs.values.rate;

            const comment = `## 📊 Load Test Results

            | Metric | Value | Threshold | Status |
            |--------|-------|-----------|--------|
            | P95 Latency | ${p95.toFixed(0)}ms | < 500ms | ${p95 < 500 ? '✅' : '❌'} |
            | Error Rate | ${errorRate.toFixed(2)}% | < 1% | ${errorRate < 1 ? '✅' : '❌'} |
            | Throughput | ${rps.toFixed(0)} RPS | > 100 RPS | ${rps > 100 ? '✅' : '❌'} |
            `;

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: comment
            });

GitLab CI with Locust:

# .gitlab-ci.yml
stages:
  - build
  - test
  - load-test
  - deploy

variables:
  TARGET_URL: "https://staging.example.com"
  LOCUST_USERS: "100"
  LOCUST_SPAWN_RATE: "10"
  LOCUST_RUN_TIME: "5m"

load_test:
  stage: load-test
  image: locustio/locust:latest

  services:
    - docker:dind

  before_script:
    # uv isn't bundled in the locust image, so install it first (house preference)
    - pip install uv
    - uv pip install --system locust-plugins  # For extra functionality

  script:
    # Run Locust in headless mode
    - |
      locust \
        -f tests/locustfile.py \
        --headless \
        --users ${LOCUST_USERS} \
        --spawn-rate ${LOCUST_SPAWN_RATE} \
        --run-time ${LOCUST_RUN_TIME} \
        --host ${TARGET_URL} \
        --html report.html \
        --csv results \
        --exit-code-on-error 1

    # Check performance thresholds
    - python3 scripts/check_thresholds.py results_stats.csv

  after_script:
    # Send metrics to monitoring system
    - curl -X POST https://metrics.example.com/api/v1/load-test \
        -H "Content-Type: application/json" \
        -d @results_stats.csv

  artifacts:
    when: always
    paths:
      - report.html
      - results_stats.csv
      - results_failures.csv
    reports:
      junit: results.xml
    expire_in: 30 days

  only:
    - merge_requests
    - main

  allow_failure: false  # Block pipeline if test fails

# Nightly comprehensive test
load_test_nightly:
  extends: load_test
  variables:
    LOCUST_USERS: "1000"
    LOCUST_RUN_TIME: "30m"
  only:
    - schedules
  allow_failure: true  # Don't block, just alert

Jenkins with JMeter:

// Jenkinsfile
pipeline {
    agent any

    parameters {
        string(name: 'THREADS', defaultValue: '100', description: 'Number of threads')
        string(name: 'DURATION', defaultValue: '300', description: 'Test duration in seconds')
        string(name: 'RAMPUP', defaultValue: '60', description: 'Ramp-up period in seconds')
        choice(name: 'ENVIRONMENT', choices: ['staging', 'production'], description: 'Target environment')
    }

    environment {
        JMETER_HOME = '/opt/apache-jmeter-5.5'
        PATH = "${JMETER_HOME}/bin:${env.PATH}"

        // Performance thresholds
        MAX_P95_LATENCY = 500
        MAX_ERROR_RATE = 1.0
        MIN_THROUGHPUT = 100
    }

    stages {
        stage('Prepare') {
            steps {
                script {
                    // Set environment-specific variables
                    if (params.ENVIRONMENT == 'production') {
                        env.TARGET_URL = 'https://api.example.com'
                    } else {
                        env.TARGET_URL = 'https://staging.example.com'
                    }
                }

                // Clean workspace
                sh 'rm -rf results reports'
                sh 'mkdir -p results reports'
            }
        }

        stage('Run Load Test') {
            steps {
                script {
                    // Run JMeter test
                    sh """
                        jmeter -n \
                          -t tests/load-test.jmx \
                          -Jthreads=${params.THREADS} \
                          -Jduration=${params.DURATION} \
                          -Jrampup=${params.RAMPUP} \
                          -JBASE_URL=${env.TARGET_URL} \
                          -l results/results.jtl \
                          -j results/jmeter.log \
                          -e -o reports/html
                    """
                }
            }
        }

        stage('Generate Reports') {
            steps {
                // Generate HTML report
                sh """
                    jmeter -g results/results.jtl \
                      -o reports/dashboard
                """

                // Parse results for metrics
                sh """
                    python3 scripts/parse_jtl.py \
                      results/results.jtl \
                      > results/metrics.json
                """
            }
        }

        stage('Check Thresholds') {
            steps {
                script {
                    def metrics = readJSON file: 'results/metrics.json'

                    def p95 = metrics.percentiles.p95
                    def errorRate = metrics.error_rate
                    def throughput = metrics.throughput

                    echo "P95 Latency: ${p95}ms (threshold: ${env.MAX_P95_LATENCY}ms)"
                    echo "Error Rate: ${errorRate}% (threshold: ${env.MAX_ERROR_RATE}%)"
                    echo "Throughput: ${throughput} RPS (threshold: ${env.MIN_THROUGHPUT} RPS)"

                    def failures = []

                    if (p95 > env.MAX_P95_LATENCY.toInteger()) {
                        failures << "P95 latency exceeded threshold"
                    }

                    if (errorRate > env.MAX_ERROR_RATE.toDouble()) {
                        failures << "Error rate exceeded threshold"
                    }

                    if (throughput < env.MIN_THROUGHPUT.toInteger()) {
                        failures << "Throughput below threshold"
                    }

                    if (failures) {
                        error("Performance thresholds failed:\n" + failures.join('\n'))
                    }
                }
            }
        }

        stage('Compare with Baseline') {
            steps {
                script {
                    // Compare with previous results
                    sh """
                        python3 scripts/compare_baseline.py \
                          --current results/metrics.json \
                          --baseline /var/lib/jenkins/baselines/${params.ENVIRONMENT}.json \
                          --threshold 10
                    """
                }
            }
        }

        stage('Publish Results') {
            steps {
                // Publish HTML reports
                publishHTML([
                    reportDir: 'reports/html',
                    reportFiles: 'index.html',
                    reportName: 'JMeter Report',
                    keepAll: true
                ])

                // Archive artifacts
                archiveArtifacts artifacts: 'results/*, reports/**/*', fingerprint: true

                // Send to InfluxDB
                sh """
                    python3 scripts/send_to_influxdb.py \
                      --file results/results.jtl \
                      --url http://influxdb:8086 \
                      --database jmeter \
                      --tags environment=${params.ENVIRONMENT},build=${env.BUILD_NUMBER}
                """
            }
        }
    }

    post {
        always {
            // Send notification
            script {
                def status = currentBuild.result ?: 'SUCCESS'
                def color = status == 'SUCCESS' ? 'good' : 'danger'

                slackSend(
                    channel: '#performance-tests',
                    color: color,
                    message: """
                        Load Test ${status}
                        Environment: ${params.ENVIRONMENT}
                        Users: ${params.THREADS}
                        Duration: ${params.DURATION}s
                        Build: ${env.BUILD_URL}
                    """
                )
            }
        }

        failure {
            // Alert on failure
            emailext(
                to: 'performance-team@example.com',
                subject: "Load Test Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                body: """
                    The load test has failed threshold checks.

                    View results: ${env.BUILD_URL}

                    Check the attached report for details.
                """,
                attachLog: true
            )
        }

        success {
            // Update baseline on success
            sh """
                cp results/metrics.json \
                  /var/lib/jenkins/baselines/${params.ENVIRONMENT}.json
            """
        }
    }
}

Examples

Threshold checking script:

# check_thresholds.py - Validate performance metrics
import sys
import json
import csv
from typing import Dict, List, Tuple

class ThresholdChecker:
    """Check performance test results against defined thresholds"""

    def __init__(self, thresholds: Dict):
        self.thresholds = thresholds
        self.failures = []

    def check_metric(self, name: str, value: float, threshold: float, operator: str = 'lt') -> bool:
        """Check a single metric against threshold"""
        operators = {
            'lt': lambda v, t: v < t,
            'gt': lambda v, t: v > t,
            'lte': lambda v, t: v <= t,
            'gte': lambda v, t: v >= t,
        }

        passed = operators[operator](value, threshold)

        if not passed:
            self.failures.append({
                'metric': name,
                'value': value,
                'threshold': threshold,
                'operator': operator
            })

        return passed

    def check_locust_results(self, stats_file: str) -> bool:
        """Check Locust CSV results"""
        with open(stats_file, 'r') as f:
            reader = csv.DictReader(f)
            rows = list(reader)

        # Aggregate row has Type='Aggregated'
        agg_row = next((r for r in rows if r['Type'] == 'Aggregated'), None)

        if not agg_row:
            print("❌ No aggregated results found")
            return False

        # Check thresholds
        avg_response = float(agg_row['Average Response Time'])
        p95_response = float(agg_row['95%'])
        p99_response = float(agg_row['99%'])

        failure_count = int(agg_row['Failure Count'])
        request_count = int(agg_row['Request Count'])
        error_rate = (failure_count / request_count * 100) if request_count > 0 else 0

        rps = float(agg_row['Requests/s'])

        # Apply threshold checks
        all_passed = True

        if 'avg_response_time' in self.thresholds:
            all_passed &= self.check_metric(
                'Average Response Time',
                avg_response,
                self.thresholds['avg_response_time'],
                'lt'
            )

        if 'p95_response_time' in self.thresholds:
            all_passed &= self.check_metric(
                'P95 Response Time',
                p95_response,
                self.thresholds['p95_response_time'],
                'lt'
            )

        if 'p99_response_time' in self.thresholds:
            all_passed &= self.check_metric(
                'P99 Response Time',
                p99_response,
                self.thresholds['p99_response_time'],
                'lt'
            )

        if 'error_rate' in self.thresholds:
            all_passed &= self.check_metric(
                'Error Rate',
                error_rate,
                self.thresholds['error_rate'],
                'lt'
            )

        if 'min_throughput' in self.thresholds:
            all_passed &= self.check_metric(
                'Throughput (RPS)',
                rps,
                self.thresholds['min_throughput'],
                'gt'
            )

        return all_passed

    def print_report(self) -> None:
        """Print threshold check report"""
        if not self.failures:
            print("✅ All thresholds passed!")
            return

        print("❌ Threshold failures detected:\n")
        print(f"{'Metric':<30} {'Value':<15} {'Threshold':<15} {'Status':<10}")
        print("-" * 70)

        for failure in self.failures:
            print(f"{failure['metric']:<30} "
                  f"{failure['value']:<15.2f} "
                  f"{failure['threshold']:<15.2f} "
                  f"{'FAIL':<10}")

if __name__ == '__main__':
    # Define thresholds
    thresholds = {
        'avg_response_time': 300,    # ms
        'p95_response_time': 500,    # ms
        'p99_response_time': 1000,   # ms
        'error_rate': 1.0,           # percent
        'min_throughput': 100,       # RPS
    }

    if len(sys.argv) < 2:
        print("Usage: python check_thresholds.py <stats_csv_file>")
        sys.exit(1)

    checker = ThresholdChecker(thresholds)
    passed = checker.check_locust_results(sys.argv[1])

    checker.print_report()

    sys.exit(0 if passed else 1)

Baseline comparison script:

# compare_baseline.py - Detect performance regressions
import json
import sys
import argparse
from typing import Dict

def load_metrics(file_path: str) -> Dict:
    """Load metrics from JSON file"""
    with open(file_path, 'r') as f:
        return json.load(f)

def compare_metrics(current: Dict, baseline: Dict, threshold_pct: float = 10) -> bool:
    """Compare current metrics against baseline"""

    regressions = []

    # Metrics to compare (lower is better)
    lower_better = ['avg_response_time', 'p95_response_time', 'p99_response_time', 'error_rate']

    # Metrics to compare (higher is better)
    higher_better = ['throughput']

    for metric in lower_better:
        if metric in current and metric in baseline:
            current_val = current[metric]
            baseline_val = baseline[metric]

            increase_pct = ((current_val - baseline_val) / baseline_val * 100) if baseline_val > 0 else 0

            if increase_pct > threshold_pct:
                regressions.append({
                    'metric': metric,
                    'current': current_val,
                    'baseline': baseline_val,
                    'change_pct': increase_pct,
                    'direction': 'increased'
                })

    for metric in higher_better:
        if metric in current and metric in baseline:
            current_val = current[metric]
            baseline_val = baseline[metric]

            decrease_pct = ((baseline_val - current_val) / baseline_val * 100) if baseline_val > 0 else 0

            if decrease_pct > threshold_pct:
                regressions.append({
                    'metric': metric,
                    'current': current_val,
                    'baseline': baseline_val,
                    'change_pct': -decrease_pct,
                    'direction': 'decreased'
                })

    # Print report
    if not regressions:
        print(f"✅ No performance regressions detected (threshold: {threshold_pct}%)")
        return True

    print(f"❌ Performance regressions detected:\n")
    print(f"{'Metric':<25} {'Current':<15} {'Baseline':<15} {'Change':<15}")
    print("-" * 70)

    for reg in regressions:
        change_str = f"{reg['change_pct']:+.2f}%"
        print(f"{reg['metric']:<25} "
              f"{reg['current']:<15.2f} "
              f"{reg['baseline']:<15.2f} "
              f"{change_str:<15}")

    return False

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Compare performance metrics against baseline')
    parser.add_argument('--current', required=True, help='Current metrics JSON file')
    parser.add_argument('--baseline', required=True, help='Baseline metrics JSON file')
    parser.add_argument('--threshold', type=float, default=10.0, help='Regression threshold percentage')

    args = parser.parse_args()

    try:
        current = load_metrics(args.current)
        baseline = load_metrics(args.baseline)
    except FileNotFoundError as e:
        print(f"❌ Error: {e}")
        sys.exit(1)

    passed = compare_metrics(current, baseline, args.threshold)

    sys.exit(0 if passed else 1)

Interpreting Bottlenecks and Saturation

Analysing load test results to identify system constraints, resource exhaustion, and performance degradation patterns.

Key Concepts

graph TB
    subgraph "Performance Degradation Patterns"
        A[Linear Scaling] -->|Good| B[Optimal Performance]
        C[Gradual Degradation] -->|Warning| D[Resource Contention]
        E[Cliff Effect] -->|Critical| F[Hard Limit Reached]
        G[Oscillation] -->|Problem| H[Thrashing/GC Issues]
    end

    subgraph "Common Bottlenecks"
        I[CPU Saturation] --> J[Thread Pool Exhaustion]
        K[Memory Pressure] --> L[GC Overhead]
        M[I/O Wait] --> N[Disk/Network Limits]
        O[Database Connections] --> P[Connection Pool Exhaustion]
        Q[External API] --> R[Downstream Service Limits]
    end

Bottleneck indicators:

Response time increases while throughput plateaus
Error rate spike at specific load level
Resource utilisation (CPU/memory) at 100%
Queue depth growing unbounded
Connection pool exhaustion
Garbage collection pauses

Analysis techniques:

Correlation of response time with concurrency
Resource utilisation vs throughput graphs
Percentile analysis (p50 vs p99 divergence)
Error pattern analysis
Apdex score degradation

Common Patterns

Response time percentile analysis:

# analyse_percentiles.py - Identify latency outliers
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

def analyse_percentile_divergence(results_csv: str):
    """Analyse percentile divergence to identify issues"""

    df = pd.read_csv(results_csv)

    # Calculate percentiles over time windows
    df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')
    df = df.set_index('timestamp')

    # Resample to 10-second windows
    windows = df.resample('10S')['response_time']

    percentiles = windows.quantile([0.5, 0.75, 0.9, 0.95, 0.99])
    percentiles = percentiles.unstack()

    # Calculate divergence ratio (p99/p50)
    percentiles['divergence'] = percentiles[0.99] / percentiles[0.5]

    # Identify problematic windows (divergence > 3x)
    high_divergence = percentiles[percentiles['divergence'] > 3]

    # Plot
    fig, axes = plt.subplots(2, 1, figsize=(15, 10))

    # Percentiles over time
    axes[0].plot(percentiles.index, percentiles[0.5], label='P50', linewidth=2)
    axes[0].plot(percentiles.index, percentiles[0.95], label='P95', linewidth=2)
    axes[0].plot(percentiles.index, percentiles[0.99], label='P99', linewidth=2)
    axes[0].set_ylabel('Response Time (ms)')
    axes[0].set_title('Response Time Percentiles Over Time')
    axes[0].legend()
    axes[0].grid(True)

    # Divergence ratio
    axes[1].plot(percentiles.index, percentiles['divergence'], color='red', linewidth=2)
    axes[1].axhline(y=3, color='orange', linestyle='--', label='Threshold (3x)')
    axes[1].set_ylabel('P99/P50 Ratio')
    axes[1].set_title('Percentile Divergence (P99/P50)')
    axes[1].legend()
    axes[1].grid(True)

    plt.tight_layout()
    plt.savefig('percentile_analysis.png', dpi=150)

    print(f"Analysis complete. High divergence windows: {len(high_divergence)}")

    if not high_divergence.empty:
        print("\nPeriods with high latency variance (P99/P50 > 3x):")
        print(high_divergence[['divergence', 0.5, 0.99]])

Correlation analysis:

# correlate_metrics.py - Find bottleneck correlations
import pandas as pd
import numpy as np
from scipy.stats import pearsonr

def correlate_performance_metrics(
    load_test_csv: str,
    system_metrics_csv: str
):
    """Correlate load test metrics with system resource usage"""

    # Load data
    load_df = pd.read_csv(load_test_csv, parse_dates=['timestamp'])
    system_df = pd.read_csv(system_metrics_csv, parse_dates=['timestamp'])

    # Merge on timestamp (align to nearest second)
    load_df['ts_rounded'] = load_df['timestamp'].dt.floor('S')
    system_df['ts_rounded'] = system_df['timestamp'].dt.floor('S')

    merged = pd.merge(load_df, system_df, on='ts_rounded', how='inner')

    # Calculate correlations
    correlations = {
        'Response Time vs CPU': pearsonr(merged['response_time'], merged['cpu_percent']),
        'Response Time vs Memory': pearsonr(merged['response_time'], merged['memory_percent']),
        'Response Time vs I/O Wait': pearsonr(merged['response_time'], merged['io_wait']),
        'Error Rate vs CPU': pearsonr(merged['error_rate'], merged['cpu_percent']),
        'Throughput vs CPU': pearsonr(merged['throughput'], merged['cpu_percent']),
    }

    print("Correlation Analysis:")
    print("-" * 60)
    for metric_pair, (corr, pvalue) in correlations.items():
        significance = "***" if pvalue < 0.001 else "**" if pvalue < 0.01 else "*" if pvalue < 0.05 else ""
        print(f"{metric_pair:<40} r={corr:>6.3f} {significance}")

    # Identify strongest correlations
    strong_correlations = {k: v for k, v in correlations.items() if abs(v[0]) > 0.7}

    if strong_correlations:
        print("\nStrong Correlations (|r| > 0.7):")
        for metric, (corr, _) in strong_correlations.items():
            print(f"  • {metric}: {corr:.3f}")

            if 'CPU' in metric and corr > 0.7:
                print("    → Likely CPU-bound bottleneck")
            elif 'Memory' in metric and corr > 0.7:
                print("    → Likely memory pressure or GC issues")
            elif 'I/O Wait' in metric and corr > 0.7:
                print("    → Likely I/O bottleneck (disk or network)")

    return correlations

Saturation point detection:

# find_saturation.py - Identify system saturation point
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

def find_saturation_point(results_csv: str):
    """
    Find the saturation point where throughput stops scaling linearly
    with increased load
    """

    df = pd.read_csv(results_csv)

    # Group by load level (concurrent users)
    grouped = df.groupby('concurrent_users').agg({
        'throughput': 'mean',
        'response_time': 'mean',
        'error_rate': 'mean'
    }).reset_index()

    X = grouped['concurrent_users'].values.reshape(-1, 1)
    y_throughput = grouped['throughput'].values
    y_response_time = grouped['response_time'].values

    # Test different breakpoints to find saturation
    best_r2 = 0
    saturation_point = None

    for i in range(3, len(X) - 3):
        # Fit linear model to early points
        model = LinearRegression()
        model.fit(X[:i], y_throughput[:i])

        # Predict for all points
        y_pred = model.predict(X)

        # Calculate R² for points after breakpoint
        r2 = r2_score(y_throughput[i:], y_pred[i:])

        # If R² drops significantly, we found saturation
        if r2 < 0.5 and best_r2 == 0:
            saturation_point = grouped.iloc[i]['concurrent_users']
            best_r2 = r2
            break

    if saturation_point:
        print(f"🔴 Saturation detected at {saturation_point} concurrent users")

        saturated_row = grouped[grouped['concurrent_users'] == saturation_point].iloc[0]
        print(f"   Throughput: {saturated_row['throughput']:.2f} RPS")
        print(f"   Response Time: {saturated_row['response_time']:.2f} ms")
        print(f"   Error Rate: {saturated_row['error_rate']:.2f}%")
    else:
        print("✅ No saturation detected within test range")

    # Plot throughput vs load
    import matplotlib.pyplot as plt

    fig, axes = plt.subplots(1, 2, figsize=(15, 5))

    # Throughput
    axes[0].plot(grouped['concurrent_users'], grouped['throughput'], marker='o', linewidth=2)
    if saturation_point:
        axes[0].axvline(x=saturation_point, color='red', linestyle='--', label=f'Saturation: {saturation_point} users')
    axes[0].set_xlabel('Concurrent Users')
    axes[0].set_ylabel('Throughput (RPS)')
    axes[0].set_title('Throughput vs Load')
    axes[0].grid(True)
    axes[0].legend()

    # Response time
    axes[1].plot(grouped['concurrent_users'], grouped['response_time'], marker='o', color='orange', linewidth=2)
    if saturation_point:
        axes[1].axvline(x=saturation_point, color='red', linestyle='--', label=f'Saturation: {saturation_point} users')
    axes[1].set_xlabel('Concurrent Users')
    axes[1].set_ylabel('Response Time (ms)')
    axes[1].set_title('Response Time vs Load')
    axes[1].grid(True)
    axes[1].legend()

    plt.tight_layout()
    plt.savefig('saturation_analysis.png', dpi=150)

    return saturation_point

Examples

Bottleneck patterns and diagnosis:

# Common bottleneck patterns and how to identify them

patterns:
  cpu_bound:
    symptoms:
      - High CPU utilisation (>90%)
      - Response time increases linearly with load
      - Throughput plateaus
      - No increase in error rate
    diagnosis:
      - Profile application for hot code paths
      - Check for inefficient algorithms (O(n²))
      - Look for CPU-intensive operations (regex, crypto, compression)
    solutions:
      - Optimise algorithms
      - Add caching
      - Horizontal scaling
      - Use async/non-blocking operations

  memory_pressure:
    symptoms:
      - High memory usage (>80%)
      - Frequent garbage collection
      - Response time spikes
      - OutOfMemory errors
    diagnosis:
      - Heap dump analysis
      - GC logs
      - Memory leak detection
    solutions:
      - Increase heap size
      - Fix memory leaks
      - Optimise object allocation
      - Implement object pooling

  database_bottleneck:
    symptoms:
      - High database query times
      - Database CPU/IO at capacity
      - Connection pool exhaustion
      - Lock wait timeouts
    diagnosis:
      - Slow query logs
      - Database profiling
      - Connection pool metrics
      - Lock monitoring
    solutions:
      - Add database indexes
      - Optimise queries
      - Increase connection pool
      - Read replicas
      - Query caching

  io_bottleneck:
    symptoms:
      - High I/O wait time
      - Disk queue length growing
      - Network bandwidth saturation
    diagnosis:
      - iostat, iotop
      - Network monitoring
      - Disk utilisation graphs
    solutions:
      - SSD upgrade
      - I/O optimisation
      - Caching layer
      - CDN for static assets

  thread_pool_exhaustion:
    symptoms:
      - Thread count at maximum
      - Request queue growing
      - Timeout errors
      - Response time degradation under load
    diagnosis:
      - Thread dump analysis
      - Monitor active threads
      - Check for blocking operations
    solutions:
      - Increase thread pool size
      - Use async operations
      - Reduce blocking calls
      - Circuit breakers for slow dependencies

  external_api_limits:
    symptoms:
      - 429 (rate limit) errors
      - Timeout errors to external services
      - Error rate correlates with load
    diagnosis:
      - API response codes
      - Latency to external services
      - Rate limit headers
    solutions:
      - Request throttling
      - Caching API responses
      - Retry with backoff
      - Request batching
      - Negotiate higher limits

  garbage_collection:
    symptoms:
      - Periodic response time spikes
      - Stop-the-world pauses
      - Sawtooth memory pattern
    diagnosis:
      - GC logs
      - JVM metrics
      - Heap dump at peak
    solutions:
      - Tune GC parameters
      - Use G1GC or ZGC
      - Reduce allocation rate
      - Increase heap size

Complete analysis workflow:

#!/bin/bash
# analyse_load_test.sh - Comprehensive load test analysis

RESULTS_DIR="results"
REPORT_DIR="analysis"

mkdir -p "$REPORT_DIR"

echo "🔍 Starting load test analysis..."

# 1. Basic statistics
echo "📊 Calculating basic statistics..."
python3 scripts/calculate_stats.py \
  "$RESULTS_DIR/results.csv" \
  > "$REPORT_DIR/stats.json"

# 2. Percentile analysis
echo "📈 Analysing percentile distribution..."
python3 scripts/analyse_percentiles.py \
  "$RESULTS_DIR/results.csv"
mv percentile_analysis.png "$REPORT_DIR/"

# 3. Saturation point detection
echo "🔴 Detecting saturation point..."
python3 scripts/find_saturation.py \
  "$RESULTS_DIR/results_by_load.csv"
mv saturation_analysis.png "$REPORT_DIR/"

# 4. Correlation with system metrics
echo "🔗 Correlating with system metrics..."
python3 scripts/correlate_metrics.py \
  "$RESULTS_DIR/results.csv" \
  "$RESULTS_DIR/system_metrics.csv" \
  > "$REPORT_DIR/correlations.txt"

# 5. Error analysis
echo "❌ Analysing error patterns..."
python3 scripts/analyse_errors.py \
  "$RESULTS_DIR/errors.log" \
  > "$REPORT_DIR/error_analysis.txt"

# 6. Resource utilisation
echo "💻 Analysing resource utilisation..."
python3 scripts/plot_resources.py \
  "$RESULTS_DIR/system_metrics.csv"
mv resource_utilisation.png "$REPORT_DIR/"

# 7. Generate summary report
echo "📝 Generating summary report..."
python3 scripts/generate_report.py \
  --stats "$REPORT_DIR/stats.json" \
  --correlations "$REPORT_DIR/correlations.txt" \
  --output "$REPORT_DIR/summary.html"

echo "✅ Analysis complete! Report available at: $REPORT_DIR/summary.html"

# 8. Check for common issues
echo ""
echo "🔍 Common Issue Detection:"

# Check CPU correlation
cpu_corr=$(grep "Response Time vs CPU" "$REPORT_DIR/correlations.txt" | awk '{print $NF}')
if (( $(echo "$cpu_corr > 0.7" | bc -l) )); then
  echo "  ⚠️  High CPU correlation detected - likely CPU-bound"
fi

# Check for high error rates
error_rate=$(jq -r '.error_rate' "$REPORT_DIR/stats.json")
if (( $(echo "$error_rate > 1.0" | bc -l) )); then
  echo "  ⚠️  Error rate exceeds 1% - investigate errors"
fi

# Check p99/p50 divergence
divergence=$(jq -r '.percentile_divergence' "$REPORT_DIR/stats.json")
if (( $(echo "$divergence > 3" | bc -l) )); then
  echo "  ⚠️  High percentile divergence - inconsistent performance"
fi

echo ""
echo "📊 Key Recommendations:"
python3 scripts/generate_recommendations.py "$REPORT_DIR/stats.json"

Quick Reference

Tool Selection Matrix

Scenario	Recommended Tool	Reasoning
Modern REST APIs	k6	Native JavaScript, built-in metrics, cloud integration
Python ecosystem	Locust	Python syntax, easy integration with Python tools
Enterprise/legacy	JMeter	Mature, extensive protocol support, GUI for non-coders
Kubernetes-native	k6-operator	Native K8s CRDs, easy distributed testing
High RPS testing	k6 or Locust	Better performance than JMeter for high concurrency
Complex workflows	Locust or JMeter	Full programming capability (Python/Java)
CI/CD integration	k6	Best CLI experience, built-in thresholds

Essential k6 Commands

# Run test
k6 run test.js

# Run with custom VUs and duration
k6 run --vus 10 --duration 30s test.js

# Run with environment variables
k6 run -e BASE_URL=https://api.example.com test.js

# Output to InfluxDB
k6 run --out influxdb=http://localhost:8086/k6 test.js

# Run cloud test
k6 cloud run test.js

# Run with thresholds
k6 run --threshold http_req_duration=p95<500 test.js

Essential Locust Commands

# Start web UI
locust -f locustfile.py

# Headless mode
locust -f locustfile.py --headless -u 100 -r 10 --run-time 10m

# Distributed master
locust -f locustfile.py --master

# Distributed worker
locust -f locustfile.py --worker --master-host=192.168.1.10

# Custom host
locust -f locustfile.py --host https://api.example.com

# Export results
locust -f locustfile.py --headless --html report.html --csv results

Essential JMeter Commands

# Run test
jmeter -n -t test.jmx -l results.jtl

# Run with properties
jmeter -n -t test.jmx -Jthreads=50 -Jduration=300

# Generate HTML report
jmeter -g results.jtl -o report

# Distributed mode
jmeter -n -t test.jmx -R worker1,worker2,worker3

# Start server (worker)
jmeter-server

# Validate test plan
jmeter -n -t test.jmx --validate

Key Metrics to Monitor

Metric	Description	Good Target
P50 Latency	Median response time	< 100ms
P95 Latency	95th percentile	< 500ms
P99 Latency	99th percentile	< 1000ms
Error Rate	Failed requests %	< 0.1%
Throughput	Requests per second	Depends on capacity
Concurrent Users	Active virtual users	Test target

Threshold Examples

// k6 thresholds
export const options = {
    thresholds: {
        'http_req_duration': ['p(95)<500', 'p(99)<1000'],
        'http_req_failed': ['rate<0.01'],
        'http_reqs': ['rate>100'],
        'checks': ['rate>0.95'],
    },
};

Common Issues and Solutions

Issue	Symptoms	Solution
Load generator bottleneck	CPU at 100% on load generator, throughput plateaus	Use distributed testing, optimise test script, use faster tool (k6)
Clock skew in distributed tests	Inconsistent timestamps, aggregation errors	Use NTP synchronisation, timestamp from controller
Memory leaks in test script	Generator memory grows over time, eventual crash	Fix variable scoping, clear arrays, use data streaming
Connection pool exhaustion	Timeout errors, "connection refused"	Increase system file descriptors, tune `ulimit`, use HTTP keep-alive
Test data depletion	Same data reused, test becomes unrealistic	Use larger dataset, implement data generation, cycle through data
Network bandwidth limits	High latency despite low CPU, packet loss	Test from multiple regions, increase network capacity, reduce payload size
Target system overwhelmed	All requests failing, system unresponsive	Reduce load, implement gradual ramp-up, add autoscaling
Inconsistent results	High variance between runs	Warm up period, consistent environment, eliminate background jobs
SSL/TLS overhead	High CPU on load generator with HTTPS	Disable SSL verification (test only), use keep-alive, distribute load
DNS resolution delays	Initial requests slow, subsequent faster	Use IP addresses, implement DNS caching, pre-resolve hostnames
Results not aggregating	Missing data in distributed mode	Check network connectivity, verify compatible tool versions, check logs

Debugging Tips

# Check system limits
ulimit -a

# Increase file descriptors
ulimit -n 65536

# Check network connectivity
curl -w "@curl-format.txt" -o /dev/null -s https://api.example.com

# curl-format.txt
# time_namelookup: %{time_namelookup}s\n
# time_connect: %{time_connect}s\n
# time_starttransfer: %{time_starttransfer}s\n
# time_total: %{time_total}s\n

# Monitor load generator resources
htop  # or top
iostat -x 1
netstat -an | grep ESTABLISHED | wc -l

# Check for port exhaustion
cat /proc/sys/net/ipv4/ip_local_port_range
# Increase range if needed:
# echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range

# Enable TIME_WAIT reuse
sysctl net.ipv4.tcp_tw_reuse=1

# Verify Docker resource limits (if using containers)
docker stats

# Check k6 detailed output
k6 run --verbose test.js

# Locust debug mode
locust -f locustfile.py --loglevel DEBUG

Load Testing (k6/Locust/JMeter)

Overview

Test Plan Design and Parameterisation

Key Concepts

Common Patterns

Examples

Distributed Load Generation

Key Concepts

Common Patterns

Examples

Metrics Capture and Dashboards

Key Concepts

Common Patterns

Examples

CI Integration for Performance Gates

Key Concepts

Common Patterns

Examples

Interpreting Bottlenecks and Saturation

Key Concepts

Common Patterns

Examples

Quick Reference

Tool Selection Matrix

Essential k6 Commands

Essential Locust Commands

Essential JMeter Commands

Key Metrics to Monitor

Threshold Examples

Common Issues and Solutions

Debugging Tips

Related Topics