Chaos Engineering: Breaking Things on Purpose

Introduction

You cannot test resilience by reading code. A system that handles normal traffic perfectly can fail spectacularly under partial failures. Dependencies fail in combinations you did not anticipate. Caches expire simultaneously. Network partitions isolate data centers.

Traditional testing cannot find these weaknesses. Unit tests test components. Integration tests test connections. Load tests test performance. None test what happens when parts of your system fail.

Chaos engineering fills this gap. You deliberately break things to discover what breaks. Then you fix it before users discover it instead.

Core Concepts

Netflix formalized chaos engineering principles. Rather than guessing what might fail, you design experiments to discover weaknesses systematically. Each principle builds toward a culture of proactive resilience testing.

Building a Hypothesis

Before running an experiment, state what you expect to happen. “If we kill one instance of Service A, requests will reroute to other instances with less than 1% errors.” This gives your experiment a clear pass/fail criterion.

Varying Real-World Events

Focus on real failures. Server crashes, network latency, disk full, CPU exhaustion. Not theoretical attacks on hypothetical vulnerabilities.

Running in Production

Test environments never match production. Traffic patterns differ. Dependencies differ. Load differs. Run experiments in production, but with safeguards.

Automating Experiments

Manual chaos is not chaos engineering. Run experiments continuously. Automate the injection and measurement.

Minimizing Blast Radius

Start small. Kill one instance, not three. Introduce 100ms latency, not 10 seconds. You want to learn something without causing an actual outage.

Types of Chaos Experiments

Different failure modes reveal different weaknesses. Infrastructure failures expose gaps in your deployment and recovery logic. Network failures test how your services handle unreliable communication. Application failures validate your error handling and resource management. Dependency failures check whether your system degrades gracefully when upstream services misbehave.

Infrastructure Chaos

Kill servers. Stop containers. Fill disks. Exhaust memory. These experiments test whether your infrastructure recovers from node failures.

# Chaos experiment: kill a random instance
def kill_random_instance():
    instances = ec2.describe_instances(Filters=[{'Name': 'tag:Service', 'Values': ['payment']}])
    target = random.choice(instances)
    ec2.terminate_instances(InstanceIds=[target['InstanceId']])
    return target

Network Chaos

Introduce latency. Drop packets. Partition network. DNS failure. These experiments test how your system handles network issues.

# Introduce 200ms latency to a service
tc qdisc add dev eth0 root netem delay 200ms

# Remove latency
tc qdisc del dev eth0 root netem

Application Chaos

Crash application processes. Throw exceptions. Consume resources. These experiments test application-level fault tolerance.

# Kill a random process of a service
def kill_process(service_name):
    processes = psutil.process_iter(['pid', 'name', 'cmdline'])
    for p in processes:
        if service_name in p.info['name']:
            p.kill()
            return p.info['pid']

Dependency Chaos

Fail external services. Introduce latency to databases. Return errors from APIs. These experiments test how your system handles upstream failures.

# Chaos experiment: make downstream service fail
def inject_service_failure(service_url):
    # Return 503 for all requests to the service
    iptables -A OUTPUT -p tcp -d {service_ip} -j DROP

Running a Chaos Experiment

A chaos experiment follows a structured lifecycle: define what normal looks like, form a hypothesis about what will happen, inject the failure, observe the results, decide whether to stop or continue, then restore the system. This loop builds confidence that your system handles real failures gracefully.

Experiment Steps

Step 1: Define Steady State

Before breaking things, measure normal behavior. What is your baseline error rate, latency, and throughput? Know what “normal” looks like so you recognize problems when they appear.

def measure_steady_state():
    errors = 0
    total = 0
    for _ in range(1000):
        try:
            response = requests.get('http://api.example.com/health')
            if response.status_code != 200:
                errors += 1
        except:
            errors += 1
        total += 1
    return errors / total  # Baseline error rate

Step 2: Form a Hypothesis

State what you expect: “If we introduce 500ms latency to the database, error rate will stay below 1% because connections will timeout and retry.”

Step 3: Inject Failure

Start small. Introduce the failure.

# Introduce 500ms latency to database
chaos_controller.inject_latency('database', 500)

Step 4: Observe

Measure the same metrics you measured for steady state. Did the system behave as you expected?

error_rate = measure_steady_state()
if error_rate > 0.01:  # Hypothesis violated
    alert_team("Chaos experiment failed: error rate exceeded threshold")
    rollback()

Step 5: Stop or Mitigate

If the system handled the failure, document the finding. If the system failed catastrophically, stop the experiment and escalate. Fix the weakness before running more experiments.

Step 6: Restore

Always restore the system to normal state. Automated rollbacks ensure nothing stays broken.

Game Days

A game day turns chaos engineering into a team sport. Rather than running experiments in isolation, you bring the whole team together to practice failure scenarios. The goal is to build muscle memory for incident response so that when real failures occur, everyone knows their role and executes without panic.

Preparing for a Game Day

1. Define scenarios to test
2. Assign roles: injector, observers, incident commander
3. Set start and end times
4. Prepare rollback procedures
5. Notify stakeholders

Running the Exercise

1. Start with a scenario walkthrough
2. Inject the failure
3. Observe and document
4. Call the incident if needed
5. Debrief after

Game Day Scenarios

• Single data center failure
• Database becomes read-only
• API rate limit exceeded
• TLS certificate expires
• Message queue backs up

Tools for Chaos Engineering

The chaos tooling landscape spans from simple scripts to enterprise platforms. Netflix built the first widely-known tool, but the ecosystem has grown to support Kubernetes-native chaos, cloud-provider managed services, and custom automation. Choosing the right tool depends on your environment, team size, and maturity with chaos engineering.

Tool Categories

Chaos Monkey

Netflix’s original. Randomly kills EC2 instances. Simple but effective for testing basic resilience.

Gremlin

A commercial chaos tool with Kubernetes support, targeting options, and safety features. Suitable for teams beginning with chaos engineering.

Litmus

Open source chaos for Kubernetes. Chaos experiments defined as custom resources. Integrates with Prometheus for monitoring.

Your Own Scripts

For simple experiments, scripts work fine. You do not need a full chaos platform to get started.

Tool Evaluation Matrix

Tool Comparison Matrix

Tool	Type	Kubernetes Support	Target Options	Safety Features	Cost	Best For
Chaos Monkey	Open Source	No (AWS only)	Instance-based	Basic	Free	Netflix-style EC2 killing
Gremlin	Commercial	Yes	Service, Pod, Node	Attack visualizer, halt button	Paid	Teams starting with chaos; enterprise support
Litmus	Open Source	Yes (native)	Pod, Node, Network	Custom resources, Argo integration	Free	Kubernetes-native environments; GitOps workflows
AWS FIS	Managed Service	N/A (AWS-native)	AWS resources	CloudWatch integration, IAM controls	Pay-per-use	AWS workloads without third-party tools
Azure Chaos Studio	Managed Service	N/A (Azure-native)	VM, Kubernetes, PaaS	Logic Apps integration	Pay-per-use	Azure workloads
Custom Scripts	DIY	Varies	Highly flexible	Depends on implementation	Free (dev time)	Simple experiments; specific edge cases

Reference and Operations

Selection Criteria

• AWS workloads: Use AWS Fault Injection Simulator (native, no extra tools)
• Azure workloads: Use Azure Chaos Studio (native, no extra tools)
• Kubernetes-first: Litmus for open source, Gremlin for enterprise features
• Simple EC2 chaos: Chaos Monkey or custom scripts
• Enterprise requirements: Gremlin provides SLA, support, and advanced targeting

CI/CD Integration

Integrate chaos experiments into your deployment pipeline to validate resilience automatically:

# GitHub Actions workflow for chaos testing
name: Chaos Engineering Pipeline

on:
  push:
    branches: [main]
  schedule:
    # Run chaos experiments weekly in production
    - cron: "0 2 * * 0"

jobs:
  chaos-experiment:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Run Steady State Check
        run: |
          ./scripts/measure_steady_state.sh
          echo "baseline_error_rate=$(cat steady_state.txt)" >> $GITHUB_ENV

      - name: Inject Chaos - Database Latency
        run: |
          kubectl label namespace chaos-testing experiment=active
          helm install chaos-litmus litmuschaos/litmus -n chaos-testing

          # Wait for steady state to stabilize
          sleep 30

          # Run the experiment
          kubectl apply -f experiments/database-latency.yaml -n payment

      - name: Monitor Impact
        run: |
          ./scripts/monitor_during_chaos.sh
          ERROR_RATE=$(cat error_rate.txt)

          # Compare to baseline
          if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
            echo "Experiment failed: error rate $ERROR_RATE exceeds threshold"
            exit 1
          fi

      - name: Cleanup
        if: always()
        run: |
          kubectl delete -f experiments/database-latency.yaml -n payment || true
          helm uninstall chaos-litmus -n chaos-testing || true

      - name: Document Results
        if: always()
        run: |
          ./scripts/upload_chaos_results.sh

# Litmus experiment definition for CI/CD
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: db-latency-chaos
  namespace: payment
spec:
  appinfo:
    appns: payment
    applabel: "app=checkout"
  engineState: active
  chaosServiceAccount: litmus-admin
  experiments:
    - name: db-latency
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: LATENCY
              value: "500"
            - name: DB_component
              value: "postgres"

Cloud Provider Chaos Services

AWS and Azure both ship built-in chaos testing that ties directly into their infrastructure. These managed services handle agent deployment, IAM integration, and rollback automation out of the box. The tradeoff is vendor lock-in, but for teams running predominantly on one cloud, native tools remove operational overhead.

AWS Fault Injection Simulator (FIS)

AWS FIS lets you inject faults into EC2, ECS, EKS, and RDS without managing separate tooling. You define experiments in JSON templates and run them against target resources.

# AWS FIS: Simulate EC2 instance failure
- action: aws:ec2:terminate-instances
  targets:
    instances: target-group
  description: "Terminate random EC2 instance"
  parameters:
    instanceTerminationPercentage: 50

AWS FIS integrates with CloudWatch alarms to automatically roll back experiments if metrics degrade beyond thresholds. This is safer than open-loop chaos.

Azure Chaos Studio

Azure Chaos Studio works similarly, with support for virtual machines, AKS, Service Bus, and Cosmos DB. Experiments use a visual designer or programmatic API.

{
  "actions": [
    {
      "type": " faults.azure.vms.terminate",
      "parameters": {
        "nodeCount": 1,
        "abruptionDuration": "PT30S"
      }
    }
  ],
  "selectors": [
    {
      "type": "TagSelector",
      "tags": [{ "key": "chaos", "value": "enabled" }]
    }
  ]
}

Cloud-native tools have a real advantage: no agent deployment, native IAM integration, and rollback automation built in. The tradeoff is being locked to one cloud provider.

When to Use Cloud-Native vs Third-Party

Criteria	Cloud-Native (FIS, Chaos Studio)	Third-Party (Gremlin, Litmus)
Multi-cloud	No	Yes
Deep infrastructure integration	Yes	Requires agents
Managed service	Yes	No
Cost	Pay-per-use	Subscription

Steady-State Automation

After running chaos experiments, feed the results back into your system automatically. Steady-state automation means your system continuously validates its own resilience instead of treating chaos testing as a one-off exercise.

A minimal steady-state loop:

def steady_state_loop():
    while True:
        # Define what "healthy" looks like
        baseline = {
            'p99_latency_ms': 200,
            'error_rate': 0.01,
            'availability': 0.999
        }

        # Run a small chaos experiment
        result = run_minimal_chaos()

        # Measure actual state
        actual = measure_system_health()

        # Compare against baseline
        if mismatches(actual, baseline):
            alert_oncall()
            rollback_experiment()

        sleep(interval=3600)  # Run every hour

This does not replace full chaos engineering. It catches regressions before they reach production.

Building a Chaos Practice

Moving from occasional experiments to a mature chaos practice requires structure. You start small to build confidence, then gradually increase scope as you learn what breaks. Automation keeps experiments running continuously, and sharing findings ensures the whole organization benefits from what you discover.

Start Small

Kill one server in a test environment. See what happens. Fix what breaks. This teaches more than any book.

Graduate to Production

Once comfortable in test, run experiments in production. Start with low-risk experiments: kill instances in a redundant service, introduce small latency.

Automate

Move from manual to automated experiments. Run them continuously in production. The more you automate, the more you learn.

Share Findings

When an experiment reveals a weakness, share the finding. Write post-mortems. Add to runbooks. The goal is organizational learning.

When to Use / When Not to Use Chaos Engineering

Use chaos engineering when:

• You have production systems with real users who would be affected by outages
• Your team has monitoring and observability in place to measure impact
• You have rollback procedures that can stop experiments safely
• You have identified specific failure modes you want to validate

Do not use chaos engineering when:

• Your system is unstable and cannot handle normal load without chaos
• You lack observability to distinguish experiment impact from real failures
• You have no way to stop experiments if things go wrong
• Your organization is not prepared to act on findings

Trade-off Analysis

Kubernetes-based vs VM-based Chaos

Aspect	Kubernetes Chaos	VM/Instance Chaos
Blast radius control	Namespace + pod selectors	Tag-based instance targeting
Injection precision	Container-level resource exhaustion	Instance-level termination
Recovery speed	Auto-healing via controllers	Manual or ASG-based recovery
Network chaos depth	Pod-to-pod, service-level partitioning	Subnet, security group blocking
Tool requirements	Litmus, Chaos Mesh, service mesh faults	Chaos Monkey, Gremlin, custom scripts
Learning curve	Higher (Kubernetes knowledge required)	Lower (standard cloud DevOps)
Failure mode coverage	Container crashes, OOMKilled, evicted pods	Instance termination, network partition

Trade-off Summary

Factor	With Chaos Engineering	Without Chaos Engineering
Failure Discovery	Proactive - find weaknesses before users	Reactive - users discover failures first
Confidence	High - validated under controlled conditions	Moderate - assumes architecture is sound
Risk	Controlled experiments with kill switches	Uncontrolled production failures
Cost	Tooling, training, experiment time	Outages, incident response, recovery
Team Skills	Requires chaos expertise and experiment design	Standard DevOps skills
Culture	Requires blameless post-mortem culture	Standard incident response
Time to Value	Weeks to months for meaningful experiments	Immediate, but reactive
Coverage	Discovers unknown-unknowns	Only known-knowns from monitoring

Real-world Failure Scenarios

Actual chaos engineering deployments have uncovered critical weaknesses that traditional testing missed. These documented cases illustrate how controlled experiments reveal systemic risks.

Netflix - Cascading Database Failures

During a chaos experiment targeting database availability, Netflix discovered that losing a single database instance caused a cascading failure across multiple services. The root cause: connection pool exhaustion when retries flooded surviving instances. The fix was implementing connection pool limits and per-service circuit breakers.

LinkedIn - Service Mesh Partition Blind Spots

LinkedIn’s team ran network partition experiments and found that their service mesh’s health checks passed while actual inter-service communication was failing. The mesh marked pods healthy even when they could not receive traffic. This revealed a gap between liveness and availability that health checks alone did not catch.

Slack - Memory Leak Cascade

A chaos experiment that killed worker processes revealed a hidden memory leak in Slack’s job queue. When workers restarted, the queue would flood them with pending work, causing memory exhaustion again. The experiment exposed a restart loop that only occurred under specific load conditions impossible to reproduce in staging.

Amazon - Availability Zone Failure Simulation

Amazon’s internal chaos practice simulated full availability zone failures and discovered that their auto-scaling policies were too conservative for AZ-level loss. The scaling policies assumed gradual traffic increases, not sudden 33% capacity loss. This finding led to multi-AZ deployment requirements for all critical services.

Chess.com - DNS Chaos Revealed Cache Dependency

When Chess.com injected DNS failures, they discovered their application incorrectly cached DNS lookups for critical service discovery. Services failed to reconnect after DNS TTL expired because the application treated DNS failures as permanent rather than transient.

Common Pitfalls / Anti-Patterns

Teams new to chaos engineering often make predictable mistakes. These pitfalls can turn valuable experiments into real outages or wasted effort. Recognizing them helps you design safer experiments and extract more value from your chaos practice.

No Rollback Plan

Running chaos without a way to stop is reckless. Always have a rollback plan. If things go wrong, you need to be able to stop.

Testing in a Vacuum

Experiments that run but nobody watches teach nothing. Monitor the system during experiments. If you do not see the failure, you did not learn anything.

Too Aggressive Too Fast

Starting with large-scale failures causes real outages. Start with small failures. Gradually increase scope as you build confidence.

Ignoring Results

Experiments that reveal weaknesses but nobody fixes them are wasted. Follow through. The point is to improve resilience, not to watch things break.

Interview Questions

Further Reading

Internal Resources

• Disaster Recovery — Planning for failures at the infrastructure and data layer
• Resilience Patterns — Retry, timeout, circuit breaker, and bulkhead patterns that work with chaos engineering
• Circuit Breaker Pattern — Preventing cascade failures that chaos experiments may uncover
• System Design Roadmap — Complete learning path covering chaos engineering in context

External Resources

• Principles of Chaos Engineering — The foundational document defining chaos engineering scope and methodology
• Chaos Monkey — Netflix’s original chaos engineering tool that randomly kills production instances
• Gremlin — Commercial chaos engineering platform with guided experiments and safety features
• AWS Fault Injection Simulator — Managed chaos engineering service with built-in safety controls and automation
• LitmusChaos — CNCF-hosted chaos engineering platform for Kubernetes and cloud-native workloads
• Chaos Engineering at Netflix and Beyond (PDF) — Academic treatment of chaos engineering principles, Netflix’s implementation, and lessons learned
• The Veritas Session — Netflix talk on the evolution from Chaos Monkey to proactive failure injection

Conclusion

Chaos engineering is not about breaking things. It is about finding weaknesses before users find them. The goal is to build confidence that your system will survive failures.

The first time you run chaos, you will find problems. Fix them. Run again. Eventually, your system handles chaos and your team handles incidents with confidence.

Key Bullets:

• Chaos engineering is about finding weaknesses before users find them
• Every experiment needs a hypothesis and measurable pass/fail criteria
• Start small: one instance, small latency injection, limited scope
• Automate experiments for continuous validation
• Follow through on findings. Track to completion and fix the weaknesses

Copy/Paste Checklist:

Before running chaos experiment:
[ ] Define steady-state baseline metrics
[ ] State hypothesis: what do you expect to happen?
[ ] Define pass/fail criteria
[ ] Assign roles: injector, observer, incident commander
[ ] Prepare rollback/kill switch
[ ] Notify stakeholders if production
[ ] Verify monitoring is capturing all metrics
[ ] Document expected duration

During experiment:
[ ] Monitor steady-state metrics continuously
[ ] Watch for cascading effects
[ ] Stop if criteria exceeded

After experiment:
[ ] Restore system to normal state
[ ] Document findings
[ ] Schedule follow-up for any weaknesses found
[ ] Update runbooks with lessons learned

Observability Checklist

• Metrics:

  • Error rate (baseline vs experiment)
  • Latency P50/P95/P99
  • Throughput (requests per second)
  • Resource utilization (CPU, memory, disk, network)
  • Dependency health indicators

• Logs:

  • Experiment start/end timestamps with hypothesis
  • System behavior observations during experiment
  • Any anomalies detected
  • Post-experiment findings and recommendations

• Alerts:

  • Error rate exceeds 1% during experiment (warning) / 5% (stop experiment)
  • Latency P99 doubles during experiment
  • Any resource exhaustion detected
  • Experiment duration exceeds planned window

Security Checklist

• Limit chaos tooling access to authorized personnel only
• Audit log all experiment executions with timestamps and owners
• Prevent experiments from affecting security controls (authentication, encryption)
• Ensure experiments cannot exfiltrate data or breach isolation
• Validate that rollback procedures do not introduce security gaps
• Test chaos tooling for vulnerabilities (injection, privilege escalation)

For more on resilience, see Disaster Recovery and Resilience Patterns.