Resilience and chaos basics

Most test suites check that a system behaves correctly under normal conditions. Chaos engineering asks a harder question: what happens when something breaks? It is the practice of deliberately injecting failure into a running system, killed instances, slow networks, timed-out dependencies, to see whether it degrades gracefully or falls over completely.

The goal is not to break things for fun. It is to find weaknesses before an outage finds them, in a controlled way with a rollback plan and a clear hypothesis. Teams that build test suites at scale eventually run into this discipline once functional and performance coverage stop catching production incidents.

Why inject failure on purpose

Traditional testing proves a system works when everything cooperates. Production rarely cooperates. Disks fill up, a downstream API times out, a region loses connectivity, or a single instance gets killed by the orchestrator mid-request.

If the first time your system meets these conditions is during a real incident, you learn about your architecture's weak points at the worst possible moment. Chaos experiments move that discovery earlier and put it on your own schedule.

You form a hypothesis first. For example: "if the payments service drops 50% of requests, checkout should fall back to a queued retry instead of showing a 500 error." Then you inject exactly that failure, in staging or carefully in production, and watch whether the hypothesis holds.

Common failure modes to inject

Chaos tooling generally targets a handful of failure categories:

• Instance or process kills: terminate a container, pod, or VM and confirm the system reroutes traffic without dropping requests.
• Network faults: add latency, packet loss, or full partitions between services to test timeout and retry logic.
• Dependency failures: force a database, cache, or third-party API to return errors or hang, and confirm circuit breakers trip.
• Resource exhaustion: fill disk, spike CPU, or exhaust memory to see whether the system sheds load instead of crashing.
• Clock and configuration drift: skew system time or push a bad config to catch assumptions baked into code.

Each of these maps to a specific resilience mechanism you are trying to prove: a health check, a retry with backoff, a circuit breaker, or a load shedding rule.

Tools of the trade

Netflix's Chaos Monkey, part of the older Simian Army suite, popularized the idea. It randomly terminated production instances to force engineers to build for failure from day one. It proved the concept but stayed narrow in scope, mostly instance kills within Netflix's own AWS setup.

Modern tooling generalizes the idea considerably. Gremlin offers a managed platform with a library of "attacks" covering CPU, network, state, and process failures, plus scheduling and blast-radius controls so an experiment cannot spiral.

Litmus Chaos is a CNCF project built specifically for Kubernetes. It lets teams define chaos experiments as custom resources that fit naturally into a GitOps workflow and version control.

AWS Fault Injection Simulator (FIS) is the managed option for teams already on AWS. It has native integrations for EC2, ECS, EKS, and RDS, so faults can be injected without standing up separate infrastructure.

The game day practice

A "game day" is a scheduled event where a team runs a chaos experiment together, live, with an incident commander and observers watching dashboards. Unlike ad hoc chaos runs, a game day is announced and time-boxed, treated like a fire drill.

Everyone knows it is a test, but the failure and the response are real. A typical game day agenda:

1. Pick one hypothesis, for example "the checkout service survives a database failover with under 5 seconds of degraded latency."
2. Define the blast radius and an abort condition before starting.
3. Inject the fault and watch dashboards, alerts, and on-call response in real time.
4. Debrief immediately: what worked, what didn't, and what ticket gets filed.

Game days convert chaos engineering from a one-off experiment into an organizational habit. They also double as incident response training for engineers who have never paged during a real outage.

That manifest deletes a checkout-service pod for 60 seconds and lets Kubernetes reschedule it. It is a minimal worked example of the instance-kill failure mode, running as a repeatable, version-controlled experiment instead of a manual kubectl delete pod.

How this connects to load testing

Chaos engineering and load profile testing are close siblings. A load profile plans for expected demand, ramping traffic to match real usage patterns and watching how the system holds up.

Chaos engineering is the less predictable cousin. Instead of asking "can it handle more," it asks "can it handle less": less capacity, less network, less of a dependency working at all. Running both together, high load plus an injected fault, is often when the most realistic failures surface. Production incidents rarely happen during quiet periods. Combining the two disciplines mirrors that reality.

Well-designed test suites also need trustworthy inputs. Reliable test data management and clear API contract checks make it far easier to tell a genuine resilience failure apart from an artifact of bad fixtures or a stale schema.