🚀 Advanced & Capstone · Days 25–30
Day 27: Chaos Under Load
Objective
Kill a node or inject a failure mid-test; observe resilience.
Scenario
Run a steady load. While the test is active, restart a service, kill a pod, or block a downstream dependency. Watch error rate, failover time, and recovery speed.
Metrics to Watch
When It Clicks
Resilience is only real when proven under load.
Solution
Run a steady-state load at 60% of capacity. In a second terminal: kill a pod (`kubectl delete pod <name>`), block a port (`sudo iptables -A INPUT -p tcp --dport 8080 -j DROP`), or pause a container (`docker pause <id>`). Watch the JMeter error rate spike. A well-architected system shows: a brief error spike (< 5s), automatic failover or retry, then error rate returns to near-zero. Catastrophic failure is when the error rate stays high until manual intervention. Restore with `kubectl rollout restart` or `docker unpause`. Document the mean time to recover (MTTR).
Reflection
How long did the system take to recover? Were errors graceful or catastrophic?
Deliverable
A chaos + load combined scenario with resilience notes.
⚠️ Never run load tests against infrastructure you do not own or have explicit written permission to test. Always use a dedicated test environment. Unauthorized load injection can cause outages, trigger legal liability, and violate terms of service.