Chaos Engineering & Resilience

Design a chaos engineering strategy that integrates with performance testing for [application name]:

Architecture: [describe: microservices on Kubernetes / monolith on VMs / serverless / etc.]
Current observability: [Datadog / Grafana / Prometheus / etc.]
Current testing maturity: [level 1–4]

Provide:
- What chaos engineering adds beyond traditional load testing
- Steady-state hypothesis definition for [application]
- Chaos experiment catalog relevant to performance (latency injection, resource exhaustion, dependency failure)
- How to run chaos experiments in parallel with JMeter / k6 load tests
- Tooling options: Chaos Monkey, LitmusChaos, Gremlin, AWS Fault Injection Simulator, Chaos Mesh
- Entry criteria before running chaos experiments (must have load baseline first)
- Blast radius control strategy
- Rollback procedures

I want to inject artificial network latency to test my application's resilience:

Target: inject [N]ms latency on calls from [Service A] to [Service B / database / external API]
Load: simultaneous JMeter / k6 load test at [N] VUs
Hypothesis: application degrades gracefully. p99 increases but error rate stays < [N]%

Provide:
- tc (traffic control) commands to inject latency on Linux
- Toxiproxy setup as a chaos proxy between services
- AWS Fault Injection Simulator equivalent
- Istio fault injection YAML (if service mesh is available)
- LitmusChaos NetworkChaos experiment YAML for Kubernetes
- How to measure the impact: JMeter response time delta vs injected latency
- Expected cascading effects: timeout propagation, retry storms, circuit breaker activation
- How to restore normal network conditions after the test

I want to test how my application behaves when server resources are constrained during load:

Experiments to run:
1. CPU stress: 80% CPU utilization on app server while running load test
2. Memory pressure: consume 70% of available memory
3. Disk I/O saturation: heavy disk write contention

For each experiment:
- Tool to inject the stress: stress-ng, sysbench, dd, or container resource limits
- Exact commands / Kubernetes resource quota YAML
- JMeter / k6 load profile to run simultaneously
- Metrics to watch: CPU steal time, memory swap usage, disk await, app response time
- Expected degradation patterns and how to identify graceful vs catastrophic failure
- Recovery test: remove stress and measure time-to-recover to baseline performance
- Go / No-Go criteria for production readiness

My application depends on the following external services:
1. [Service A]. [REST API / gRPC / message queue]
2. [Service B]. [Database / Cache / S3]
3. [Service C]. [Third-party API]

Design chaos experiments to test failure resilience under load:

For each dependency:
- Failure mode: full outage / slow response ([N]ms latency) / partial failure ([N]% error rate)
- Chaos tool: Toxiproxy / Istio fault injection / Gremlin / AWS FIS
- Simultaneous JMeter load: [N] VUs
- Steady-state hypothesis: what should the app do? (return cached data / circuit open / graceful error / queue messages)
- How to implement with Toxiproxy (provide config JSON)
- How to implement with Istio VirtualService fault injection (provide YAML)
- Success criteria: error handling works, no cascading failures, circuit breaker trips correctly
- Metrics to validate: circuit breaker open rate, fallback invocation count, user-facing error rate

I am running my application on Kubernetes and want to validate resilience with pod-level chaos while running a load test.

Experiments:
1. Pod Kill: randomly kill [N]% of pods during peak load
2. Pod CPU Hog: inject CPU pressure inside the pod
3. Node Drain: drain a worker node while load test is running
4. Network Partition: block traffic between pods in different namespaces

Tool: [LitmusChaos / Chaos Mesh / AWS FIS / kubectl manual]

For each experiment provide:
- LitmusChaos ChaosEngine YAML or Chaos Mesh YAML
- kubectl equivalent for manual reproduction
- JMeter / k6 load profile to run simultaneously
- Kubernetes metrics to watch: pod restarts, HPA scaling events, pending pods, node pressure
- Expected k8s recovery behavior: ReplicaSet respawn, rolling update, HPA scale-out
- How to verify zero-downtime deployment under chaos
- Pass criteria: application recovers within [N] seconds, error rate spike < [N]%

My application uses a circuit breaker pattern (Resilience4j / Hystrix / Istio) for [service name].

Circuit breaker config:
- Failure rate threshold: [N]%
- Slow call threshold: [N]ms
- Wait duration in open state: [N]s
- Half-open permitted calls: [N]

Design a performance test that validates circuit breaker behavior:
- Phase 1: Normal load. circuit CLOSED, validate baseline
- Phase 2: Inject failures. circuit should OPEN when threshold is breached
- Phase 3: Wait duration. validate requests are rejected fast (fail-fast behavior)
- Phase 4: Half-open. validate recovery probe calls
- Phase 5: Recovery. circuit CLOSED again, validate normal behavior resumes

Provide:
- JMeter test plan structure for each phase (Thread Groups / Controllers)
- How to trigger failures: Toxiproxy / Istio fault injection / mock server returning 500
- Groovy / JSR223 script to log circuit state transitions during test
- Metrics: circuit breaker state (Micrometer / Prometheus), fallback invocation rate, error rate per phase
- Pass/fail criteria per phase

Generate a chaos experiment report template for the following completed experiment:

Experiment name: [e.g., Network latency injection on payment-service]
Date: [date]
Team: [team name]
System under test: [service / component]
Load during experiment: [N VUs / N RPS]

Report structure:
1. Executive Summary (3 sentences max)
2. Steady-State Hypothesis (before and after verification)
3. Experiment Setup (blast radius, tooling, duration)
4. Results (metrics before / during / after with tables)
5. Observations (what happened, what didn't)
6. Weaknesses Discovered
7. Recommendations (circuit breakers, timeouts, retries, fallbacks)
8. Action Items with owners and due dates
9. Lessons Learned

Format as a professional Markdown document ready for Confluence.

During chaos testing with [N]% packet loss injected, my load test showed:
- Response times spiked from [N]ms to [N]ms
- Throughput dropped but then spiked back above baseline
- Error rate increased and then decreased but with high latency

This looks like a retry storm. Diagnose and fix:

1. What is a retry storm and why does it amplify failures?
2. How to detect retry storm in JMeter results (throughput spike pattern, response time bimodal)
3. How to detect it from APM metrics (request rate spike on server side vs client side)
4. Retry configuration best practices: max retries, exponential backoff, jitter
5. Resilience4j Retry configuration with exponential backoff + jitter (Java code snippet)
6. k6 retry simulation for testing backoff behavior
7. How to distinguish retry traffic from legitimate traffic in JMeter results
8. Circuit breaker + retry interaction: which should trigger first?