Failure is inevitable. Resilience patterns limit blast radius, isolate faults, and enable graceful degradation to keep systems available.
1. Limiting Failure: Circuit Breakers
Definition: Prevents repeated calls to a failing dependency, avoiding timeouts and cascades.
A. How it Works
- Closed (Normal): Requests flow through. If failure rate exceeds threshold, circuit trips.
- Open (Failure): Short-circuits calls; returns error, cached value, or fallback immediately.
- Half-Open (Test): After cooldown, allows limited probes; success returns to Closed, failure re-opens.
B. Resilience Benefits
- Rapid Failure: Clients fail fast instead of waiting on timeouts.
- Recovery: Protects the failing service from retry storms.
2. Isolating Failure: Bulkheads
Definition: Resource partitioning to prevent one noisy neighbor from starving others.
A. System-Level Application
- Resource Partitioning: Separate thread/connection pools for risky vs critical traffic.
- Containment: Failure in one pool does not impact mission-critical pools.
B. Resilience Benefits
- Containment: Limits scope of resource exhaustion.
- Graceful Degradation: Non-critical features can fail without taking down core flows.
3. Managing Clients: Retries with Exponential Backoff
Definition: Safe retry strategy that spaces attempts to avoid overload.
A. Mechanism
- Retries: Retry on transient faults (HTTP 500/503, timeouts).
- Exponential Backoff: Delay grows (e.g., 1s, 2s, 4s, 8s, ...).
- Jitter: Randomized offset to prevent synchronized retry bursts.
B. Resilience Benefits
- Transient Handling: Recovers from temporary network/service issues automatically.
- Prevents Overload: Throttles retry storms protecting downstream services.
Exponential Backoff with Jitter
Example retry delays over attempts for different strategies.
4. Preventing Duplicates: Idempotency
Definition: Multiple executions of an operation yield the same effect as one.
A. Idempotency for APIs (Producers)
- Problem: Retries of non-idempotent POSTs can create duplicates (e.g., double charge).
- Mechanism: Clients send an Idempotency-Key; server/gateway deduplicates and returns prior response.
B. Idempotency for Consumers (Messaging)
- Problem: At-least-once delivery causes duplicate processing.
- Mechanism: Track processed message IDs in durable store; skip duplicates.
C. Resilience Benefits
- Safety with Retries: Enables aggressive retry policies without corrupting state.