Resilience and reliability
Anything that calls something over a network will eventually see timeouts, overloads, outages and bad data. This module gathers the reliability patterns spread across your three projects (timeouts, retries with backoff and jitter, circuit breakers, rate limiters, idempotency, dead-letter queues, graceful degradation and fault injection) into one coherent toolkit, with the exact numbers from your configuration.
By the end you can answer
- "What should you retry, and what shouldn't you?"
- "Why exponential backoff? Why add jitter?"
- "Explain the circuit breaker's three states. Which errors should count as failures?"
- "In what order did you compose retry, circuit breaker and rate limiter, and why?"
- "Why did you turn off the SDK's built-in retries?"
- "How does your system behave when Claude, Arena or Redis is down?"
- "How did you test failure handling?"
1. Why distributed calls fail
A method call either returns or throws. A network call can also hang, be slow, half-succeed (the server did the work but the reply was lost), or fail because the other side is overloaded and asking you to slow down. The classic "fallacies of distributed computing" begin: the network is reliable; latency is zero; bandwidth is infinite... none of them hold.
Your projects call three external systems: Arena (from Telemetry and the Judge), the Claude API (from the Judge) and Redis (from Arena), plus Kafka between everything. Every pattern below exists because one of those calls can misbehave.
2. Timeouts: the first line of defence
Without a timeout, a hung server holds your thread (and possibly a database connection or a Kafka partition) forever. Every remote call in your code has one:
| Call | Timeout |
|---|---|
| Judge → Claude API | request-timeout: 60s (LLM responses with tool reasoning can legitimately take many seconds) |
| Judge → Arena | connect-timeout: 2s, read-timeout: 5s |
| Producer → Kafka | delivery.timeout.ms: 120000 (total time for a send, including retries) |
Connect timeout: how long to establish the TCP connection. Read timeout: how long to wait for data once connected. Pick values from the dependency's normal latency (p99) plus margin, not from hope.
3. Retries: only for the right failures
Classify first
| Failure | Transient (retry)? | Reason |
|---|---|---|
| Connection refused or reset, timeout | Yes | Network blip, restart in progress |
| HTTP 429 Too Many Requests | Yes, after waiting (honour Retry-After) | You're over a rate limit; it resets |
| HTTP 500, 502, 503, 529 overloaded | Yes | The server is struggling now |
| HTTP 400 / 422 / 404 / 409 | No | The request itself is wrong or conflicts; the same request gives the same answer |
| HTTP 401 / 403 | No (except one re-login on 401) | A credentials or permission problem |
| Malformed message | No: a poison pill, send to a DLT | Will fail forever |
static boolean isRetryable(Throwable t) {
return t instanceof RateLimitException // 429
|| t instanceof InternalServerException // 5xx, including 529 overloaded
|| t instanceof AnthropicIoException; // network errors, timeouts
}
Retries require idempotency. Retrying a non-idempotent operation after an ambiguous failure can do it twice. The Claude call is safe to repeat (it only reads and costs tokens). Arena's POST /matches is made idempotent by externalId. Moderation actions are protected by Arena's state machine plus the Judge's re-read on 409.
Exponential backoff with jitter
Retrying immediately hammers a struggling service and makes things worse. Exponential backoff waits longer each time: wait = initial × multiplier^(attempt−1), capped at a maximum. Jitter randomises each wait. Without it, all the clients that failed at the same moment retry at the same moment again (a "thundering herd"), recreating the overload in synchronised waves.
IntervalFunction backoff = IntervalFunction.ofExponentialRandomBackoff(
s.initialBackoff(), // 1s
s.multiplier(), // 2.0
0.5, // randomisation: each wait is somewhere in [50%, 150%] of the base
s.maxBackoff()); // 20s cap
// max-attempts: 4 → 1 call + up to 3 retries, base waits about 1s, 2s, 4s (each jittered)
Honouring Retry-After
When Claude answers 429, it may say how long to wait. waitMillis uses the larger of the computed backoff and the server's Retry-After (seconds, whole or fractional), capped at 60 s so a bad header can't park a thread for hours. A server that tells you when to come back knows better than your formula.
Retry amplification: why the SDK's retries are off
The Anthropic SDK retries failed requests by itself by default. If Resilience4j also retries (4 attempts) and each attempt contains SDK retries, one logical call could become many physical calls, and nested retry layers multiply load exactly when the provider is already struggling. The README puts it as "every failure would be retried up to 12 times". The client is built with .maxRetries(0), so Resilience4j is the only retry layer for model calls, with one place to reason about attempts, backoff and metrics. General rule: retry at one layer, usually the one with the most context.
4. The circuit breaker
If a dependency is down, retrying every call still wastes time: each caller waits for timeouts, threads pile up, and the dependency is flooded the moment it tries to recover. A circuit breaker watches recent results and, past a failure threshold, stops calling for a while, failing fast instead.
failure rate ≥ 50% over the last 20 calls (min 10 calls)
┌────────┐ ──────────────────────────────────────────────► ┌────────┐
│ CLOSED │ │ OPEN │ calls rejected instantly
│ normal │ ◄──────────────┐ │ │ (CallNotPermittedException)
└────────┘ 3 trial calls │ └───┬────┘
▲ succeed │ │ after 30 s
│ ┌────┴──────┐ │
└───────────────│ HALF_OPEN │ ◄────────────────────────────┘
│ 3 trials │ ── trial fails ──► OPEN again
└───────────┘
CircuitBreakerConfig.custom()
.slidingWindowType(COUNT_BASED).slidingWindowSize(20) // judge the last 20 calls
.minimumNumberOfCalls(10) // don't trip on 1 failure out of 2
.failureRateThreshold(50) // open at 50% failures
.waitDurationInOpenState(Duration.ofSeconds(30)) // stay open 30 s
.permittedNumberOfCallsInHalfOpenState(3) // then try 3 calls
.recordExceptions(InternalServerException.class, AnthropicIoException.class) // "it is down"
.ignoreExceptions(BadRequestException.class, RateLimitException.class,
UnauthorizedException.class, PermissionDeniedException.class) // "it is up"
.build();
Only errors that mean the service is unhealthy (5xx, network failures) should open the breaker. A 400 means your request is wrong; a 401/403 means credentials; a 429 means "you're sending too fast, I'm healthy". Counting those would open the breaker during a perfectly healthy period and block every flag because of one bad request. Getting this classification right is what interviewers look for.
Every state transition is logged at WARN, and breaker, retry and rate-limiter metrics are bound to Micrometer (TaggedCircuitBreakerMetrics and friends), so an outage is visible on a dashboard. The Judge's Arena client has its own, separate retry (3 attempts from 500 ms) and breaker. Breakers are per dependency, so a Claude outage never blocks Arena calls, and the reverse.
5. The client-side rate limiter
RateLimiterConfig.custom()
.limitForPeriod(120) // 120 calls...
.limitRefreshPeriod(Duration.ofMinutes(1)) // ...per minute
.timeoutDuration(Duration.ofSeconds(60)) // wait up to 60 s for a permit, else RequestNotPermitted
The provider enforces quotas and answers 429 when you exceed them. A client-side limiter keeps you under the quota proactively, smoothing bursts instead of bouncing off the server. One AnthropicResilience instance is shared by all three listener threads and the evaluation runner, so they share one budget. It's conceptually a token bucket: permits refill each period, and callers wait for a permit or give up after the timeout.
6. Composing them: the decorator order
Supplier<Message> call = () -> send(params); // the HTTP call
Supplier<Message> limited = RateLimiter.decorateSupplier(rateLimiter, call); // innermost
Supplier<Message> guarded = CircuitBreaker.decorateSupplier(circuitBreaker, limited);
Supplier<Message> retried = Retry.decorateSupplier(retry, guarded); // outermost
try {
return retried.get();
} catch (BadRequestException e) {
throw new NonRetryableJudgeException("anthropic rejected request", e); // → DLT
} catch (CallNotPermittedException | RequestNotPermitted | AnthropicServiceException | AnthropicIoException e) {
throw new LlmUnavailableException(e); // → hold and retry later
}
Retry ─► CircuitBreaker ─► RateLimiter ─► HTTP call to Claude (outer) (inner) every retry attempt passes through the breaker and the limiter
- Retry outermost: each attempt is a separate trip through the breaker (so the breaker counts every failed attempt and can open mid-retry) and through the limiter (so retries consume rate budget like any other call and can't sneak around it).
- When the breaker is open,
CallNotPermittedExceptionis not in the retryable set, so the retry gives up immediately instead of waiting through backoffs for a dependency known to be down. The gateway turns it intoLlmUnavailableException. - If the breaker were outside the retry, it would see only the final outcome of a whole retry sequence, react slower, and miss individual failures.
The gateway's job ends with exception translation: callers see exactly two outcomes besides success. "Your request is wrong, give up" (NonRetryableJudgeException) or "the service is unavailable, try later" (LlmUnavailableException). The Kafka layer acts on precisely that distinction.
7. Two retry layers with different time scales
| Layer | Time scale | Handles | Mechanism |
|---|---|---|---|
| In-call (Resilience4j) | Seconds (up to 4 attempts) | Blips: a single 429, a 529, a reset connection | Retry + breaker + limiter |
| Per-message (Spring Kafka) | Indefinitely, every 5 s | Outages: Claude or Arena down for minutes, the breaker open | DefaultErrorHandler + FixedBackOff(5000, UNLIMITED), offset not committed |
The in-call layer smooths small hiccups without bothering Kafka. When it gives up (LlmUnavailableException, ArenaUnavailableException), the listener throws, the Kafka error handler re-seeks the same record and tries the whole flag again 5 seconds later, forever, until the dependency recovers. No flag is lost and none is dead-lettered because of an outage. Only NonRetryableJudgeException (a malformed flag, a Claude 400) goes to suspicious-players.judge.DLT.
Blocking retries hold up the rest of the partition (head-of-line blocking). Here that's desirable: during an outage every flag would fail anyway, and flags are kept in order. The consumer also keeps polling within max.poll.interval.ms, so it isn't kicked out of the group. Monitor lag so a long outage is visible.
8. Idempotency: making "at least once" safe
Retries and redeliveries mean the same operation can arrive more than once. The fix isn't to prevent that (impossible across systems) but to make repeats harmless. Your projects use every common technique:
| Technique | Where |
|---|---|
| A client-supplied idempotency key + unique constraint | Arena POST /matches with externalId (201 first, 200 on replay, and a concurrent duplicate answered with the winner) |
| Dedup store of processed ids | Judge ProcessedFlags, rebuilt from the compacted judge-verdicts topic: hasFlag(flagId) means skip |
| Naturally idempotent operations | Redis ZADD with an absolute score; compacted topics (re-writing the latest value) |
| State checks before acting | Judge pre-checks (already BANNED → skip); Arena's state machine rejects illegal repeats (409); the applier re-reads and re-decides |
| Transactions (Kafka-internal) | Kafka Streams exactly_once_v2 |
| Producer idempotence | Kafka producers: broker-side dedup of retried batches |
"Exactly-once delivery across systems isn't achievable, but exactly-once effects are: deliver at least once and make every handler idempotent."
Order of side effects matters too. The Judge publishes its verdict only after Arena accepted the action. If it crashes in between, the flag is redelivered; the pre-check sees BANNED and records a skip, or Arena answers 409 and the policy re-runs. The worst case is a duplicated NOTE in the audit trail. Publishing the verdict first would risk a recorded verdict whose action never happened.
9. Graceful degradation: what happens when X is down
| Outage | Behaviour |
|---|---|
| Redis (Arena) | Rating and ban updates still commit to PostgreSQL; the Redis sync failure is caught, logged and counted; leaderboard reads degrade; startup rebuilds Redis from PostgreSQL |
| Arena (seen from the bridge) | Exponential retries (1, 2, 4 s), then match-summaries.DLT; detection keeps running; matches can be replayed from the DLT |
| Arena (seen from the Judge) | Flags held in Kafka and retried every 5 s; none lost |
| Claude API | Breaker opens; flags held and retried; when it recovers, all are judged (proved by ResilienceIT) |
| One detector instance | Partitions move to survivors; exactly-once preserved; ~14.4 s failover, then catch-up |
| Bad input data | Dead-letter topics with diagnostics; the stream continues |
| Missing configuration or secret | The app refuses to start, naming the missing setting (fail fast) |
10. Proving it: fault injection and chaos
Resilience code that has never seen a failure is a guess. Both streaming projects inject failures on purpose.
Judge: ResilienceIT with a WireMock container (measured)
| Injected fault | Result |
|---|---|
429 with retry-after, twice | Retried; verdict produced after 3 calls |
| 529 overloaded, twice | Retried; verdict produced after 3 calls |
| A slow answer beyond the request timeout | Timeout, retried, verdict produced |
| Connection reset | Retried, verdict produced |
| 400 invalid request | Dead-letter topic at once; no retry, no verdict |
| 500 on every call for 20 s while 20 flags are sent | The breaker opens; no flag gets a verdict or reaches the DLT during the outage; afterwards all 20 get a BAN verdict |
WireMock runs as a container (wiremock/wiremock:3.13.2) with stateful scenarios ("fail twice, then succeed"), and the Judge's Anthropic client is pointed at it through ANTHROPIC_BASE_URL, read from Spring's Environment, which is exactly why AnthropicConfig doesn't use the SDK's fromEnv().
Telemetry: the chaos kill (measured)
scripts/chaos-kill.sh runs docker kill on one of three detector replicas about 90 s into a run, prints how long the survivors took to settle, and shows all 6 partitions owned again. Result: zero lost or duplicated events, 14.4 s failover (target missed), lag drained in about 2 minutes. The bridge's retry and DLT behaviour is tested against a stub Arena (StubArenaServer) in ArenaBridgeIT.
11. Observability for reliability
- Metrics: retry attempts, breaker state and failure rate, rate-limiter waits (Judge);
telemetry.bridge.dead_lettered;arena.leaderboard.sync.failures; consumer lag. - Logs: breaker transitions at WARN; Anthropic request ids (for support) on 400s; never secrets or prompt text.
- Alerts you'd set: breaker open for more than N minutes; DLT count rising; lag growing; sync failures > 0.
12. Interview questions
Which errors do you retry? Basic
Transient ones only: timeouts and connection errors, 429 (honouring Retry-After), and 5xx including 529 overloaded. Not 400/404/409/422, where the same request gives the same answer, and not malformed messages, which go to a dead-letter topic. A 401 gets one re-login and one retry. And only operations that are idempotent or made idempotent.
Why exponential backoff with jitter? Basic
Backoff gives a struggling service progressively more room instead of hammering it. Jitter randomises each wait so clients that failed together don't retry in lockstep and recreate the spike. My Claude calls use 1 s × 2 per attempt, ±50% randomisation, capped at 20 s, at most 4 attempts, and never less than the server's Retry-After (itself capped at 60 s).
Explain the circuit breaker states and your settings. Intermediate
CLOSED: calls flow and results are recorded in a sliding window of the last 20 calls. Once at least 10 calls are recorded and the failure rate reaches 50%, it goes OPEN and rejects calls instantly for 30 s. Then HALF_OPEN lets 3 trial calls through: if they succeed it closes, otherwise it reopens. It protects my threads from waiting on a dead service, and the service from a flood while it's trying to recover.
Why don't 429 or 400 count as breaker failures? Deep
The breaker should measure service health. 5xx and network failures mean it's unhealthy. A 400 means my request is wrong; a 401/403 means a credential issue; a 429 means the service is healthy and asking me to slow down, which the retry with Retry-After and the rate limiter handle. Counting those would open the breaker during healthy periods, blocking every flag because of one bad request or a burst.
In what order did you compose the decorators, and why? Deep
Retry outermost, then circuit breaker, then rate limiter, then the call. Every retry attempt passes through the breaker, so it sees each failure and can open mid-sequence, and through the limiter, so retries consume rate budget. When the breaker is open, its rejection isn't retryable, so we fail fast instead of backing off against a known-down service. With the breaker outside the retry, it would only see final outcomes and react slowly.
Why maxRetries(0) on the SDK client? Intermediate
To avoid retry amplification. The SDK retries by default; with Resilience4j retrying too, attempts multiply (the README puts it at up to 12 per failure), hitting the provider hardest exactly when it's struggling. It would also hide attempts from my breaker and metrics. Retrying in one layer keeps behaviour predictable and observable.
What happens to flags when the Claude API is down for 10 minutes? Intermediate
The first calls fail with 5xx or network errors; after a few, the breaker opens and further calls fail fast. The gateway throws LlmUnavailableException; the listener propagates it; Spring Kafka's DefaultErrorHandler doesn't commit the offset and redelivers the same flag every 5 seconds, indefinitely. Nothing goes to the dead-letter topic. When the API recovers, the half-open trial succeeds, the breaker closes and flags are judged in order. ResilienceIT verifies this with a 20-second run of 500s: 20 of 20 judged afterwards, none dead-lettered.
What's a poison pill and how is it different from an outage? Intermediate
A poison pill is a message that will always fail, like malformed JSON or a request the provider rejects as invalid. Retrying it blocks its partition forever. An outage is a temporary problem in a dependency that affects all messages and will end. Poison goes straight to a DLT (non-retryable exceptions); outages are retried. Misclassifying either way is bad: dead-lettering during an outage loses work, and retrying poison stalls the pipeline.
How do you get exactly-once effects across services? Deep
Deliver at least once and make handlers idempotent: idempotency keys with unique constraints (externalId), dedup stores (processed flag ids replayed from a compacted topic), naturally idempotent writes (ZADD with absolute scores), and state checks (already banned → skip; 409 → re-read). Order side effects so a crash leaves a safe state: the Judge records its verdict only after Arena accepted the action.
How did you test that the resilience actually works? Intermediate
Fault injection. A WireMock container impersonates Claude and Arena in ResilienceIT, returning 429 with retry-after, 529, slow responses, connection resets, a 400, and 20 seconds of 500s, with assertions on call counts, verdicts and the DLT. In Telemetry, a chaos script kills a detector replica mid-run, and the verifier proves no event was lost or duplicated. I also measured failover time and reported the miss.
Isn't retrying forever dangerous? Curveball
It can be: it blocks the partition and could hide a permanent failure misclassified as transient. I mitigate by classifying strictly (only availability errors are retried forever; anything request-specific is dead-lettered), keeping the in-call retry short, staying within max.poll.interval.ms, and making the condition visible through breaker metrics and consumer lag. For flags, holding is the right trade-off: losing a cheater flag is worse than delay.
Why not a bulkhead? Curveball
A bulkhead isolates resources (thread pools, connection pools) per dependency so one slow dependency can't exhaust everything. In the Judge, concurrency is already bounded by the Kafka listener (3 consumer threads) and the shared rate limiter, and the Claude and Arena clients have separate breakers. If the Judge served other traffic too, such as an HTTP API, I'd add semaphore bulkheads per dependency.
13. Quick quiz
1. Which should NOT be retried?
The same request gives the same answer.
2. Jitter exists to:
Randomised waits spread the load.
3. After 30 s in OPEN, the breaker moves to:
The trials decide whether it closes or reopens.
4. With retry outermost, when the breaker is open:
Then the Kafka layer retries the whole flag every 5 s.
5. Which makes a repeated POST /matches harmless?
Idempotency, not resilience decorators, prevents duplicates.
6. Why was the SDK's own retry disabled?
Retry in one layer.
14. Cheat sheet
- Timeouts everywhere: Claude 60 s; Arena 2 s connect / 5 s read; Kafka delivery 120 s.
- Retry only transient (I/O, 429, 5xx/529) + only idempotent operations. 400/404/409/422 and poison → no retry.
- Backoff 1 s ×2, ±50% jitter, cap 20 s, 4 attempts; honour
Retry-After(cap 60 s). - One retry layer: SDK
maxRetries(0). - Breaker: count window 20, min 10, 50%, open 30 s, half-open 3; record 5xx/IO; ignore 400/401/403/429.
- Rate limiter: 120/min, wait up to 60 s, shared.
- Order: Retry( Breaker( Limiter( call ))) → translate to
NonRetryable(DLT) orUnavailable(hold). - Two layers: in-call seconds + Kafka every 5 s forever (offset uncommitted).
- Idempotency: keys + unique constraints, dedup stores, natural idempotency, state checks, Kafka transactions. Act, then record.
- Degrade gracefully: Redis down → still commit; Arena or Claude down → hold; bad data → DLT; bad config → refuse to start.
- Prove it:
ResilienceIT(WireMock: 429, 529, slow, reset, 400, 20 s of 500s → 20/20) and the chaos kill (0 lost, 0 duplicated, 14.4 s).