Fairplay Interview Prep
Module 16

What could have gone wrong, the decisions that mattered, the struggles, and what to know earlier

Senior interviewers rarely ask "what did you build?". They ask "what went wrong?", "what would you change?" and "what was the hardest part?". This module answers those questions in depth for all three projects. It catalogues the ways each system could fail, the decisions that changed the outcome, where the build ran into real trouble, the lessons worth knowing on day one, and what is still wrong today.

P1P2P3· about 2.5 hours; reread before every interview
Where this comes from

Everything here is drawn from the projects' own records: the handoff.md and todo.md files ("things learned", "deviations from the spec", "known risks"), the // DECISION: comments in the code, the bug fixes listed there, the measured reports in reports/, and a full read of the code for this guide. That read surfaced two issues no note mentions yet (section 2.1 and 2.9). The "how to tell it" boxes use first person. Adjust them so every sentence is true for you, and only claim what you can explain line by line.

By the end you can answer

  • "What was the hardest bug, and how did you find it?"
  • "What could go wrong in this system, and what stops it?"
  • "Which decision mattered most? Which would you reverse?"
  • "What do you wish you'd known before starting?"
  • "What's still broken or unfinished?"
  • "Tell me about a time a test, a number or a tool misled you."

1. A map of failure: eleven ways systems like these break

When you're asked "what could go wrong?", don't list random bugs. Walk a taxonomy. That shows you think about failure systematically:

1. Concurrency

Races, lost updates, deadlocks, check-then-act.

2. Consistency across stores

Two systems written without one transaction.

3. Delivery semantics

Duplicates, losses, side effects outside the transaction.

4. Contracts and evolution

Renamed fields, new enum values, changed partition counts.

5. Configuration and startup

Precedence, missing secrets, empty databases, wiring.

6. Time

Clocks, event vs wall time, expiry, windows, grace.

7. Performance and capacity

Wrong data structure, hidden network hops, pools, memory.

8. Security

Access control, secrets, injection, abuse.

9. Measurement

Numbers that look right but measure the wrong thing.

10. Tooling and environment

OS, shells, Docker, line endings, build plugins.

11. AI-specific

Unverified APIs, output drift, injection, cost blow-ups.

2. Everything that could have gone wrong, category by category

For each risk: what would happen, what prevents it (or doesn't), and the proof. Guarded means code plus a test; Open means a known gap.

2.1 Concurrency

RiskWhat would happenWhat stops itStatus
Two matches with the same player update the rating at once (Arena)Lost update: one rating change disappearsSELECT ... FOR UPDATE on all participants inside the transactionGuarded
Two matches lock overlapping players in different ordersDeadlock; PostgreSQL aborts one after about 1 sSort UUIDs in Java + ORDER BY id in the locking queryGuarded
Two identical POST /matches race (a bridge retry)Both pass "does it exist?"; the loser's insert violates the unique key, which used to surface as a 500Unique constraint + TransactionTemplate; catch after rollback, return the winner (200). Test: concurrentRequestsWithSameExternalIdCreateOneMatchAndReplayTheOther. The notes admit whether a given run hits the violation path depends on timingGuarded (fixed during the build)
Two moderators (or a moderator and the Judge) ban at the same instantDouble audit rows, conflicting state@Version + the state machine: exactly one wins; the other gets 409. Test: parallelBansGiveExactlyOneSuccessAndOneConflictGuarded
Two registrations of "Alice" and "alice" at onceDuplicate accountsUnique expression index on lower(username); the violation is mapped to 409 by constraint nameGuarded
Two concurrent refreshes with the same refresh token (Arena)AuthService.refresh reads the token without a lock, and RefreshToken has no @Version. Under Read Committed both requests see "not revoked", both issue a new pair, and the second UPDATE simply overwrites replaced_by. Result: two valid token families from one token, and reuse detection never fires. An attacker racing the real user could keep a parallel sessionNothing yet. Fixes: SELECT ... FOR UPDATE on the token row (a @Lock query), or an atomic UPDATE refresh_tokens SET revoked_at = :now WHERE id = :id AND revoked_at IS NULL and treat "0 rows updated" as reuse, or add @VersionOpen (found while writing this guide)
A shard thread's periodic task throws (Telemetry)scheduleAtFixedRate silently cancels all future ticks; the shard's players stop producingShardWorker.run() catches Throwable. Test: aFailingTickIsLoggedAndDoesNotEscapeSoTheScheduleSurvivesGuarded
Producer callbacks and shard threads update counters togetherLost counts in the manifest; the integrity proof would be wrongLongAdder / AtomicLong in ManifestTracker; thread confinement elsewhereGuarded
Haiku and Sonnet call the same tool for one flagDuplicate Arena readsMemoizingToolExecutor.computeIfAbsent, one instance per flagGuarded

2.2 Consistency across stores and services

RiskWhat would happenWhat stops itStatus
Redis updated inside a transaction that then rolls backThe leaderboard shows a rating that never existed@TransactionalEventListener(AFTER_COMMIT)Guarded
Crash between the database commit and the Redis updateA stale or missing leaderboard entryStartup rebuild if counts differ; a failure counterPartly open: missing members repaired; stale scores are not (the rebuild compares counts only)
Two after-commit listeners for one player run out of orderRedis keeps the older ratingNothing; the window is tinyOpen (fix: version-guarded Lua update or an outbox)
The Judge crashes after banning in Arena but before publishing its verdictThe flag is redelivered and judged againPre-check sees BANNED and skips; Arena's 409 path; the worst case is a duplicate NOTEGuarded by design
The verdict is published first, then the Arena call failsA recorded verdict whose action never happenedThe order is act, then record (FlagListener publishes after JudgeService returns)Guarded
The bridge replays a match after a timeoutRatings applied twiceArena's externalId idempotency (200 on replay)Guarded

2.3 Delivery semantics and "exactly once"

RiskWhat would happenWhat stops itStatus
A detector instance dies mid-transactionDuplicated or lost counts and flagsexactly_once_v2; consumers use read_committed. Measured: 0 lost and 0 duplicated across a kill (7,466,329 events)Guarded and measured
Processor side effects run twice after a retried transactionMicrometer counters (parsed, flagged) over-countDocumented in TopologyMetrics: counters are dashboard-only; exact numbers come from committed topicsKnown and documented
A consumer reads aborted transactional recordsPhantom flags that look like duplicatesread_committed everywhere; TopicSnapshotReaderIT.neverShowsRecordsOfAnAbortedTransactionGuarded
The integrity checker itself is broken and always says "0 duplicates"False confidence in exactly-onceA negative control in ExactlyOnceIT (one deliberate duplicate must be detected)Guarded
A failed send hides itselfIntegrity passes despite a missing eventfinalSeq = generated seq, not highest acked; the verifier also requires failed == 0Guarded

2.4 Contracts and evolution

RiskWhat would happenWhat stops itStatus
Telemetry renames skillTierThe Judge silently reads null (unknown fields are ignored), so the HIGH_TIER_BAN escalation never fires and a PRO could be auto-banned on Haiku's verdict aloneOnly convention ("contracts are frozen"); a schema registry would enforce itOpen (convention only)
A flag arrives with a numeric field missingJackson 3 can't put null into a double (FAIL_ON_NULL_FOR_PRIMITIVES), so the flag counts as malformed and goes to the DLTDocumented gotcha; real flags carry every fieldKnown
Arena adds a new account statusPlayerStatus.valueOf would throwPreChecks.statusOf catches it and treats the player as unknown (skip, no crash)Guarded
Someone adds partitions to a player-keyed topicKeys remap: ordering breaks, and the stream-table join silently stops matching (co-partitioning broken)Topics.PLAYER_PARTITIONS constant; the end-to-end IT asserts the repartition topic has 6 partitions like profilesPartly guarded
A code change shifts auto-generated internal topic namesA redeploy abandons in-flight repartition dataThe repartition step is explicitly named flags; stores are namedGuarded

2.5 Configuration, startup and wiring

RiskWhat happened or would happenFixStatus
Overriding one element of a YAML list in testsHappened. Boot replaced the whole arena.accounts.services list; usernames became null; all 9 integration test classes failed to loadRepeat the usernames; correct the spec's snippetFixed
An environment variable silently overrides YAMLHappened. The shared .env's SERVER_PORT=8082 moved the simulator off 8081Set it explicitly for the simulator in ComposeFixed
An aggregate over an empty table returns NULLHappened. percentile_cont returned NULL, JdbcClient rejected a null result, and Arena crashed at startup on a fresh database, so a clean docker compose up would have failedReturn Optional from the extractorFixed
A bean with two constructorsHappened. The verifier couldn't start inside Spring, despite 100% unit coverage@Autowired on one; VerifierContextTestFixed
Shared config class outside the component scanTopics never created@Import(TopicsConfig.class)Guarded
A missing or blank secretA confusing crypto or 401 error at first useStartup validation that names the missing setting without printing values (all three projects)Guarded
An auto-ban threshold configured below Arena's floorThe Judge would send bans Arena rejectsJudgeSettings refuses to start with less than 0.80Guarded

2.6 Time

2.7 Performance and capacity

RiskWhat happened or would happenOutcome
Wrong data structure for rankThe database rank is O(N): 18 s p95 at 1M playersRedis sorted set: 63.5 ms p95, measured
A hidden repartition per windowThe spec's topology would have pushed ~24,000 events/s through the broker twice moreCaught by printing topology.describe(); fixed with a key-preserving re-parse; guarded by a shape test
Disabling the state-store cacheEvery update went to the changelog; the detector fell behind; median detection latency rose from ~3 s to ~30 sKeep the 64 MB cache; measured in the no-cache run
Unlimited JVM memory in containersThree detector JVMs stalled a 16 GB laptop until Docker had to be restartedmem_limit: 1g
Too little headroom for failoverTwo survivors had ~5,000 events/s spare; lag peaked near 380,000 and took ~2 min to drainReported; fix is capacity, not code
Pool saturation200 virtual users on 20 connections: database latency rose from 1.5 s server-side to 15 s client-sideExplained in the README
save() on entities with assigned ids and no @VersionSpring Data calls merge, which SELECTs before every INSERT (Match, MatchParticipant, ModerationAction, RefreshToken; participants cost one extra SELECT per player in every recorded match)Open; fix with Persistable (module 07)
The Judge's ProcessedFlags never evictsEvery flag id ever judged stays in memory, and startup replays the whole compacted verdicts topic. Memory and startup time grow foreverOpen; fine at portfolio scale, a real problem at scale (module 18)

2.8 Security

RiskStatus and mitigation
Broken access control (editing another player)"me" endpoints take the id from the token; role and business-rule checks
Passwords longer than bcrypt's 72-byte limitRequest validation caps passwords at 72 characters (multi-byte characters could still exceed 72 bytes)
LIKE wildcard injection in player search!, % and _ escaped with an escape '!' clause
Path traversal through the prompt version[A-Za-z0-9._-]+ enforced at startup
Username enumeration by timing on loginOpen: no dummy bcrypt for unknown users
Brute force on login and appealsOpen: no rate limiting (on Arena's next-steps list)
Refresh-token race (2.1)Open
Prompt injection through player-controlled textUsernames reach the model in flags and the profile tool; past moderation reasons (written by humans, or by earlier Judge runs, so model output fed back as input) reach it through the history tool, truncated to 200 characters. Mitigated structurally: read-only guarded tools, schema + validator, deterministic policy, 0.90 threshold, Arena's own floor, audit and appeals. The model can be fooled into a wrong verdict, never into an action
Secrets in images or logs.dockerignore; env-only secrets; startup checks print names, never values; logs never contain prompts, tool results or flag text (tested)
Actuator exposurePrometheus endpoint unauthenticated by design for local scraping; documented in the README

2.9 Measurement: numbers that could mislead

2.10 Tooling and environment

2.11 AI-specific risks (the Judge)

3. The decisions that mattered most

For each: the alternatives, why this one, and what it cost. Being able to argue against your own choice is what makes it a decision rather than a default.

#DecisionAlternativesWhyCost / would I change it?
1Write a full implementation spec first, with frozen contracts and a step-by-step build order with tests per stepCode first, discover the designThree services must agree on APIs, topics and JSON; acceptance criteria and "how each number is measured" are defined before any codeThe specs contained wrong snippets (SDK calls, the topology, test properties). Deviations had to be documented. Keep it.
2PostgreSQL as truth, Redis as a derived index, synced after commitRedis only; DB only; outboxO(log N) ranks without risking truth; Redis is disposableIn-memory events can be lost; the outbox is the next step
3Pessimistic, sorted locking for matchesOptimistic + retry; atomic SQL updatesCollisions are likely; retrying a 100-player match is expensive; sorting makes deadlocks impossibleLocks serialise hot players. Keep.
4The audit log enforced by a database triggerApplication discipline onlyBugs can't rewrite historyDoesn't stop TRUNCATE or a DBA; add role permissions
5A pure state machine and a pure policy tableRules inside servicesExhaustive tests (35 pairs; every policy row)None. The best decision in Arena.
6HS256 JWT + 15-minute access + rotating hashed refresh tokens with reuse detectionSessions; long-lived JWTs; RS256Stateless scaling; revocation through the refresh tokenThe refresh path needs a row lock (2.1). RS256 if verification spreads
7Parse inside the topology (raw bytes → processor → DLT)Deserialization exception handlerBad records become dead letters with offsets, and the stream livesEach event parsed twice (decision 8). Keep.
8Re-parse with mapValues instead of grouping process() outputFollow the spec (two repartition topics)Avoids rewriting ~24,000 events/s through the brokerDouble JSON parsing. Clearly the right trade.
9Hopping 5 s/1 s windows that emit on every update, plus a wall-clock cooldownTumbling; suppress until close; sliding windowsDetection about 1 s after crossing a threshold; no boundary missesRepeat emissions to de-duplicate; replay semantics are odd (2.6)
10Prove exactly-once with per-player sequence numbers and a negative controlCompare totals; trust the configurationCatches loss + duplicate pairs; a test that can failRe-emitting all players every 5 s doesn't scale to millions (module 18)
11Standbys + a 6 s session timeoutDefaults (45 s)Fast detection, warm stateRebalances still cost ~9 s; target missed. Tune further
12No agent frameworkLangChain4j, Spring AIFull visibility, testability and cost accountingMore code to own; provider lock-in. Keep for this scope.
13The model recommends, code actsLet the model call action toolsBounded blast radius; injection-resistantLess flexible. Non-negotiable.
14Cheap model first, strong model fresh on escalation, shared tool cacheOne model; reviewer sees the first answerCost plus an independent second opinionTwo models to evaluate
15The Kafka topic as the Judge's only stateA databaseOne less system; replayable; idempotentUnbounded replay and memory (2.7). At scale add a TTL'd store
16Move ArenaToolExecutor and ArenaReader into coreKeep them in the service (the spec)The evaluation records snapshots with the exact production tool codeA departure from the spec, documented. Keep.
17Testcontainers over mocks for infrastructure; coverage gatesH2, embedded Kafka, mocksFound real bugs mocks couldn'tSlower builds, Docker required. Keep.
18Report measured numbers including missesReport only successesCredibilitySome uncomfortable README rows. Always keep.

4. Where the build struggled most

Ranked by how much each cost, according to the notes and fixes. For each: what happened, how it was diagnosed, how it was solved, and the lesson.

1. Brand-new framework versions (Boot 4, Jackson 3, Kafka 4, Testcontainers 2, SDK 3.10)

What happened: most examples online, and parts of the specs, were written for older versions. Imports didn't resolve (@MockBean, Jackson 2 packages, the health API, AutoConfigureMockMvc's package), Kafka Streams' transform family was off-limits, Kafka turned out to be 4.2.2 rather than 4.1.x, and the Anthropic SDK lacked the builders the spec used. Diagnosis: reading the dependency jars (javap), release notes, and dependency:tree. Solution: a pitfalls table in each spec and CLAUDE.md; never downgrade, find where the class moved. Lesson: on new major versions, the primary sources (javadoc, the jars, release notes) beat tutorials, and a five-minute smoke test of each critical API up front saves hours.

2. Bugs that only appear when everything runs for real

What happened: the first full ./mvnw verify with Docker found two production bugs and two wrong tests in Arena (the empty-database crash; the list-override that broke all integration tests; a 403 test sending an invalid body; a 405 test that was unauthenticated). Telemetry's verifier had 100% coverage and couldn't start. Diagnosis: running the real stack and the full suite, not just unit tests. Solution: fixes plus tests that start the real context. Lesson: run the whole thing early and often; coverage measures execution, not correctness.

3. Kafka Streams' hidden behaviour

What happened: the repartition trap (found by printing topology.describe()); flags emitted on every window update, so a cooldown was needed; with the 64 MB cache, integration tests saw a different "first" window than TopologyTestDriver; failover took three rebalances. Solution: shape tests, a cooldown store, ITs that don't assert exact evidence, and an honest failover report with a timeline from logs. Lesson: with stream processors, inspect what the framework derives (topology, internal topics, emission timing), not just your code.

4. Running a distributed system on one laptop

What happened: memory stalls that needed a Docker restart; a port hijacked by a shared .env; containers left behind by down -v; Git Bash path rewriting that broke the first chaos report; lag metrics that lag. Solution: memory limits, explicit per-service settings, named profiles, MSYS_NO_PATHCONV=1, and the authoritative lag command in the measured-run script. Lesson: automate the measured run end to end (measured-run.sh) so every run is identical, and keep the raw outputs as evidence.

5. Results that didn't cooperate

What happened: failover missed its 10 s target (14.4 s); lag didn't stay flat; the at-least-once comparison showed nothing; the Judge's evaluation couldn't run without an API key and a special Telemetry run. Solution: report exactly what was measured, try harder where the spec asked (the no-cache rerun), label the Judge's figures as imaginary, and document how to produce the real ones. Lesson: the credibility of every other number depends on how you handle the ones that missed.

6. Time and concurrency in tests

What happened: the calendar-dependent token test; a concurrency test whose interesting path depends on timing. Solution: a fixed clock everywhere, including library validators; latch-synchronised threads; invariants that hold whichever path runs. Lesson: nondeterminism hides in time, threads and ordering, so design tests around invariants.

7. Uncertainty at the LLM boundary

What happened: SDK differences from the spec, strict-schema limitations, two Jackson major versions in one process, and an API shape that can only be confirmed live. Solution: an interface you own (MessagesGateway), a smoke test written and ready, a string bridge between Jackson 2 and 3, and the range check in the guard. Lesson: wrap third-party AI APIs behind your own seam on day one, and verify the request shape before building on it.

How to tell "the hardest part" (pick one and adapt)

"The hardest part was that the bugs that mattered only appeared when everything ran for real. My unit tests were green, but the first full run with real PostgreSQL crashed Arena on an empty database: percentile_cont returns NULL with no rows, and Spring's JdbcClient rejects a null result, so the app died at startup. The same run showed that overriding one element of a YAML list in tests replaces the whole list. Since then I keep at least one test that starts the real context against real containers, and I run the full stack early instead of at the end."

5. What I wish I knew earlier: 30 lessons

Frameworks and versions

  1. New major versions move classes. When an import fails, look in the jar before searching the web.
  2. Boot replaces lists from higher-priority sources; it doesn't merge them.
  3. Environment variables beat YAML, and a shared .env reaches every service.
  4. A bean with two constructors needs @Autowired on one, and only a context test will tell you.
  5. Jackson 3 fails on missing primitive fields; use wrapper types when absence is legal.

Data

  1. Aggregates over empty sets return NULL. Design startup code for an empty world.
  2. In PostgreSQL an error aborts the whole transaction. Catch constraint violations outside it.
  3. Check-then-act is never enough; the unique constraint or row lock is the real guarantee. This includes refresh tokens.
  4. Pick the data structure for the query (rank needs an order-statistic structure, not an index).
  5. save() with assigned ids means merge unless the entity tells Spring it's new.
  6. Define metrics precisely (an average of per-match averages is not an average).

Kafka and streaming

  1. KStream#process output counts as key-changing; check topology.describe() for unplanned repartition topics.
  2. Windowed aggregations emit on every update; plan de-duplication.
  3. Exactly-once ends at Kafka's edge; metrics and HTTP calls inside processors can run twice.
  4. Failover = detection + rebalances; the session timeout is only the first part.
  5. Lag metrics on clients are windowed maxima; trust kafka-consumer-groups.sh.
  6. Advertised listeners decide what clients connect to; one per network audience in Docker.
  7. Leave headroom: N−1 instances must absorb the full load plus a backlog.

LLM engineering

  1. Verify the provider's real request shape first with one live call, before building the loop around it.
  2. Turn off SDK retries when you add your own.
  3. Strict JSON schemas have limits; validate again in code.
  4. Read requirements against definitions: "confidence" meant probability of cheating, which changed the escalation rule.
  5. Anything the model reads is an injection surface, including your own earlier outputs stored as moderation reasons.
  6. Evaluation needs data you can't fake. Plan the dataset run (and the API budget) at the start.

Testing, operations and process

  1. Every time-aware component must use the injected clock, including library validators.
  2. Always set JVM container memory limits.
  3. On Windows, Git Bash rewrites paths; set MSYS_NO_PATHCONV=1 in scripts that call into containers.
  4. Surefire ANDs groups and excludedGroups; make exclusions overridable properties.
  5. Write a negative control for every checker, otherwise "0 errors" proves nothing.
  6. Automate measured runs and keep raw outputs; they're your evidence in interviews.

6. What is still wrong or unfinished today

Know this list cold. Volunteering an item before being asked is a strong signal.

ProjectOpen itemEffort to fix
ArenaFull ./mvnw verify not re-run since the token-test fixMinutes (start Docker)
ArenaRefresh-token race (2.1)Small: a locking query or a conditional UPDATE, plus a latch-based test
ArenaNothing ever deletes expired or revoked refresh tokens: every login and refresh adds a row to refresh_tokens forever (module 18 shows what that means at scale)Small: a scheduled cleanup of rows past expiry, or time-partitioning and dropping old partitions
ArenaWrong role + invalid body gives 400, not 403 (validation runs before @PreAuthorize)Duplicate role rules as URL rules; decide if worth it
ArenaThe rebuild compares counts, not scores; possible out-of-order Redis updatesReconciliation job, or a versioned Lua update, or an outbox
ArenaTagged counters appear on /actuator/prometheus only after first usePre-register them
ArenaNo rate limiting; login timing leakGateway or Bucket4j; dummy bcrypt
ArenaReaction average definition; placement-only ratingStore shot-weighted totals; consider Elo or Glicko
TelemetryFailover 14.4 s vs < 10 s; lag not flatRebalance tuning, new protocol, more headroom
TelemetryThe bridge never run against a real ArenaOne live run
TelemetrySingle broker, RF 1A three-broker cluster in Compose
TelemetryCI workflow written but never run on GitHubPush
JudgeLive smoke test never run; request shape unverifiedOne command with an API key (a fraction of a cent)
JudgeNo dataset, no evaluation reports, prompts v2–v6 not written; README shows imaginary valuesA Telemetry run with loosened thresholds, collect, four runs, iterate
JudgeLOW_CONFIDENCE reading undecidedA product decision, then one line
JudgeProcessedFlags grows foreverExpire entries older than the window; a TTL store at scale
PlatformNever run end to end liveFollow the Judge README's "Run the whole system"

7. Interview questions

What's the hardest bug you found? Intermediate

Pick from section 4 and use STAR. Strong candidates: the empty-database startup crash (NULL from percentile_cont rejected by JdbcClient), the concurrent duplicate externalId returning 500 (a PostgreSQL aborted transaction means you must catch after rollback), or the repartition trap (found with topology.describe()). Always end with what you changed in your process, not just the fix.

What could go wrong in Arena under load, and what stops it? Deep

Walk the concurrency table: lost rating updates (row locks), deadlocks (sorted lock order), duplicate matches (unique key + catch after rollback), double bans (@Version + state machine), duplicate usernames (unique expression index). Then name the open one: two concurrent refreshes of the same refresh token can both succeed because the token row isn't locked, which I'd fix with a conditional UPDATE that requires revoked_at IS NULL. Then capacity: pool saturation, and the database rank path.

Explain the refresh-token race and how you'd fix it. Deep

refresh() loads the token by hash, checks it isn't revoked, issues a new pair and marks the old one revoked, with no lock and no version column. Two requests with the same token can both pass the "not revoked" check under Read Committed; the second UPDATE waits for the first's row lock, then overwrites it, and both callers get valid new refresh tokens, so reuse detection never fires. Fixes: lock the row (@Lock(PESSIMISTIC_WRITE) on the lookup), or make revocation conditional (UPDATE ... SET revoked_at = now WHERE id = ? AND revoked_at IS NULL, and if 0 rows changed, treat it as reuse), or add @Version. I'd prove it with a latch-based test that fires two refreshes together and asserts exactly one succeeds.

Which decision would you reverse or revisit? Intermediate

Good honest answers: (1) the in-memory after-commit events for Redis: I'd move to a transactional outbox for guaranteed, ordered delivery; (2) the Judge's unbounded in-memory ProcessedFlags: I'd expire entries beyond the 30-minute window or use a TTL store; (3) running the integrity proof by re-emitting every player every 5 s, which is fine for 1,000 players and wasteful for millions. Explain why each was reasonable at the time: simplicity at portfolio scale.

Tell me about a time a test lied to you. Intermediate

Two good ones. The verifier had 100% line coverage but couldn't start, because Spring couldn't choose between two constructors and no unit test started the context. And TokenServiceTest passed only before a certain time of day, because the JWT decoder checked expiry against the real clock while the token used a fixed one. Lessons: keep a context test, and make every time-aware component use the injected clock.

Tell me about a time a tool or environment misled you. Basic

Git Bash on Windows rewrote /opt/kafka/bin/... into a Windows path when passed to docker compose exec, so the first chaos report's consumer-group check failed. The evidence is still in chaos-exactly_once_v2.txt. Setting MSYS_NO_PATHCONV=1 fixed it, and the next run printed the full group table. Also: client-side Kafka lag metrics are windowed maxima, so I switched to kafka-consumer-groups.sh for the authoritative lag.

Where could prompt injection reach your agent? Deep

Anywhere text the model reads is influenced by someone else: usernames in the flag and profile tool, and moderation reasons in the history tool, which include human moderators' notes and the Judge's own earlier explanations (model output becoming future input). I limit what a successful injection can do rather than trying to detect it: tools are read-only and restricted to the flagged player, output must pass a schema and a validator, a pure policy decides with a 0.90 bar for bans, Arena enforces its own floor and state machine, and everything is audited and appealable. Worst case is a wrong verdict a human can review, never an action.

What do you wish you'd known before starting? Basic

Pick three from section 5 that you can explain with a story. For example: "that exactly-once stops at Kafka's edge, so counters inside processors can double; that Kafka Streams treats process() output as key-changing, which nearly cost a repartition per window; and that I should verify an LLM provider's real request shape with one live call before building the loop around it."

Your Telemetry comparison didn't show what you expected. What does that tell you about your experiment design? Curveball

That one kill per run is a weak experiment: a duplicate under at-least-once requires the kill to land between a state write and the offset commit, a narrow window. A better design would run many automated kills per run, or inject the fault precisely at that point, and report the rate of duplicates across trials. What made the existing result trustworthy is the negative control: the checker provably detects duplicates, so the zeros mean "none happened", not "couldn't see them".

Is your Arena "average reaction time" correct? Curveball

It's consistent but not the true per-shot mean: Arena stores sum_reaction_ms as the sum of each match's average and divides by matches played, so every match weighs the same regardless of how many shots it had. The percentile distribution uses the same definition, so comparisons between players are fair, but the absolute number is biased toward short matches. Storing total shots and total reaction milliseconds and dividing would give the weighted mean.

What's the biggest technical risk left in the Judge? Intermediate

That the real API rejects the request shape: four strict tools plus output_config.format, and an optional property in a strict tool schema. Everything is tested against WireMock stubs that model my understanding of the API. One live smoke test, costing a fraction of a cent, would confirm or refute it, and it's the first thing I'd run.

What did the spec get wrong, and how did you handle it? Intermediate

Examples: Telemetry's topology snippet would have created two repartition topics; Arena's test-property snippet broke list binding; the Judge's SDK calls used builders that don't exist, its command for the live smoke test would have run nothing (Surefire ANDs groups with excludedGroups), and a literal reading of the confidence rule would have escalated every CLEAR. In each case I implemented the intent rather than the text, marked the spot with a DECISION comment, recorded it in the handoff notes, and added a test that guards the choice.

8. Cheat sheet

  • Failure taxonomy: concurrency, cross-store consistency, delivery, contracts, config, time, capacity, security, measurement, tooling, AI.
  • Fixed during the build: empty-DB crash; list override; duplicate-externalId 500; refresh check order; verifier constructors; token-test clock; SERVER_PORT; Git Bash paths; down -v leftovers; memory stalls; the repartition trap; the Surefire groups AND.
  • Found while writing this guide: the refresh-token race; ProcessedFlags unbounded; average-of-averages reaction; save() merge overhead.
  • Top decisions: spec-first contracts; truth vs derived Redis; sorted pessimistic locks; DB trigger; pure state machine and policy; parse-in-topology; mapValues re-parse; seq proof + negative control; no framework; model recommends, code acts; fresh Sonnet; topic as state; Testcontainers; honest numbers.
  • Struggles: version churn; real-run-only bugs; Streams internals; laptop-scale distributed ops; uncooperative results; time and concurrency in tests; LLM API uncertainty.
  • Still open: Arena verify, refresh race, 400-vs-403, score repair, rate limiting; Telemetry failover, real Arena run, RF 1, CI on GitHub; Judge live test, dataset and reports, confidence decision, unbounded state; platform E2E.
Saved in this browser.