What could have gone wrong, the decisions that mattered, the struggles, and what to know earlier
Senior interviewers rarely ask "what did you build?". They ask "what went wrong?", "what would you change?" and "what was the hardest part?". This module answers those questions in depth for all three projects. It catalogues the ways each system could fail, the decisions that changed the outcome, where the build ran into real trouble, the lessons worth knowing on day one, and what is still wrong today.
Everything here is drawn from the projects' own records: the handoff.md and todo.md files ("things learned", "deviations from the spec", "known risks"), the // DECISION: comments in the code, the bug fixes listed there, the measured reports in reports/, and a full read of the code for this guide. That read surfaced two issues no note mentions yet (section 2.1 and 2.9). The "how to tell it" boxes use first person. Adjust them so every sentence is true for you, and only claim what you can explain line by line.
By the end you can answer
- "What was the hardest bug, and how did you find it?"
- "What could go wrong in this system, and what stops it?"
- "Which decision mattered most? Which would you reverse?"
- "What do you wish you'd known before starting?"
- "What's still broken or unfinished?"
- "Tell me about a time a test, a number or a tool misled you."
1. A map of failure: eleven ways systems like these break
When you're asked "what could go wrong?", don't list random bugs. Walk a taxonomy. That shows you think about failure systematically:
1. Concurrency
Races, lost updates, deadlocks, check-then-act.
2. Consistency across stores
Two systems written without one transaction.
3. Delivery semantics
Duplicates, losses, side effects outside the transaction.
4. Contracts and evolution
Renamed fields, new enum values, changed partition counts.
5. Configuration and startup
Precedence, missing secrets, empty databases, wiring.
6. Time
Clocks, event vs wall time, expiry, windows, grace.
7. Performance and capacity
Wrong data structure, hidden network hops, pools, memory.
8. Security
Access control, secrets, injection, abuse.
9. Measurement
Numbers that look right but measure the wrong thing.
10. Tooling and environment
OS, shells, Docker, line endings, build plugins.
11. AI-specific
Unverified APIs, output drift, injection, cost blow-ups.
2. Everything that could have gone wrong, category by category
For each risk: what would happen, what prevents it (or doesn't), and the proof. Guarded means code plus a test; Open means a known gap.
2.1 Concurrency
| Risk | What would happen | What stops it | Status |
|---|---|---|---|
| Two matches with the same player update the rating at once (Arena) | Lost update: one rating change disappears | SELECT ... FOR UPDATE on all participants inside the transaction | Guarded |
| Two matches lock overlapping players in different orders | Deadlock; PostgreSQL aborts one after about 1 s | Sort UUIDs in Java + ORDER BY id in the locking query | Guarded |
Two identical POST /matches race (a bridge retry) | Both pass "does it exist?"; the loser's insert violates the unique key, which used to surface as a 500 | Unique constraint + TransactionTemplate; catch after rollback, return the winner (200). Test: concurrentRequestsWithSameExternalIdCreateOneMatchAndReplayTheOther. The notes admit whether a given run hits the violation path depends on timing | Guarded (fixed during the build) |
| Two moderators (or a moderator and the Judge) ban at the same instant | Double audit rows, conflicting state | @Version + the state machine: exactly one wins; the other gets 409. Test: parallelBansGiveExactlyOneSuccessAndOneConflict | Guarded |
| Two registrations of "Alice" and "alice" at once | Duplicate accounts | Unique expression index on lower(username); the violation is mapped to 409 by constraint name | Guarded |
| Two concurrent refreshes with the same refresh token (Arena) | AuthService.refresh reads the token without a lock, and RefreshToken has no @Version. Under Read Committed both requests see "not revoked", both issue a new pair, and the second UPDATE simply overwrites replaced_by. Result: two valid token families from one token, and reuse detection never fires. An attacker racing the real user could keep a parallel session | Nothing yet. Fixes: SELECT ... FOR UPDATE on the token row (a @Lock query), or an atomic UPDATE refresh_tokens SET revoked_at = :now WHERE id = :id AND revoked_at IS NULL and treat "0 rows updated" as reuse, or add @Version | Open (found while writing this guide) |
| A shard thread's periodic task throws (Telemetry) | scheduleAtFixedRate silently cancels all future ticks; the shard's players stop producing | ShardWorker.run() catches Throwable. Test: aFailingTickIsLoggedAndDoesNotEscapeSoTheScheduleSurvives | Guarded |
| Producer callbacks and shard threads update counters together | Lost counts in the manifest; the integrity proof would be wrong | LongAdder / AtomicLong in ManifestTracker; thread confinement elsewhere | Guarded |
| Haiku and Sonnet call the same tool for one flag | Duplicate Arena reads | MemoizingToolExecutor.computeIfAbsent, one instance per flag | Guarded |
2.2 Consistency across stores and services
| Risk | What would happen | What stops it | Status |
|---|---|---|---|
| Redis updated inside a transaction that then rolls back | The leaderboard shows a rating that never existed | @TransactionalEventListener(AFTER_COMMIT) | Guarded |
| Crash between the database commit and the Redis update | A stale or missing leaderboard entry | Startup rebuild if counts differ; a failure counter | Partly open: missing members repaired; stale scores are not (the rebuild compares counts only) |
| Two after-commit listeners for one player run out of order | Redis keeps the older rating | Nothing; the window is tiny | Open (fix: version-guarded Lua update or an outbox) |
| The Judge crashes after banning in Arena but before publishing its verdict | The flag is redelivered and judged again | Pre-check sees BANNED and skips; Arena's 409 path; the worst case is a duplicate NOTE | Guarded by design |
| The verdict is published first, then the Arena call fails | A recorded verdict whose action never happened | The order is act, then record (FlagListener publishes after JudgeService returns) | Guarded |
| The bridge replays a match after a timeout | Ratings applied twice | Arena's externalId idempotency (200 on replay) | Guarded |
2.3 Delivery semantics and "exactly once"
| Risk | What would happen | What stops it | Status |
|---|---|---|---|
| A detector instance dies mid-transaction | Duplicated or lost counts and flags | exactly_once_v2; consumers use read_committed. Measured: 0 lost and 0 duplicated across a kill (7,466,329 events) | Guarded and measured |
| Processor side effects run twice after a retried transaction | Micrometer counters (parsed, flagged) over-count | Documented in TopologyMetrics: counters are dashboard-only; exact numbers come from committed topics | Known and documented |
| A consumer reads aborted transactional records | Phantom flags that look like duplicates | read_committed everywhere; TopicSnapshotReaderIT.neverShowsRecordsOfAnAbortedTransaction | Guarded |
| The integrity checker itself is broken and always says "0 duplicates" | False confidence in exactly-once | A negative control in ExactlyOnceIT (one deliberate duplicate must be detected) | Guarded |
| A failed send hides itself | Integrity passes despite a missing event | finalSeq = generated seq, not highest acked; the verifier also requires failed == 0 | Guarded |
2.4 Contracts and evolution
| Risk | What would happen | What stops it | Status |
|---|---|---|---|
Telemetry renames skillTier | The Judge silently reads null (unknown fields are ignored), so the HIGH_TIER_BAN escalation never fires and a PRO could be auto-banned on Haiku's verdict alone | Only convention ("contracts are frozen"); a schema registry would enforce it | Open (convention only) |
| A flag arrives with a numeric field missing | Jackson 3 can't put null into a double (FAIL_ON_NULL_FOR_PRIMITIVES), so the flag counts as malformed and goes to the DLT | Documented gotcha; real flags carry every field | Known |
| Arena adds a new account status | PlayerStatus.valueOf would throw | PreChecks.statusOf catches it and treats the player as unknown (skip, no crash) | Guarded |
| Someone adds partitions to a player-keyed topic | Keys remap: ordering breaks, and the stream-table join silently stops matching (co-partitioning broken) | Topics.PLAYER_PARTITIONS constant; the end-to-end IT asserts the repartition topic has 6 partitions like profiles | Partly guarded |
| A code change shifts auto-generated internal topic names | A redeploy abandons in-flight repartition data | The repartition step is explicitly named flags; stores are named | Guarded |
2.5 Configuration, startup and wiring
| Risk | What happened or would happen | Fix | Status |
|---|---|---|---|
| Overriding one element of a YAML list in tests | Happened. Boot replaced the whole arena.accounts.services list; usernames became null; all 9 integration test classes failed to load | Repeat the usernames; correct the spec's snippet | Fixed |
| An environment variable silently overrides YAML | Happened. The shared .env's SERVER_PORT=8082 moved the simulator off 8081 | Set it explicitly for the simulator in Compose | Fixed |
| An aggregate over an empty table returns NULL | Happened. percentile_cont returned NULL, JdbcClient rejected a null result, and Arena crashed at startup on a fresh database, so a clean docker compose up would have failed | Return Optional from the extractor | Fixed |
| A bean with two constructors | Happened. The verifier couldn't start inside Spring, despite 100% unit coverage | @Autowired on one; VerifierContextTest | Fixed |
| Shared config class outside the component scan | Topics never created | @Import(TopicsConfig.class) | Guarded |
| A missing or blank secret | A confusing crypto or 401 error at first use | Startup validation that names the missing setting without printing values (all three projects) | Guarded |
| An auto-ban threshold configured below Arena's floor | The Judge would send bans Arena rejects | JudgeSettings refuses to start with less than 0.80 | Guarded |
2.6 Time
- A test that broke with the calendar (happened):
TokenServiceTestissued tokens with a fixed clock, but the JWT decoder's expiry validator used the system clock, so the test passed only before 10:15 UTC on 2026-11-17. The fix gave the decoder aJwtTimestampValidatoron the same fixed clock. Lesson: injecting aClockonly works if every time-aware component uses it, library ones included. - Event time vs wall time: windows use event time (the record timestamp from
MatchEvent.ts), but the cooldown uses the injected wall clock (as the spec says). Mixing the two is deliberate, and a source of surprise when replaying old data: a replay would apply cooldowns by today's clock, not by the events' times. - Late events: events more than 2 s late (in stream time) after a window ends are dropped. Fine for live play; a problem for clients that buffer events offline.
- Wall-clock match boundaries make whole simulation runs non-repeatable even with a fixed seed, so the README states it rather than claiming bit-for-bit reproducibility.
2.7 Performance and capacity
| Risk | What happened or would happen | Outcome |
|---|---|---|
| Wrong data structure for rank | The database rank is O(N): 18 s p95 at 1M players | Redis sorted set: 63.5 ms p95, measured |
| A hidden repartition per window | The spec's topology would have pushed ~24,000 events/s through the broker twice more | Caught by printing topology.describe(); fixed with a key-preserving re-parse; guarded by a shape test |
| Disabling the state-store cache | Every update went to the changelog; the detector fell behind; median detection latency rose from ~3 s to ~30 s | Keep the 64 MB cache; measured in the no-cache run |
| Unlimited JVM memory in containers | Three detector JVMs stalled a 16 GB laptop until Docker had to be restarted | mem_limit: 1g |
| Too little headroom for failover | Two survivors had ~5,000 events/s spare; lag peaked near 380,000 and took ~2 min to drain | Reported; fix is capacity, not code |
| Pool saturation | 200 virtual users on 20 connections: database latency rose from 1.5 s server-side to 15 s client-side | Explained in the README |
save() on entities with assigned ids and no @Version | Spring Data calls merge, which SELECTs before every INSERT (Match, MatchParticipant, ModerationAction, RefreshToken; participants cost one extra SELECT per player in every recorded match) | Open; fix with Persistable (module 07) |
The Judge's ProcessedFlags never evicts | Every flag id ever judged stays in memory, and startup replays the whole compacted verdicts topic. Memory and startup time grow forever | Open; fine at portfolio scale, a real problem at scale (module 18) |
2.8 Security
| Risk | Status and mitigation |
|---|---|
| Broken access control (editing another player) | "me" endpoints take the id from the token; role and business-rule checks |
| Passwords longer than bcrypt's 72-byte limit | Request validation caps passwords at 72 characters (multi-byte characters could still exceed 72 bytes) |
| LIKE wildcard injection in player search | !, % and _ escaped with an escape '!' clause |
| Path traversal through the prompt version | [A-Za-z0-9._-]+ enforced at startup |
| Username enumeration by timing on login | Open: no dummy bcrypt for unknown users |
| Brute force on login and appeals | Open: no rate limiting (on Arena's next-steps list) |
| Refresh-token race (2.1) | Open |
| Prompt injection through player-controlled text | Usernames reach the model in flags and the profile tool; past moderation reasons (written by humans, or by earlier Judge runs, so model output fed back as input) reach it through the history tool, truncated to 200 characters. Mitigated structurally: read-only guarded tools, schema + validator, deterministic policy, 0.90 threshold, Arena's own floor, audit and appeals. The model can be fooled into a wrong verdict, never into an action |
| Secrets in images or logs | .dockerignore; env-only secrets; startup checks print names, never values; logs never contain prompts, tool results or flag text (tested) |
| Actuator exposure | Prometheus endpoint unauthenticated by design for local scraping; documented in the README |
2.9 Measurement: numbers that could mislead
- Illustrative numbers mistaken for results. The Judge's README shows "optimal imaginary values" (added on request and clearly labelled). The risk is entirely in how they're quoted. Never quote them.
- A comparison that didn't separate the hypotheses. At-least-once showed 0 duplicates too. Without the negative control you couldn't even trust the 0s.
- Client vs server timing. 15 s vs 1.5 s for the same database query: the difference is queueing, not the query.
- Windowed lag metrics.
records-lag-maxtrails reality; the authoritative lag comes fromkafka-consumer-groups.sh --describe, and the measured-run script waits for it to reach 0 before verifying. - An average of averages (Arena). A player's "average reaction" is
sum_reaction_ms / matches_played: the mean of each match's average, not weighted by shots. A match with 3 shots counts as much as one with 300. The percentile distribution uses the same definition, so it's internally consistent, but it's not the true per-shot average. Storing total shots and total reaction time would fix it. - The rating system.
RatingCalculatoris placement-based (+30 for first, −30 for last), ignoring opponent strength. Ratings are consistent and testable but not a true skill estimate like Elo, Glicko or TrueSkill. That's fine for this project's goals; say so if asked. - Single-machine benchmarks. The load generator shared the CPU with the services; PostgreSQL ran with defaults; only the third run was fully captured.
2.10 Tooling and environment
- Git Bash rewrote container paths (happened, and the evidence survives). In the first measured run,
reports/chaos-exactly_once_v2.txtends withexec: "C:/Program Files/Git/opt/kafka/bin/kafka-consumer-groups.sh": no such file: Git Bash turned/opt/kafka/...into a Windows path.export MSYS_NO_PATHCONV=1fixed it, and the later at-least-once report shows the full consumer-group table. docker compose down -vleaves profile services behind (happened). The first at-least-once attempt would have reused the previous run's finished simulator container. Name the profiles:--profile sim --profile tools down -v.- Line endings and executable bits.
.gitattributesforces LF formvnwand*.sh;core.autocrlf false; the executable bit is recorded in git; the Judge's CI runssh ./mvnwin case the bit is lost. - A shell heredoc with an apostrophe in a comment broke a file once, so source files are written with an editor tool, not shell heredocs.
- Surefire combines
groupsandexcludedGroupswith AND. The spec's hard-codedexcludedGroups=liveplus-Dgroups=livewould have run nothing. The fix makes the exclusion a property you can override. - Docker Desktop not running blocks every integration test; it's why Arena's full suite hasn't been re-run since the last fix.
mvnisn't installed; everything uses the wrapper. In a multi-module build that's never installed to~/.m2, single-module commands need-am.
2.11 AI-specific risks (the Judge)
- The spec's SDK calls didn't all exist.
MessageParamhas no builder; the SDK's Jackson mapper lives atObjectMappers.jsonMapper(); structured output goes through the untyped schema path. Each was found withjavapagainst SDK 3.10.0 and marked// DECISION. - The real API may still reject the request shape. Four strict tools plus
output_config.format, and an optionallimitin a strict tool, are modelled in WireMock but unverified until the live smoke test runs. If rejected, the loop, gateway and stubs need rework. This is the biggest technical risk in the Judge. - Strict tool schemas can't express numeric ranges, so
limit's 1–20 range lives in the guard. - Output drift. Prose around JSON, fences, wrong capitalisation: handled by a strict parser (counted as PARSE failures) and case-insensitive enum parsing.
- Ambiguous requirement. "Escalate when confidence < 0.85" would escalate every CLEAR. The symmetric reading is documented and still awaits a decision.
- Cost blow-ups. Bounded by turn, tool and token limits; a shared rate limiter; routing; caching; SDK retries off.
- Anchoring. A reviewer model shown the first answer tends to agree, so Sonnet starts fresh.
- Evaluation leakage. Labels never in flags or snapshots, shuffled case ids, a dev/holdout split, ground truth hidden from the detector: all tested.
3. The decisions that mattered most
For each: the alternatives, why this one, and what it cost. Being able to argue against your own choice is what makes it a decision rather than a default.
| # | Decision | Alternatives | Why | Cost / would I change it? |
|---|---|---|---|---|
| 1 | Write a full implementation spec first, with frozen contracts and a step-by-step build order with tests per step | Code first, discover the design | Three services must agree on APIs, topics and JSON; acceptance criteria and "how each number is measured" are defined before any code | The specs contained wrong snippets (SDK calls, the topology, test properties). Deviations had to be documented. Keep it. |
| 2 | PostgreSQL as truth, Redis as a derived index, synced after commit | Redis only; DB only; outbox | O(log N) ranks without risking truth; Redis is disposable | In-memory events can be lost; the outbox is the next step |
| 3 | Pessimistic, sorted locking for matches | Optimistic + retry; atomic SQL updates | Collisions are likely; retrying a 100-player match is expensive; sorting makes deadlocks impossible | Locks serialise hot players. Keep. |
| 4 | The audit log enforced by a database trigger | Application discipline only | Bugs can't rewrite history | Doesn't stop TRUNCATE or a DBA; add role permissions |
| 5 | A pure state machine and a pure policy table | Rules inside services | Exhaustive tests (35 pairs; every policy row) | None. The best decision in Arena. |
| 6 | HS256 JWT + 15-minute access + rotating hashed refresh tokens with reuse detection | Sessions; long-lived JWTs; RS256 | Stateless scaling; revocation through the refresh token | The refresh path needs a row lock (2.1). RS256 if verification spreads |
| 7 | Parse inside the topology (raw bytes → processor → DLT) | Deserialization exception handler | Bad records become dead letters with offsets, and the stream lives | Each event parsed twice (decision 8). Keep. |
| 8 | Re-parse with mapValues instead of grouping process() output | Follow the spec (two repartition topics) | Avoids rewriting ~24,000 events/s through the broker | Double JSON parsing. Clearly the right trade. |
| 9 | Hopping 5 s/1 s windows that emit on every update, plus a wall-clock cooldown | Tumbling; suppress until close; sliding windows | Detection about 1 s after crossing a threshold; no boundary misses | Repeat emissions to de-duplicate; replay semantics are odd (2.6) |
| 10 | Prove exactly-once with per-player sequence numbers and a negative control | Compare totals; trust the configuration | Catches loss + duplicate pairs; a test that can fail | Re-emitting all players every 5 s doesn't scale to millions (module 18) |
| 11 | Standbys + a 6 s session timeout | Defaults (45 s) | Fast detection, warm state | Rebalances still cost ~9 s; target missed. Tune further |
| 12 | No agent framework | LangChain4j, Spring AI | Full visibility, testability and cost accounting | More code to own; provider lock-in. Keep for this scope. |
| 13 | The model recommends, code acts | Let the model call action tools | Bounded blast radius; injection-resistant | Less flexible. Non-negotiable. |
| 14 | Cheap model first, strong model fresh on escalation, shared tool cache | One model; reviewer sees the first answer | Cost plus an independent second opinion | Two models to evaluate |
| 15 | The Kafka topic as the Judge's only state | A database | One less system; replayable; idempotent | Unbounded replay and memory (2.7). At scale add a TTL'd store |
| 16 | Move ArenaToolExecutor and ArenaReader into core | Keep them in the service (the spec) | The evaluation records snapshots with the exact production tool code | A departure from the spec, documented. Keep. |
| 17 | Testcontainers over mocks for infrastructure; coverage gates | H2, embedded Kafka, mocks | Found real bugs mocks couldn't | Slower builds, Docker required. Keep. |
| 18 | Report measured numbers including misses | Report only successes | Credibility | Some uncomfortable README rows. Always keep. |
4. Where the build struggled most
Ranked by how much each cost, according to the notes and fixes. For each: what happened, how it was diagnosed, how it was solved, and the lesson.
1. Brand-new framework versions (Boot 4, Jackson 3, Kafka 4, Testcontainers 2, SDK 3.10)
What happened: most examples online, and parts of the specs, were written for older versions. Imports didn't resolve (@MockBean, Jackson 2 packages, the health API, AutoConfigureMockMvc's package), Kafka Streams' transform family was off-limits, Kafka turned out to be 4.2.2 rather than 4.1.x, and the Anthropic SDK lacked the builders the spec used. Diagnosis: reading the dependency jars (javap), release notes, and dependency:tree. Solution: a pitfalls table in each spec and CLAUDE.md; never downgrade, find where the class moved. Lesson: on new major versions, the primary sources (javadoc, the jars, release notes) beat tutorials, and a five-minute smoke test of each critical API up front saves hours.
2. Bugs that only appear when everything runs for real
What happened: the first full ./mvnw verify with Docker found two production bugs and two wrong tests in Arena (the empty-database crash; the list-override that broke all integration tests; a 403 test sending an invalid body; a 405 test that was unauthenticated). Telemetry's verifier had 100% coverage and couldn't start. Diagnosis: running the real stack and the full suite, not just unit tests. Solution: fixes plus tests that start the real context. Lesson: run the whole thing early and often; coverage measures execution, not correctness.
3. Kafka Streams' hidden behaviour
What happened: the repartition trap (found by printing topology.describe()); flags emitted on every window update, so a cooldown was needed; with the 64 MB cache, integration tests saw a different "first" window than TopologyTestDriver; failover took three rebalances. Solution: shape tests, a cooldown store, ITs that don't assert exact evidence, and an honest failover report with a timeline from logs. Lesson: with stream processors, inspect what the framework derives (topology, internal topics, emission timing), not just your code.
4. Running a distributed system on one laptop
What happened: memory stalls that needed a Docker restart; a port hijacked by a shared .env; containers left behind by down -v; Git Bash path rewriting that broke the first chaos report; lag metrics that lag. Solution: memory limits, explicit per-service settings, named profiles, MSYS_NO_PATHCONV=1, and the authoritative lag command in the measured-run script. Lesson: automate the measured run end to end (measured-run.sh) so every run is identical, and keep the raw outputs as evidence.
5. Results that didn't cooperate
What happened: failover missed its 10 s target (14.4 s); lag didn't stay flat; the at-least-once comparison showed nothing; the Judge's evaluation couldn't run without an API key and a special Telemetry run. Solution: report exactly what was measured, try harder where the spec asked (the no-cache rerun), label the Judge's figures as imaginary, and document how to produce the real ones. Lesson: the credibility of every other number depends on how you handle the ones that missed.
6. Time and concurrency in tests
What happened: the calendar-dependent token test; a concurrency test whose interesting path depends on timing. Solution: a fixed clock everywhere, including library validators; latch-synchronised threads; invariants that hold whichever path runs. Lesson: nondeterminism hides in time, threads and ordering, so design tests around invariants.
7. Uncertainty at the LLM boundary
What happened: SDK differences from the spec, strict-schema limitations, two Jackson major versions in one process, and an API shape that can only be confirmed live. Solution: an interface you own (MessagesGateway), a smoke test written and ready, a string bridge between Jackson 2 and 3, and the range check in the guard. Lesson: wrap third-party AI APIs behind your own seam on day one, and verify the request shape before building on it.
"The hardest part was that the bugs that mattered only appeared when everything ran for real. My unit tests were green, but the first full run with real PostgreSQL crashed Arena on an empty database: percentile_cont returns NULL with no rows, and Spring's JdbcClient rejects a null result, so the app died at startup. The same run showed that overriding one element of a YAML list in tests replaces the whole list. Since then I keep at least one test that starts the real context against real containers, and I run the full stack early instead of at the end."
5. What I wish I knew earlier: 30 lessons
Frameworks and versions
- New major versions move classes. When an import fails, look in the jar before searching the web.
- Boot replaces lists from higher-priority sources; it doesn't merge them.
- Environment variables beat YAML, and a shared
.envreaches every service. - A bean with two constructors needs
@Autowiredon one, and only a context test will tell you. - Jackson 3 fails on missing primitive fields; use wrapper types when absence is legal.
Data
- Aggregates over empty sets return NULL. Design startup code for an empty world.
- In PostgreSQL an error aborts the whole transaction. Catch constraint violations outside it.
- Check-then-act is never enough; the unique constraint or row lock is the real guarantee. This includes refresh tokens.
- Pick the data structure for the query (rank needs an order-statistic structure, not an index).
save()with assigned ids meansmergeunless the entity tells Spring it's new.- Define metrics precisely (an average of per-match averages is not an average).
Kafka and streaming
KStream#processoutput counts as key-changing; checktopology.describe()for unplanned repartition topics.- Windowed aggregations emit on every update; plan de-duplication.
- Exactly-once ends at Kafka's edge; metrics and HTTP calls inside processors can run twice.
- Failover = detection + rebalances; the session timeout is only the first part.
- Lag metrics on clients are windowed maxima; trust
kafka-consumer-groups.sh. - Advertised listeners decide what clients connect to; one per network audience in Docker.
- Leave headroom: N−1 instances must absorb the full load plus a backlog.
LLM engineering
- Verify the provider's real request shape first with one live call, before building the loop around it.
- Turn off SDK retries when you add your own.
- Strict JSON schemas have limits; validate again in code.
- Read requirements against definitions: "confidence" meant probability of cheating, which changed the escalation rule.
- Anything the model reads is an injection surface, including your own earlier outputs stored as moderation reasons.
- Evaluation needs data you can't fake. Plan the dataset run (and the API budget) at the start.
Testing, operations and process
- Every time-aware component must use the injected clock, including library validators.
- Always set JVM container memory limits.
- On Windows, Git Bash rewrites paths; set
MSYS_NO_PATHCONV=1in scripts that call into containers. - Surefire ANDs
groupsandexcludedGroups; make exclusions overridable properties. - Write a negative control for every checker, otherwise "0 errors" proves nothing.
- Automate measured runs and keep raw outputs; they're your evidence in interviews.
6. What is still wrong or unfinished today
Know this list cold. Volunteering an item before being asked is a strong signal.
| Project | Open item | Effort to fix |
|---|---|---|
| Arena | Full ./mvnw verify not re-run since the token-test fix | Minutes (start Docker) |
| Arena | Refresh-token race (2.1) | Small: a locking query or a conditional UPDATE, plus a latch-based test |
| Arena | Nothing ever deletes expired or revoked refresh tokens: every login and refresh adds a row to refresh_tokens forever (module 18 shows what that means at scale) | Small: a scheduled cleanup of rows past expiry, or time-partitioning and dropping old partitions |
| Arena | Wrong role + invalid body gives 400, not 403 (validation runs before @PreAuthorize) | Duplicate role rules as URL rules; decide if worth it |
| Arena | The rebuild compares counts, not scores; possible out-of-order Redis updates | Reconciliation job, or a versioned Lua update, or an outbox |
| Arena | Tagged counters appear on /actuator/prometheus only after first use | Pre-register them |
| Arena | No rate limiting; login timing leak | Gateway or Bucket4j; dummy bcrypt |
| Arena | Reaction average definition; placement-only rating | Store shot-weighted totals; consider Elo or Glicko |
| Telemetry | Failover 14.4 s vs < 10 s; lag not flat | Rebalance tuning, new protocol, more headroom |
| Telemetry | The bridge never run against a real Arena | One live run |
| Telemetry | Single broker, RF 1 | A three-broker cluster in Compose |
| Telemetry | CI workflow written but never run on GitHub | Push |
| Judge | Live smoke test never run; request shape unverified | One command with an API key (a fraction of a cent) |
| Judge | No dataset, no evaluation reports, prompts v2–v6 not written; README shows imaginary values | A Telemetry run with loosened thresholds, collect, four runs, iterate |
| Judge | LOW_CONFIDENCE reading undecided | A product decision, then one line |
| Judge | ProcessedFlags grows forever | Expire entries older than the window; a TTL store at scale |
| Platform | Never run end to end live | Follow the Judge README's "Run the whole system" |
7. Interview questions
What's the hardest bug you found? Intermediate
Pick from section 4 and use STAR. Strong candidates: the empty-database startup crash (NULL from percentile_cont rejected by JdbcClient), the concurrent duplicate externalId returning 500 (a PostgreSQL aborted transaction means you must catch after rollback), or the repartition trap (found with topology.describe()). Always end with what you changed in your process, not just the fix.
What could go wrong in Arena under load, and what stops it? Deep
Walk the concurrency table: lost rating updates (row locks), deadlocks (sorted lock order), duplicate matches (unique key + catch after rollback), double bans (@Version + state machine), duplicate usernames (unique expression index). Then name the open one: two concurrent refreshes of the same refresh token can both succeed because the token row isn't locked, which I'd fix with a conditional UPDATE that requires revoked_at IS NULL. Then capacity: pool saturation, and the database rank path.
Explain the refresh-token race and how you'd fix it. Deep
refresh() loads the token by hash, checks it isn't revoked, issues a new pair and marks the old one revoked, with no lock and no version column. Two requests with the same token can both pass the "not revoked" check under Read Committed; the second UPDATE waits for the first's row lock, then overwrites it, and both callers get valid new refresh tokens, so reuse detection never fires. Fixes: lock the row (@Lock(PESSIMISTIC_WRITE) on the lookup), or make revocation conditional (UPDATE ... SET revoked_at = now WHERE id = ? AND revoked_at IS NULL, and if 0 rows changed, treat it as reuse), or add @Version. I'd prove it with a latch-based test that fires two refreshes together and asserts exactly one succeeds.
Which decision would you reverse or revisit? Intermediate
Good honest answers: (1) the in-memory after-commit events for Redis: I'd move to a transactional outbox for guaranteed, ordered delivery; (2) the Judge's unbounded in-memory ProcessedFlags: I'd expire entries beyond the 30-minute window or use a TTL store; (3) running the integrity proof by re-emitting every player every 5 s, which is fine for 1,000 players and wasteful for millions. Explain why each was reasonable at the time: simplicity at portfolio scale.
Tell me about a time a test lied to you. Intermediate
Two good ones. The verifier had 100% line coverage but couldn't start, because Spring couldn't choose between two constructors and no unit test started the context. And TokenServiceTest passed only before a certain time of day, because the JWT decoder checked expiry against the real clock while the token used a fixed one. Lessons: keep a context test, and make every time-aware component use the injected clock.
Tell me about a time a tool or environment misled you. Basic
Git Bash on Windows rewrote /opt/kafka/bin/... into a Windows path when passed to docker compose exec, so the first chaos report's consumer-group check failed. The evidence is still in chaos-exactly_once_v2.txt. Setting MSYS_NO_PATHCONV=1 fixed it, and the next run printed the full group table. Also: client-side Kafka lag metrics are windowed maxima, so I switched to kafka-consumer-groups.sh for the authoritative lag.
Where could prompt injection reach your agent? Deep
Anywhere text the model reads is influenced by someone else: usernames in the flag and profile tool, and moderation reasons in the history tool, which include human moderators' notes and the Judge's own earlier explanations (model output becoming future input). I limit what a successful injection can do rather than trying to detect it: tools are read-only and restricted to the flagged player, output must pass a schema and a validator, a pure policy decides with a 0.90 bar for bans, Arena enforces its own floor and state machine, and everything is audited and appealable. Worst case is a wrong verdict a human can review, never an action.
What do you wish you'd known before starting? Basic
Pick three from section 5 that you can explain with a story. For example: "that exactly-once stops at Kafka's edge, so counters inside processors can double; that Kafka Streams treats process() output as key-changing, which nearly cost a repartition per window; and that I should verify an LLM provider's real request shape with one live call before building the loop around it."
Your Telemetry comparison didn't show what you expected. What does that tell you about your experiment design? Curveball
That one kill per run is a weak experiment: a duplicate under at-least-once requires the kill to land between a state write and the offset commit, a narrow window. A better design would run many automated kills per run, or inject the fault precisely at that point, and report the rate of duplicates across trials. What made the existing result trustworthy is the negative control: the checker provably detects duplicates, so the zeros mean "none happened", not "couldn't see them".
Is your Arena "average reaction time" correct? Curveball
It's consistent but not the true per-shot mean: Arena stores sum_reaction_ms as the sum of each match's average and divides by matches played, so every match weighs the same regardless of how many shots it had. The percentile distribution uses the same definition, so comparisons between players are fair, but the absolute number is biased toward short matches. Storing total shots and total reaction milliseconds and dividing would give the weighted mean.
What's the biggest technical risk left in the Judge? Intermediate
That the real API rejects the request shape: four strict tools plus output_config.format, and an optional property in a strict tool schema. Everything is tested against WireMock stubs that model my understanding of the API. One live smoke test, costing a fraction of a cent, would confirm or refute it, and it's the first thing I'd run.
What did the spec get wrong, and how did you handle it? Intermediate
Examples: Telemetry's topology snippet would have created two repartition topics; Arena's test-property snippet broke list binding; the Judge's SDK calls used builders that don't exist, its command for the live smoke test would have run nothing (Surefire ANDs groups with excludedGroups), and a literal reading of the confidence rule would have escalated every CLEAR. In each case I implemented the intent rather than the text, marked the spot with a DECISION comment, recorded it in the handoff notes, and added a test that guards the choice.
8. Cheat sheet
- Failure taxonomy: concurrency, cross-store consistency, delivery, contracts, config, time, capacity, security, measurement, tooling, AI.
- Fixed during the build: empty-DB crash; list override; duplicate-externalId 500; refresh check order; verifier constructors; token-test clock; SERVER_PORT; Git Bash paths;
down -vleftovers; memory stalls; the repartition trap; the Surefire groups AND. - Found while writing this guide: the refresh-token race;
ProcessedFlagsunbounded; average-of-averages reaction;save()merge overhead. - Top decisions: spec-first contracts; truth vs derived Redis; sorted pessimistic locks; DB trigger; pure state machine and policy; parse-in-topology; mapValues re-parse; seq proof + negative control; no framework; model recommends, code acts; fresh Sonnet; topic as state; Testcontainers; honest numbers.
- Struggles: version churn; real-run-only bugs; Streams internals; laptop-scale distributed ops; uncooperative results; time and concurrency in tests; LLM API uncertainty.
- Still open: Arena verify, refresh race, 400-vs-403, score repair, rate limiting; Telemetry failover, real Arena run, RF 1, CI on GitHub; Judge live test, dataset and reports, confidence decision, unbounded state; platform E2E.