Fairplay Judge, explained from scratch
A guided tour of Project 3 for someone who has never built an LLM agent, never used Resilience4j and is not sure what a "record" is. Every idea is explained the first time it appears. Every code sample is copied from the real files (shortened where marked with ...). The last sections turn all of it into answers you can say out loud in an interview.
0. Where the project stands
The project is built in 12 steps (the plan is in Fairplay Judge — Implementation Specification.md, section 17). Steps 1 and 3 to 11 are finished, and step 12 is finished except the parts that need an API key. Everything in this document marked built exists and is tested. Anything marked needs you cannot be done without your API key or a finished Telemetry run.
| Step | What | Status |
|---|---|---|
| 1 | Maven skeleton, three modules, shared settings with validation | done |
| 2 | SDK smoke test: one real call to Claude to check the request shape | written and compiles; never run live (needs an API key) |
| 3 | The verdict: data shape, JSON schema, strict parser, validator | done |
| 4 | Prompts, the four tools, the guard, the per-flag cache | done |
| 5 | The agent loop | done |
| 6 | Resilience (retry, breaker, rate limit) and cost calculation | done |
| 7 | Model routing, escalation rule, decision policy, JudgeEngine | done |
| 8 | Arena side of the service (HTTP client, tools backed by Arena, applying actions) | done |
| 9 | Kafka side of the service (listener, publishing verdicts, error handling, metrics) | done |
| 10 | Fault injection tests (WireMock container) | done |
| 11 | Evaluation harness: build the 200-case dataset, replay it, score it | code done and tested; dataset not collected yet (needs a Telemetry run) |
| 12 | Prompt versions v2 to v6, Docker, CI, Grafana, README | Docker, Compose, CI, dashboard and README done; v2 to v6 need evaluation runs |
Today's tests: 194 in judge-core (97.2% of lines executed by tests), 77 unit tests and 12 integration tests in judge-service (96.9%) and 105 unit tests and 1 integration test in judge-eval (96.2%).
collect, all six prompt versions and the cost runs. You have not. Every result figure (precision, recall, cost per verdict, latency, review rate, token counts of a real run) is therefore marked optimal imaginary value: a value chosen to show what a good run would look like, picked so that it is consistent with the prices, the token arithmetic and the project targets. It is not a measurement. The only measured results are the test results (for example: in ResilienceIT, 20 of 20 flags were judged during a simulated outage). In an interview, say "the design aims at this" until you replace the imaginary values with your own, using evals/README.md. The same imaginary values are in the README's Results table, marked the same way.1. The story: what is this project for?
Imagine an online shooting game. Some players cheat with an aimbot (software that aims for them) or a speedhack (moving faster than the game allows). The Fairplay system fights them with three projects:
Project 1: Arena
The game's backend. It stores players, matches, ratings and moderation history. It can ban a player. A web API.Project 2: Telemetry
Watches a stream of in-game events (shots, positions) in real time and raises a flag when a player looks suspicious in a 5-second window. Fast, but noisy.Project 3: Judge (this one)
Reads each flag, investigates the player with an AI model that can look things up in Arena, and decides: ban, watch, or clear.Why not just ban every flag?
A 5-second window is tiny. A genuinely great player can look like a cheater for 5 seconds: a lucky streak, a headshot burst. If you banned every flag you would ban many innocent people, and banning an innocent player is the worst outcome. So someone has to look at the player's long-run record: hundreds of shots, many matches, past moderation history. That is a human moderator's job, and it is slow and expensive. The Judge does that investigation automatically, and sends only the genuinely unclear cases to a human.
An analogy
Think of a referee system in a sport. A fast camera system (Telemetry) flags a possibly illegal play. A junior reviewer (the cheap model, Claude Haiku) looks at the replay and the player's history. If the junior is confident, that is the answer. If the junior is unsure, or the player is a star, a senior reviewer (the strong, expensive model, Claude Sonnet) reviews it from scratch, without being told what the junior thought. Finally, a rules clerk (plain Java code, no AI) reads the reviewer's recommendation and decides what actually happens, using strict rules. The reviewers can only recommend. They can never press the "ban" button themselves.
Three promises the design keeps
- The model recommends, code acts. The AI returns a verdict (data). A plain Java class,
DecisionPolicy, turns it into an Arena action. The model never chooses a web request or a URL. This is the single most important design idea; you will be asked about it. - A ban needs strong confidence. Auto-ban only at confidence 0.90 or higher and with no "please ask a human" flag. Arena itself refuses service bans below 0.80 as a second safety net.
- A flag is never lost. If Claude or Arena is down, the flag waits and is retried. Only malformed flags go to a dead-letter topic.
2. Ideas you need first
If you already know a topic, skip it. Each one is short on purpose.
2.1 Calling an LLM, and what a token is
An LLM (large language model) such as Claude is used through a web API. You send a request: the model name, some instructions (the system prompt), the conversation so far (a list of messages, each from the user or the assistant), and a limit on how long the answer may be. You get back a response containing the answer and a stop reason (why it stopped, for example "I finished" or "I hit your length limit").
The model reads and writes tokens, which are chunks of text, roughly three quarters of a word each. You pay per token, and output tokens cost more than input tokens. Prices in this project come from judge-defaults.yml in USD per million tokens, for example Haiku input $0.10 and Sonnet input $2.00. Haiku is the small, cheap, fast model; Sonnet is the larger, more capable, pricier one.
Calls fail: the service can be overloaded (HTTP 529), you can be rate-limited (429), the network can drop. A production system has to survive all of that. See 2.6.
2.2 Tools and the agent loop
A plain model only knows what is in its prompt. A tool (also called a function call) lets the model ask your program to look something up. You describe each tool to the model by name, description and input shape. The model can then answer not with text but with a request: "please run get_stat_percentiles with this player id". Your program runs it and sends the result back. The model continues. This repeat-until-done pattern is an agent loop:
send messages + tool list to the model
repeat:
model replies
if it asks for tools: run them, append results as a new message, go around again
if it gives an answer: stop and use the answer
if it hit a limit: give up on this attempt
Important: the model never runs anything. It only writes a request. Your code decides whether to run it. That is why the "guard" in section 5.5 can refuse any request that is not about the flagged player.
2.3 Structured output and a JSON schema
You want the final answer as data, not prose: "BAN, confidence 0.94, these 3 facts". A JSON schema describes the allowed shape (field names, types, allowed words). You send the schema with the request and the API forces the model's answer to match it. This project's schema has six fields: decision, cheatType, confidence, evidence, explanation, needsHumanReview.
The API guarantees the shape but not every limit (for example "at most 600 characters" is only a hint). So the code validates again. Never trust output just because it was "structured".
2.4 Prompt caching
Every flag sends the same long instructions (the rulebook, about 3,000 tokens) before the part that changes (the flag). Re-sending thousands of identical tokens for every flag is wasteful. Prompt caching lets Anthropic remember a prefix of the prompt: the first request writes the cache (slightly more expensive), later requests read it (about ten times cheaper than normal input). You mark the end of the part to cache with a cache_control marker.
The cache works on a prefix, so the order matters: everything identical across flags first (tools, then system text, then the rulebook), then the changing flag last. Put anything that changes (a timestamp, a player id) inside the cached part and the cache never hits. Usage numbers come back in four kinds: normal input, cache write, cache read, output, and each has its own price. This is the main reason the project can be cheap.
2.5 Java you will meet
You do not need to be an expert, but you need to recognise these.
| Thing | What it means | Example in this project |
|---|---|---|
| record | A small immutable data holder. record Verdict(Decision decision, double confidence) automatically gets a constructor, getters (v.confidence()), equals and toString. The spec requires records for all data. | Verdict, Attempt, ToolResult |
| compact constructor | The block inside a record with no parameter list. It runs on creation, so it is the place to reject bad values. | JudgeSettings.Policy refuses a ban threshold below 0.80 |
| enum | A fixed list of allowed values. | Decision { BAN, WATCH, CLEAR } |
| interface | A promise of what methods exist, without saying how. Lets you swap implementations (real, fake). | MessagesGateway, ToolExecutor |
| final | On a class: cannot be extended. On a field: assigned once. The spec wants every field private final. | almost every class |
| unchecked exception | An error you do not have to declare. Here, all project exceptions extend RuntimeException. | InvalidVerdictException |
| Optional<T> | "A value or nothing", instead of null. | response.stopReason() |
| generics | Types with a type in angle brackets: List<String> is a list of text. Read it as "a List of String". | everywhere |
| lambda | A short inline function: (a, b) -> a + b. Supplier<T> is "a function with no input that returns a T". | the gateway's decorators |
| decorator pattern | An object that wraps another object of the same type and adds behaviour, then passes the call on. | GuardedToolExecutor and MemoizingToolExecutor wrap a ToolExecutor |
| static factory | A static method that builds an object, with a meaningful name. | JudgeEngine.create(...), ToolResult.ok(...) |
| dependency injection | A class receives what it needs through its constructor rather than creating it. Makes it testable, because tests can pass fakes. | new AgentLoop(gateway, prompts, ...) |
2.6 Retry, circuit breaker, rate limiter
Three small safety devices, all from the library Resilience4j:
- Retry with exponential backoff and jitter. If a call fails for a temporary reason, try again, waiting longer each time (1 s, 2 s, 4 s...) with a random wobble ("jitter") so many callers do not all retry at the same instant.
- Circuit breaker. Like a fuse. If most recent calls are failing, the breaker "opens" and calls fail immediately without even trying, giving the service time to recover. After a pause it lets a few trial calls through ("half-open"); if they work, it "closes" again.
- Rate limiter. Allows at most N calls per minute, so you never exceed what the provider permits.
The order matters and is a favourite interview question. See 5.7.
2.7 Maven, Spring Boot, Jackson, tests
- Maven builds Java projects. A
pom.xmllists dependencies (libraries) and plugins. This repository is a multi-module build: one parentpom.xmland three child modules../mvnw verifycompiles everything, runs every test, and checks coverage.mvnwis a wrapper script that downloads the right Maven version. - Spring Boot is a framework that starts web servers, reads config files and connects Kafka. Only
judge-serviceandjudge-evaluse it.judge-coredeliberately does not, so it can be tested with plain JUnit. - Jackson converts between Java objects and JSON text. This project uses Jackson 3, whose package is
tools.jackson...(the older Jackson 2 lives undercom.fasterxml.jackson...; the Anthropic SDK still uses Jackson 2 inside). - JUnit and AssertJ write tests: a method marked
@Test, thenassertThat(actual).isEqualTo(expected)reads almost like English. - Test doubles. A fake is a simple working stand-in (a list instead of a database). A stub returns canned answers. A mock records how it was called. This project mostly uses hand-written fakes such as
ScriptedGateway.
2.8 Kafka in one minute (needed for steps 9 and later)
Kafka is a durable message log. A topic is a named log. Programs produce records into a topic and other programs consume them, each remembering how far it has read (the offset). If a consumer crashes before saving its offset, it re-reads the record: this is at-least-once delivery, so processing must be safe to repeat (idempotent). Telemetry writes flags to suspicious-players; the Judge reads it and writes judge-verdicts. None of the Kafka code exists yet (step 9), but the design decisions that depend on it appear throughout the core.
3. Map of the repository
One Maven project with three modules. Read the next tree top to bottom; "built" means the files exist today.
Reading order that works
- This page, sections 1 to 4.
JudgeEngine.java: it is short and shows what the whole core does.model/: learn the nouns first.agent/AgentLoop.java: the heart.routing/ModelRouter.javaandpolicy/DecisionPolicy.java.- The rest (settings, tools, resilience, cost), when something earlier points to it.
- Always read a class's test right after it. Tests are examples with the answers written down.
4. The journey of one flag
Here is everything that happens to one flag, in order, with the class that does each step. The numbers match the layers in the code. Steps 1, 2 and 8 to 10 belong to the service, and steps 3 to 7 are in judge-core. Everything is built and tested.
- (service, planned) A
SuspiciousPlayerJSON record arrives from the Kafka topicsuspicious-players. The service checks Arena: is the player real, not already banned, not judged in the last 30 minutes? If any check fails the flag is recorded asSKIPPEDand no model is called (free). - (service, planned) The service calls
JudgeEngine.judge(flag, arenaTools, status). JudgeEnginewraps the tools in oneMemoizingToolExecutor(a per-flag cache) and hands everything toModelRouter.ModelRouterruns attempt 1 with Haiku throughAgentLoop.AgentLoopbuilds the request (cached system text, four tool definitions, the flag, the verdict schema), sends it throughResilientMessagesGateway, and loops: when the model asks for a tool,GuardedToolExecutorchecks the request and the cache or Arena answers.- When the model answers with text,
VerdictParserandVerdictValidatorcheck it strictly and produce aVerdict. EscalationRuleasks: is Haiku unsure, or is this a ban on a top-tier player, or is a ban too thinly supported? If yes,ModelRouterruns attempt 2 with Sonnet from a blank conversation, sharing the same tool cache. Sonnet's answer is final.CostCalculatorprices each attempt. The router records anAttemptper model call chain and returns anInvestigation(final verdict or none, plus all attempts).DecisionPolicyturns the verdict and the player's status into exactly oneArenaAction.- (service, planned) The service sends that action to Arena (always
source: JUDGE), publishes aJudgeVerdictRecordtojudge-verdicts, and only then lets Kafka mark the flag as done.
Each box only knows the box to its right. That is the whole point of the layering: you can test any layer by replacing the one to its right with a fake. AgentLoopTest replaces the gateway; ModelRouterTest replaces the gateway too, but exercises the router; ResilientMessagesGatewayTest replaces the internet with a tiny local web server.
5. judge-core, file by file
5.1 Settings: JudgeSettings and judge-defaults.yml
All tunable values (model names, limits, thresholds, prices) live in one YAML file, so an experiment means editing a value, never code. The same file is imported by the service and the evaluation, so they can never judge with different settings.
judge-core/src/main/resources/judge-defaults.yml (excerpt)judge:
prompt-version: ${JUDGE_PROMPT_VERSION:v1} # environment variable, default v1
models:
triage: claude-haiku-5-5
escalation: claude-sonnet-5-5
limits:
max-turns: 6 # model calls per attempt
max-tool-calls: 8 # tool executions per attempt
max-tokens: 1500 # output tokens per model call
routing:
triage-enabled: true
escalation-enabled: true # my addition, see section 11
escalate-below-confidence: 0.85
escalate-high-tier-bans: true
min-evidence-for-ban: 2
policy:
auto-ban-min-confidence: 0.90
pricing: # USD per million tokens
"[claude-haiku-5-5]": { input: 0.10, cache-write: 0.125, cache-read: 0.01, output: 0.50 }
"[claude-sonnet-5-5]": { input: 2.00, cache-write: 2.50, cache-read: 0.10, output: 10.00 }
The Java side is JudgeSettings, one record containing nested records. Look at how a bad config is rejected:
public record Policy(double autoBanMinConfidence, Duration recentVerdictWindow) {
public Policy { // compact constructor: runs on creation
requireProbability(autoBanMinConfidence, "judge.policy.auto-ban-min-confidence");
require(autoBanMinConfidence >= ARENA_BAN_CONFIDENCE_FLOOR, // 0.80
"judge.policy.auto-ban-min-confidence must be at least 0.8 (Arena's own floor)");
requirePositive(recentVerdictWindow, "judge.policy.recent-verdict-window");
}
}
Read it like this: you cannot even construct a Policy with an auto-ban threshold under 0.80. So a typo in YAML stops the application at startup instead of silently banning on weak evidence. The same file also checks that both configured models have a price, and that prompts/<version>/system.md and rulebook.md exist.
How does YAML become the record? In the service, JudgeSettingsConfig does one line: Binder.get(environment).bind("judge", JudgeSettings.class).get(). Spring's Binder maps max-turns to maxTurns and "30m" to a Duration. The core stays free of Spring. The service-only values (Kafka topic names, Arena URL) live in a second record, JudgeProperties.
JudgeConfigBindingTest binds the real YAML files and proves the defaults match the code's expectations.5.2 The data shapes (package model)
Learn these nouns first; the rest of the code is verbs acting on them.
| Type | What it is | Key fields |
|---|---|---|
SuspiciousPlayer | The flag from Telemetry. A copy of Telemetry's record; unknown JSON fields are ignored so Telemetry can add fields freely. | flagId, playerId, username, skillTier, rule, score, detectedAt, evidence |
Evidence | The numbers behind a flag (shots, hits, headshot rate, reaction time, or speed samples). | all nullable: aimbot flags fill some, speedhack others |
Verdict | The model's structured answer. | decision, cheatType, confidence, evidence (list), explanation, needsHumanReview |
Decision, CheatType | Enums. Each has a parse that upper-cases the text first, because models sometimes change capitalisation. | BAN WATCH CLEAR; AIMBOT SPEEDHACK NONE |
TokenUsage | Token counts for one or many calls, in four kinds. | input, cacheWrite, cacheRead, output; plus() adds two |
Attempt | One model's try: which model, what happened, what it cost. | model, outcome, stats, costUsd, latencyMs, reason |
Investigation | The models' work for one flag: final verdict (or none) plus every attempt. | verdict, attempts; helpers finalModel(), totalCostUsd() |
ArenaAction | The single action the policy chose. | kind, label, reason, evidence, confidence, requiresHumanReview |
ActionKind vs JudgeAction | ActionKind is what Arena receives: BAN FLAG CLEAR NOTE. JudgeAction is what the Judge records: BAN FLAG_FOR_REVIEW NOTE CLEAR SKIPPED. They differ on purpose, see 5.10. | |
PlayerStatus | Arena's account state. | ACTIVE FLAGGED BANNED APPEALED REINSTATED |
JudgeOutcome | What JudgeEngine returns: the investigation and the action. |
public record Verdict(
Decision decision,
CheatType cheatType,
double confidence,
List<String> evidence,
String explanation,
boolean needsHumanReview) {
public Verdict {
Objects.requireNonNull(decision, "decision");
...
evidence = List.copyOf(evidence); // an unmodifiable copy: nobody can change it later
}
}
What does confidence mean? It is the model's probability that the player is cheating, from 0 to 1. A confident "innocent" verdict has confidence near 0.05, not 0.95. Keep this in mind; it explains one of the project's key decisions (5.9).
5.3 The verdict pipeline: schema, parser, validator
Three small classes make sure only a well-formed, in-range verdict ever leaves the model's output.
VerdictSchema: what we ask for
It holds the JSON schema as a Java text block and builds the SDK's OutputConfig from it. Every object in the schema has "additionalProperties": false and a fixed list of properties (no free-form maps). VerdictSchemaTest checks that.
VerdictParser: strict on purpose
VerdictParser.java (excerpt)private final JsonMapper mapper = JsonMapper.builder()
.enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS) // "{...} thanks!" is rejected
.build();
public Verdict parse(String text) {
return validator.validate(parseUnvalidated(text)); // parse first, then validate
}
...
for (String field : FIELDS) {
if (!root.has(field) || root.get(field).isNull()) {
throw parseError("missing field " + field);
}
}
for (String name : root.propertyNames()) {
if (!FIELDS.contains(name)) throw parseError("unexpected field " + name);
}
Why so strict? The model is told "answer only with the JSON object". If it adds a sentence, wraps the JSON in a markdown fence, drops a field or invents one, that is a parse failure and the router will try Sonnet. Being strict makes failures visible and countable instead of silently accepting garbage. VerdictParserTest has a case for each.
VerdictValidator: the rules the schema cannot express
| Rule | Why code must check it |
|---|---|
confidence between 0 and 1 | The schema says "number"; ranges are only descriptions |
| 1 to 6 evidence items, each 1 to 300 characters | Fits Arena's limits and keeps tokens small |
| explanation 1 to 600 characters | Same |
BAN needs a cheat type other than NONE; CLEAR needs NONE | A BAN with no cheat type is self-contradictory |
A boundary test checks each rule at the edge: 0 and 1 pass, -0.01 and 1.01 fail; 6 items pass, 7 fail. Boundary tests are where off-by-one bugs live.
InvalidVerdictException and its two kinds
Both failures throw InvalidVerdictException carrying a Kind: PARSE (not JSON, wrong shape, unknown enum word) or VALIDATION (well-formed but out of limits, or the model stopped for a reason other than finishing). They are counted separately in the metrics. The exception can also carry AgentStats: even a failed attempt used tokens and cost money, and the router needs that number.
5.4 Prompts: PromptLibrary and UserPromptBuilder
The model's instructions are plain Markdown files, not Java strings, so they can be edited and versioned. prompts/v1/system.md is the role and the working method ("always call get_moderation_history before deciding BAN"). prompts/v1/rulebook.md holds game facts (top sprint speed 7.0 units/s, reaction under 120 ms is not human) and how to weigh evidence. Later prompt versions (v2 to v6) will be copies with one change each, measured by the evaluation.
public List<TextBlockParam> systemBlocks(boolean cachingEnabled) {
TextBlockParam.Builder rulebook = TextBlockParam.builder().text(prompt.rulebook());
if (cachingEnabled) {
rulebook.cacheControl(CacheControlEphemeral.builder().build()); // the cache breakpoint
}
return List.of(TextBlockParam.builder().text(prompt.system()).build(), rulebook.build());
}
| Position in the request | Content | Changes per flag? | Cached? |
|---|---|---|---|
| 1 | The four tool definitions | no | yes (covered by the breakpoint below) |
| 2 | System block 1: system.md | no | yes |
| 3 | System block 2: rulebook.md with the cache marker | no | yes: the breakpoint ends here |
| 4 | User message: the flag and the time | yes | no |
| 5+ | Assistant tool requests and tool results | yes | no |
PromptLibraryTest protects this: it checks the rulebook block has cache control only when caching is on, and that the system text contains no date or UUID. Why a test for that? One accidental timestamp in the system text would silently turn off caching and multiply your bill.
UserPromptBuilder makes the part that does change. For a flag it returns text like:
Flagged at 2025-11-17T09:35:55.412Z. Investigate this player and return your verdict.
Flag:
{
"flagId" : "7c1e...",
"playerId" : "3f2a...",
"rule" : "AIMBOT",
...
}
No label, no hint about whether the player is truly a cheater, ever appears in a prompt. UserPromptBuilderTest checks the words "cheater" and "honest" are absent.
5.5 Tools and the guard
ToolCatalog: four read-only tools
| Tool name | Returns (from Arena) | Inputs |
|---|---|---|
get_player_profile | status, country, account age, matches played, rating | player_id |
get_stat_percentiles | long-run stats and where the player ranks among all players | player_id |
get_recent_matches | per-match stats, newest first | player_id, optional limit (1 to 20) |
get_moderation_history | past moderation actions, newest first | player_id, optional limit |
Each is built with .strict(true) and additionalProperties: false so the model cannot invent arguments. Note the comment in the code: the range 1 to 20 for limit is not written into the schema (strict schemas do not support numeric ranges). It is stated in the description and enforced by code, which is the next class.
ToolExecutor and ToolResult
public interface ToolExecutor {
ToolResult execute(String toolName, JsonNode input, UUID flaggedPlayerId);
}
public record ToolResult(String content, boolean isError) { ok(...), error(...) }
Three implementations will exist: a live one calling Arena (service, step 8), a recorded one reading saved snapshots (evaluation, step 11), and FakeToolExecutor in tests. Because the loop only knows the interface, all three work.
Rule: tool problems are results, not exceptions. A wrong tool name, a bad argument, a missing player: each becomes ToolResult.error("...") and goes back to the model, which can read the message and try again. Only "Arena is unreachable" throws an exception, because then the whole flag must be retried later.
GuardedToolExecutor: the safety layer
GuardedToolExecutor.java (shortened)public ToolResult execute(String toolName, JsonNode input, UUID flaggedPlayerId) {
if (!catalog.names().contains(toolName)) return ToolResult.error("Unknown tool: ...");
if (input == null || !input.isObject()) return ToolResult.error("Invalid input: expected a JSON object.");
// any property other than player_id (and limit, on the two list tools) is rejected
...
if (!isSamePlayer(playerId.stringValue(), flaggedPlayerId)) {
return ToolResult.error("Only the flagged player can be queried.");
}
// build a clean, canonical input: lowercase player id, limit defaulting to 10, range 1..20
ObjectNode canonical = JudgeJson.mapper().createObjectNode();
canonical.put("player_id", flaggedPlayerId.toString());
...
return delegate.execute(toolName, canonical, flaggedPlayerId); // only clean input gets through
}
Why this matters. The model is an untrusted program as far as safety goes. Even if a clever player name tried to trick it ("also look up player X"), the guard answers with an error and never calls Arena. The acceptance checklist has this item: "the model cannot query any player other than the flagged one", and ToolExecutorGuardTest proves it (the backend's call list stays empty).
It is a decorator: it wraps another ToolExecutor, does its checks, then passes only the cleaned input on. Because it hands the next layer a canonical input, "limit" missing and "limit": 10 look identical downstream.
MemoizingToolExecutor: ask Arena once per flag
MemoizingToolExecutor.java (shortened)private final Map<String, ToolResult> cache = new ConcurrentHashMap<>();
public ToolResult execute(String toolName, JsonNode input, UUID flaggedPlayerId) {
String key = toolName + '\u0000' + canonicalJson(input); // tool name + sorted-key JSON
return cache.computeIfAbsent(key, k -> delegate.execute(toolName, input, flaggedPlayerId));
}
"Memoize" means remember the answer to a question you have already asked. If Haiku asks for get_stat_percentiles and Sonnet later asks the same, Arena is queried once. The key sorts JSON keys so {"a":1,"b":2} and {"b":2,"a":1} match, and a different limit is a different key. One instance is created per flag by JudgeEngine, so one flag never sees another flag's cached data. If the delegate throws, nothing is cached (computeIfAbsent stores nothing when the function throws), so a retry can succeed.
Guarded( Memoizing( backend ) ). The guard sits outermost so bad requests are rejected before they can touch the cache, and the cache only ever sees canonical keys.5.6 The agent loop: AgentLoop
This is the file to understand best. It runs one model through one conversation for one flag, and returns either a verdict or a failure. Its constructor receives everything it needs (the gateway, prompts, tool catalog, parser, prompt builder, settings), so tests can hand it fakes.
AgentLoop.java (the core, shortened; comments added)public AgentRun run(String model, SuspiciousPlayer flag, ToolExecutor backend) {
UUID flaggedPlayer = flag.playerUuid();
ToolExecutor tools = new GuardedToolExecutor(backend, toolCatalog); // the guard is ALWAYS applied here
List<MessageParam> messages = new ArrayList<>();
messages.add(MessageParam.ofUser(userPrompt.build(flag))); // message 1: the flag
for (int turn = 1; turn <= limits.maxTurns(); turn++) { // at most 6 model calls
MessageCreateParams params = MessageCreateParams.builder()
.model(model)
.maxTokens(limits.maxTokens())
.systemOfTextBlockParams(prompts.systemBlocks(settings.caching().enabled()))
.tools(toolCatalog.toolUnions())
.messages(List.copyOf(messages)) // a snapshot, see below
.outputConfig(VerdictSchema.outputConfig())
.build();
Message response = gateway.create(params); // the network call (resilient)
usage = usage.plus(TokenUsage.fromSdk(response.usage()));
messages.add(response.toParam()); // remember what the model said
StopReason stop = response.stopReason().orElse(null);
if (StopReason.TOOL_USE.equals(stop)) {
List<ContentBlockParam> results = new ArrayList<>();
for (ContentBlock block : response.content()) { // the model may ask for SEVERAL tools
Optional<ToolUseBlock> use = block.toolUse();
if (use.isEmpty()) continue;
if (++toolCalls > limits.maxToolCalls()) throw new TurnLimitExceededException("tool calls");
ToolResult result = tools.execute(use.get().name(), toJsonNode(use.get()._input()), flaggedPlayer);
results.add(ContentBlockParam.ofToolResult(ToolResultBlockParam.builder()
.toolUseId(use.get().id()).content(result.content()).isError(result.isError()).build()));
}
messages.add(MessageParam.ofUserOfContentBlockParams(results)); // ALL results in ONE message, in order
continue; // ask the model again
}
if (StopReason.END_TURN.equals(stop)) {
String json = ...all text blocks joined...;
return new AgentRun(verdictParser.parse(json), stats(...)); // parse + validate, or throw
}
throw new InvalidVerdictException(Kind.VALIDATION, "stop reason " + ...); // max_tokens, refusal, ...
}
throw new TurnLimitExceededException("turns"); // 6 calls and still no answer
}
Walk through it in plain words
- Set up. Wrap the tools in the guard. Start the conversation with one user message: the flag.
- Loop up to 6 times. Each time, build a full request. The conversation grows by appending to
messages.List.copyOfsends a snapshot, so later additions cannot change a request already made. A test (earlierRequestsAreNotChangedByLaterTurns) exists purely for this. - Send it through the gateway. Add the token counts to the running total. Append the assistant's reply to the conversation unchanged: the API requires that its
tool_useblocks stay in the history before the results. - Look at why it stopped.
tool_use: it wants tools. Run each (one response can contain several requests). Count them; past 8 total, give up. Pack all results into one user message, in the order requested, and loop.end_turn: it is done. Join the text, parse and validate it, return.- anything else (
max_tokens: answer cut off;refusal): no usable verdict, throw.
- Ran out of turns: throw
TurnLimitExceededException.
Details worth knowing
- The guard is applied here, not left to the caller. Whoever calls
AgentLoop.runwith whatever executor, the model still cannot query another player. Safety by construction. - Failures carry their cost. At the bottom a
try/catchcatchesInvalidVerdictExceptionandTurnLimitExceededExceptionand rethrows them withAgentStatsattached (tokens, turns, tool calls, stop reasons). Reason: a failed attempt still costs money and the router must record it. - Gateway errors are not caught. If Claude is unreachable, the exception propagates all the way up so the entire flag is retried later. Contrast: a bad verdict is the model's failure (try Sonnet); an unreachable API is an infrastructure failure (try again later).
- Type conversions. The SDK gives tool input as its own
JsonValue(Jackson 2 inside); the project uses Jackson 3.toJsonNodeconverts through a string. That is marked with a// DECISIONcomment, a convention of this project: wherever the code deviates from the spec or picks between options, it says so in the source. - Why
MessageParam.ofUser(...)and not a builder? The spec's example used a builder that does not exist in SDK version 3.10.0. This was discovered by inspecting the real library withjavap, and recorded inhandoff.md. A real example of "do not trust the doc, check the library".
What AgentLoopTest proves (21 tests)
It uses ScriptedGateway: a fake that plays back prepared model replies from a queue and records every request it received. TestMessages builds real SDK Message objects by parsing API-shaped JSON. So the tests walk the real loop code with no network. Cases include: one tool round then a verdict; two tools in one turn get two results in one message in order; a tool error comes back with isError; the model asking about another player gets the guard's error and the backend is never called; max_tokens, refusal and unparsable text all fail with the right kind; the turn limit and tool-call limit; token usage summed across turns with missing cache fields treated as 0; and the request itself carries the model, token limit, four tools, the schema and the cache marker.
5.7 Resilience: ResilientMessagesGateway and AnthropicResilience
MessagesGateway is a one-method interface: Message create(MessageCreateParams params). The loop only knows this interface, so tests can substitute ScriptedGateway. The production implementation wraps the real Anthropic SDK client in the three safety devices from 2.6.
public Message create(MessageCreateParams params) {
Supplier<Message> call = () -> send(params); // the real HTTP call
Supplier<Message> limited = RateLimiter.decorateSupplier(rateLimiter, call);
Supplier<Message> guarded = CircuitBreaker.decorateSupplier(circuitBreaker, limited);
Supplier<Message> retried = Retry.decorateSupplier(retry, guarded); // retry is OUTERMOST
try {
return retried.get();
} catch (BadRequestException e) {
throw new NonRetryableJudgeException("anthropic rejected request", e); // our request is wrong
} catch (CallNotPermittedException | RequestNotPermitted | AnthropicServiceException
| AnthropicIoException e) {
throw new LlmUnavailableException(e); // hold the flag, retry later
}
}
Why retry is outermost
Read the nesting from the inside out: call is wrapped by the limiter, wrapped by the breaker, wrapped by the retry. When retried.get() runs, retry calls the breaker; the breaker calls the limiter; the limiter calls the network. Because retry is on the outside, every retry attempt passes through the breaker and the limiter again. So retried calls count toward the rate limit, and when the breaker opens, the next retry attempt is refused immediately (and an open breaker is not retried). If the order were reversed, a single retried call would count as one call to the breaker no matter how many attempts it took, hiding the outage.
The settings (AnthropicResilience)
| Device | Setting | Detail |
|---|---|---|
| Retry | 4 attempts, 1 s start, x2, jitter 0.5, cap 20 s | Retries only RateLimitException (429), InternalServerException (5xx, including 529 overloaded) and AnthropicIoException (network). A 429 with a Retry-After header waits at least that long (capped at 60 s). A 400 or 401 is not retried: waiting will not fix it. |
| Circuit breaker | window of 20 calls, needs 10, opens at 50% failures, open 30 s, 3 trial calls | Counts only 5xx and network errors as failures. 400, 401, 403 and 429 do not count: they do not mean the service is down. |
| Rate limiter | 120 calls per minute, wait up to 60 s for a slot | One limiter shared by all threads. |
static boolean isRetryable(Throwable t) {
return t instanceof RateLimitException || t instanceof InternalServerException
|| t instanceof AnthropicIoException;
}
Two details that interviewers like
maxRetries(0)on the SDK client. The Anthropic SDK retries by itself (default 2 retries). If you also retry with Resilience4j, one failure triggers retries inside retries: 3 times 4 equals 12 attempts, hammering an overloaded service. So the SDK's own retries are turned off and Resilience4j is the only retry layer.- Two exception types come out, deliberately.
NonRetryableJudgeExceptionmeans "do not try again, send it to the dead-letter topic".LlmUnavailableExceptionmeans "hold the flag and try again in 5 seconds, forever". Bad credentials (401, 403) are the second kind on purpose: when someone fixes the key, processing resumes with nothing lost.
How it was tested without the real API
ResilientMessagesGatewayTest starts a tiny local web server (StubAnthropicServer, built on the JDK's own HttpServer) that returns scripted replies: 429, 529, 500, a dropped connection, a slow answer, 400, 401. The real SDK client talks to it, so the tests prove how the SDK really maps HTTP errors to exceptions. It confirmed, for example, that 529 arrives as OverloadedException, a subclass of InternalServerException, so the retry rule covers it. Other tests prove the breaker opens after enough failures, makes the next call fail fast without sending a request, closes again after the service recovers, and is not opened by 429s.
5.8 Cost: PriceTable and CostCalculator
Every attempt is priced from config, so cost per verdict is a measured number, not an estimate. The formula (USD, with token counts divided by one million):
cost = (input * p_in + cacheWrite * p_cw + cacheRead * p_cr + output * p_out) / 1,000,000
CostCalculator.java
public BigDecimal cost(String model, TokenUsage usage) {
JudgeSettings.Price p = prices.priceOf(model);
BigDecimal total = p.input().multiply(BigDecimal.valueOf(usage.input()))
.add(p.cacheWrite().multiply(BigDecimal.valueOf(usage.cacheWrite())))
.add(p.cacheRead().multiply(BigDecimal.valueOf(usage.cacheRead())))
.add(p.output().multiply(BigDecimal.valueOf(usage.output())));
return total.divide(MILLION, 6, RoundingMode.HALF_UP);
}
Why BigDecimal and not double? Money must not accumulate floating-point error. 0.1 + 0.2 is not exactly 0.3 in double; BigDecimal is exact. Six decimal places because a Haiku call costs fractions of a cent.
Important detail: the API's input_tokens counts only the tokens after the last cache marker. Total input is cacheRead + cacheWrite + input, and each part has its own price. CostCalculatorTest checks hand-computed values for both models, including a case that rounds half up.
5.9 Routing: ModelRouter and EscalationRule
The idea in one paragraph
Haiku is about twenty times cheaper than Sonnet. Most flags are easy, so let Haiku handle everything first. Escalate to Sonnet only when Haiku is unsure, fails, or wants to ban a top-tier player (where a mistake is costliest). Sonnet starts a fresh conversation and does not see Haiku's answer, otherwise it would tend to just agree. Sonnet's verdict is final: no third attempt.
When does it escalate? (EscalationRule)
| Reason | Condition | Where it comes from |
|---|---|---|
INVALID | Haiku threw InvalidVerdictException | the router's catch block |
TURN_LIMIT | Haiku threw TurnLimitExceededException | the router's catch block |
LOW_CONFIDENCE | Haiku is unsure (see the box below) | EscalationRule |
HIGH_TIER_BAN | decision is BAN and the player is PLATINUM or PRO | EscalationRule |
THIN_EVIDENCE | decision is BAN with fewer than 2 evidence items | EscalationRule |
The first match wins and is recorded on the attempt, which feeds the judge.escalations metric.
confidence < 0.85. But confidence is the probability of cheating, and the prompt tells the model to answer below 0.50 when the player is probably innocent. Read literally, every CLEAR (confidence about 0.05) and every WATCH would escalate to Sonnet, and the "65% cheaper" goal would be impossible. So the code treats a verdict as unsure only when the cheating probability sits between 0.15 and 0.85:
static double certainty(double confidence) {
return Math.max(confidence, 1.0 - confidence); // how sure, whichever way it points
}
// escalate when certainty < 0.85
So BAN at 0.85 stays; BAN at 0.84 escalates; CLEAR at 0.15 stays; CLEAR at 0.16 escalates. This is a judgment call flagged with // DECISION and in handoff.md; switching to the literal rule is one line, and the evaluation can compare the two. In an interview, say exactly that: you found an ambiguity, you reasoned about cost, you documented it and made it testable.The router's flow (ModelRouter.investigate)
if triage is disabled: run ONLY Sonnet (the cost baseline)
try:
run = AgentLoop.run(Haiku)
if escalation disabled or rule says "no": record FINAL, done
else: record ESCALATED with the reason
catch InvalidVerdictException: record INVALID (and the reason INVALID)
catch TurnLimitExceededException: record TURN_LIMIT
(if escalation is disabled here: no verdict at all)
run AgentLoop.run(Sonnet) -> FINAL (its verdict is the answer) or INVALID / TURN_LIMIT (no verdict)
Each attempt becomes an Attempt record: model, outcome (FINAL, ESCALATED, INVALID, TURN_LIMIT), token stats, cost (from CostCalculator), latency (from the injected Clock, never Instant.now(): a project rule that makes time controllable in tests) and a human-readable reason. The result is an Investigation.
Modes. routing.triage-enabled=false gives the Sonnet-only baseline used for the cost comparison. routing.escalation-enabled=false gives haiku-only. The second key was added by this project because the spec's evaluation needs "no escalation" but its config has no switch for it.
ModelRouterTest (24 tests) has one test per escalation reason, boundary tests for the confidence rule, the proof that Sonnet's first request has exactly one user message, proof that both attempts share one tool cache (the backend is hit once even though both models asked), and cost and latency checks with hand-computed numbers.
5.10 The decision policy: DecisionPolicy
This is plain Java with no AI. It is a pure function: the same inputs always give the same output, nothing is read from the outside world. That makes it trivial to test exhaustively, and it is the answer to "what if the model misbehaves?"
DecisionPolicy.java (the decision rows, simplified into pseudo-code)if (v == null)
return FLAG, human review, reason "Judge could not reach a verdict"; // row 1
if (v.decision() == Decision.BAN) {
if (v.confidence() >= autoBanMinConfidence && !v.needsHumanReview())
return BAN, with confidence (3 decimals); // row 2
return FLAG for human review, with confidence; // row 3
}
if (v.decision() == Decision.WATCH) {
if (v.needsHumanReview()) return FLAG for human review; // row 4
return NOTE; // row 5
}
// CLEAR
if (status == PlayerStatus.FLAGGED) return CLEAR; // row 6
return NOTE "Cleared by Judge: ..."; // row 7
| # | Final verdict | Arena request | Recorded as |
|---|---|---|---|
| 1 | none (both attempts failed) | FLAG, needs review | FLAG_FOR_REVIEW |
| 2 | BAN, confidence ≥ 0.90, no review wanted | BAN | BAN |
| 3 | any other BAN | FLAG, needs review | FLAG_FOR_REVIEW |
| 4 | WATCH with review | FLAG, needs review | FLAG_FOR_REVIEW |
| 5 | WATCH | NOTE | NOTE |
| 6 | CLEAR and player is FLAGGED | CLEAR | CLEAR |
| 7 | CLEAR | NOTE | NOTE |
Arena's state machine
Arena only allows certain moves: FLAG only from ACTIVE or REINSTATED, CLEAR only from FLAGGED, BAN from ACTIVE, FLAGGED or REINSTATED, NOTE from anything. So after choosing, adjustForState fixes the request. If the policy wants to FLAG a player who is already FLAGGED, it sends a NOTE with requiresHumanReview true instead, because Arena's moderator queue shows each player's latest action. That is why there are two enums: the kind is NOTE (what Arena receives) but the label stays FLAG_FOR_REVIEW (what the Judge records), so the human-review rate is counted honestly.
Why 0.90, when Arena only needs 0.80?
Defense in depth. Arena's 0.80 floor protects against a bug in any client. The Judge's own 0.90 is stricter because a false ban is the worst outcome. Two independent safeguards means one bug is not enough to hurt an innocent player. noBanBelowTheArenaFloorEverIsProduced in DecisionPolicyTest loops confidence from 0.00 to 0.89 and asserts that no BAN is ever produced. The exact boundary is tested too: 0.90 bans, 0.899 flags for review, and 0.8996 does not auto-ban even though it rounds to 0.900 (the rule uses the raw number, rounding is only for what is sent).
5.11 JudgeEngine: the front door
JudgeEngine.java (shortened)
public static JudgeEngine create(JudgeSettings settings, MessagesGateway gateway, Clock clock) { ... } // wires everything
public Investigation investigate(SuspiciousPlayer flag, ToolExecutor tools) {
return router.investigate(flag, new MemoizingToolExecutor(tools)); // ONE cache per flag
}
public JudgeOutcome judge(SuspiciousPlayer flag, ToolExecutor tools) { // evaluation uses this
return judge(flag, tools, PlayerStatus.ACTIVE);
}
public JudgeOutcome judge(SuspiciousPlayer flag, ToolExecutor tools, PlayerStatus status) { // service uses this
Investigation investigation = investigate(flag, tools);
return new JudgeOutcome(investigation, decide(investigation, status));
}
public ArenaAction decide(Investigation investigation, PlayerStatus status) { // re-run after an Arena conflict
return policy.decide(investigation.verdict(), status, settings.promptVersion(), investigation.finalModel());
}
Both applications call this class. The service passes live Arena tools; the evaluation passes recorded ones. Everything else in the decision path is shared, so the evaluation measures exactly the code that runs in production. This is the answer to "how do you know your accuracy numbers are real?"
Notice decide is separate and does not call a model. If Arena answers "409 conflict" (another moderator acted in between), the service re-reads the player's status and calls decide again: no extra tokens spent.
5.12 Arena tools in the core: ArenaReader, ArenaToolExecutor
The four tools the model can call are answered by Arena, the game backend. Two programs need those answers: the live service (to judge real flags) and the evaluation's collect command (to record answers once, for later replay). If each wrote its own code to turn Arena's big responses into the small JSON the model sees, the recorded answers could differ from the live ones and the evaluation would measure the wrong thing. So that code lives once, in the core.
| Type | What it is |
|---|---|
ArenaReader (interface) | Three questions you can ask Arena: the player, their recent matches, their moderation history. An empty result means "404, no such player". It does not say how to talk to Arena. The service answers with a fancy client (retry, circuit breaker); collect answers with a simple one. This is called dependency inversion: the core defines what it needs, and the outside world plugs in. |
ArenaToolExecutor | Implements ToolExecutor on top of an ArenaReader. It makes the compact results: the profile (username, status, account age in days, rating), the percentiles (with a note when the player has fewer than 5 matches), matches (headshot rate and accuracy computed to 3 decimals, never dividing by zero), history (reason cut to 200 characters). It drops private fields such as email and display name, which the model never needs. |
ArenaUnavailableException | "Arena could not be reached." The only exception that aborts a whole investigation: the flag is held and retried, never judged on missing data. |
AttemptRecord | One model attempt flattened into plain numbers (model, turns, tokens, cost, latency). It appears on every judge-verdicts record and in every evaluation result, so it lives in the core and both applications reuse it. |
GET /players/{id} to see whether the player is banned. The executor receives that response in its constructor, so get_player_profile and get_stat_percentiles are answered without a second call. Arena is read once per player for those two tools.Where this appears in a test: ArenaToolExecutorTest checks that the profile has exactly the documented fields (and no email), that missing percentiles become null plus a note, that a match with zero kills has a headshot rate of 0 instead of a crash, and that an unreachable Arena throws.
6. judge-service built
The Spring Boot application that connects the core to the outside world: it reads flags from Kafka, asks the core for a verdict, acts in Arena and publishes the verdict. It is tested against a real Kafka broker and a WireMock container that pretends to be both Claude and Arena.
The classes
| Class | What it does |
|---|---|
ArenaClient | Logs in as judge-agent, caches the token, logs in again once on a 401. Reads the player, matches and moderation history, and posts the moderation action. Has its own retry and circuit breaker. Implements the core's ArenaReader. |
ArenaActionApplier | Sends the chosen action to Arena. On a 409 (another moderator acted first) it re-reads the status and re-runs engine.decide once; a second 409 is recorded as a NOTE with no Arena action. |
PreChecks | Before any tokens are spent: unknown player or not role PLAYER, already BANNED or APPEALED. JudgeService adds the third check, judged in the last 30 minutes. |
JudgeService | One flag from start to verdict record: pre-checks, JudgeEngine.judge, then ArenaActionApplier. |
FlagListener | @KafkaListener: parse the flag (malformed means dead letter), skip it if already judged, judge it, then publish the verdict and wait for the broker to acknowledge. |
ProcessedFlags, ProcessedFlagsLoader, VerdictSnapshotReader | At startup, replay the compacted judge-verdicts topic into memory, and only then start the listener. There is no database: that topic is the Judge's only state. |
KafkaConfig | Declares the two topics the Judge writes, and the error handler: retry forever every 5 seconds for outages, dead letter only NonRetryableJudgeException. |
JudgeMetrics | Prometheus counters, timers and histograms: verdicts, tokens, cost, latency, escalations, invalid outputs. |
AnthropicConfig, ArenaConfig, StartupChecks | Build the one SDK client (maxRetries(0)), the resilience stack and the engine. The service refuses to start if ANTHROPIC_API_KEY or the Arena password is missing, and says which, never the value. |
application.yml | Kafka consumer and producer settings, port 8083, Arena URL, topic names. Key line: max-poll-records: 1 with a long poll interval, because one flag can take 10 to 40 seconds of model time and Kafka must not think the consumer died. |
The integration tests
JudgeFlowIT: a flag gives one verdict record and exactly one Arena action withsource JUDGE; replaying the flag adds no call; a banned player is skipped without a model call; a malformed flag lands in the dead-letter topic; the Prometheus endpoint exposes the meters the dashboard uses.ProcessedFlagsIT: a verdict already on the topic is loaded before the listener starts, so that flag is not judged again.ResilienceIT: 429, 529, a slow answer, a connection reset, and a 400 (dead letter, no retry). The outage test sends 20 flags during 20 seconds of HTTP 500: the circuit breaker opens, no flag gets a verdict or reaches the dead-letter topic meanwhile, and after the outage all 20 are judged.
Following one flag through the service, slowly
Imagine Telemetry just put this message on the topic suspicious-players: "player sim_0042 looks like an aimbot". Here is exactly what happens, in order.
- Kafka hands the message to
FlagListener. It parses the JSON. If the text is not valid JSON, or has no flag id or player id, it throwsNonRetryableJudgeException, and the error handler copies the message to the dead-letter topic. There is no point retrying garbage. - "Have I judged this flag before?"
ProcessedFlagsis a map of flag ids already answered. If the id is there, the listener counts a duplicate and returns. This is what makes a restart safe. JudgeServiceasks Arena about the player (one call). Not a normal player, or already banned? Write aSKIPPEDverdict and stop: no tokens spent. Judged within the last 30 minutes and then banned or flagged? Also skip.- The engine investigates. Haiku calls tools (answered by
ArenaToolExecutor), then returns a verdict. Maybe Sonnet takes a second look. DecisionPolicypicks one Arena action andArenaActionAppliersends it withsource: JUDGE. If Arena says 409 ("the player's state changed while you were thinking"), re-read the status and decide once more.- Only now does the listener publish the verdict record to
judge-verdictsand wait for the broker to say "stored". Then it remembers the flag id. - Kafka's offset moves on. The offset is Kafka's bookmark. It is saved only after the method returns without an exception, so a crash anywhere above means the same flag comes back after the restart.
Why an outage does not lose flags
If Claude or Arena is unreachable after the retries, JudgeService throws. The listener does not catch it. The error handler waits 5 seconds and offers the same message again, forever, without moving the bookmark. Messages behind it in that partition wait. That is intended: it keeps order per player and drops nothing. ResilienceIT proves it: 20 flags during a simulated 20-second outage, none judged and none dead-lettered while it lasts, and all 20 judged after it ends.
The breaker, in numbers from the test
| Moment | What the circuit breaker does |
|---|---|
| First 10 calls fail with HTTP 500 | Still closed: it needs at least 10 calls in its window of 20 before judging the failure rate. |
| Failure rate reaches 50% | Switches to open: further calls fail instantly without touching the network (the service is not hammered while it is sick). |
| After the open time (2 s in tests, 30 s in production) | Half-open: lets 3 trial calls through. If they succeed it closes; if not, it opens again. |
The exactly-once question
Kafka delivers at least once. If the service dies after sending the Arena action but before publishing the verdict, the flag is judged again. The second time the pre-check sees the player is already BANNED and records SKIPPED, or Arena answers 409 and the policy's retry rule handles it. The worst case is a duplicated NOTE in an audit trail, which is acceptable. This trade-off is meant to be written in the README: a good interview answer is "I chose at-least-once plus idempotent handling instead of chasing exactly-once".
7. judge-eval code built, no data yet
A command-line application that answers: "is this prompt actually good, and what does it cost?" It does not call Arena while judging; it replays recorded data. The code and its tests are finished. The dataset and the reports do not exist yet, because they need a finished Telemetry run and an API key (needs you, see evals/README.md).
collect(DatasetCollector,CaseSampler): build a dataset of 200 labeled cases (about 120 cheaters, 80 honest-but-flagged players, including pros). The trick: run Telemetry once with loosened thresholds so strong honest players also get flagged, because a normal run produces too few hard negatives. For each case, store the flag and one saved answer for each of the four tools, made by the sameArenaToolExecutorthat answers the model in production. Labels never appear in the flag or the saved data.run --prompt=v1 --mode=routed --subset=all(EvalRunner): judge every case with the productionJudgeEngine, usingRecordedToolExecutorin place of Arena. Modes:routed(normal),baseline(Sonnet only, no cache),haiku-only. It writes a.jsonlwith every case, a.summary.jsonand a Markdown report (ReportWriter,Scorer).compare(ComparisonWriter): put all runs side by side inCOMPARISON.md, with the cost saving split into routing and caching.
A run, step by step
run --prompt=v1 --mode=routed --subset=allloads the 200 cases and builds the settings for the mode (EvalMode):routedkeeps the shared defaults,baselineturns off Haiku and caching,haiku-onlyturns off escalation.EvalRunnerjudges every case on its own lightweight (virtual) thread, at most 4 at a time, all sharing one rate limiter. Each case gets aRecordedToolExecutorthat answers from the case's saved snapshots, cutting the two lists to the requested size.- The player is treated as
ACTIVEand the service's pre-checks are skipped: every case is judged. Everything else (JudgeEngine, routing,DecisionPolicy) is the production code. - A case where Claude stays unreachable is recorded as an error and left out of the scores. The report lists it so you can re-run.
Scorerturns the 200 results into numbers, andReportWriterwrites three files: every case (.jsonl), the numbers (.summary.json) and a readable report (.md) that ends with every false ban and every missed cheater, with the evidence the model gave. That last list is what you read to write the next prompt version.
ceil(0.95 * 200) = 190, so the 190th smallest value. No averaging between neighbours.The scoring terms
| Term | Meaning |
|---|---|
| Precision | Of the players the Judge banned, what share were real cheaters? (Low precision = innocent people banned.) |
| Recall | Of all the cheaters, what share were banned? (Low recall = cheaters get away.) |
| False-ban reduction | The "baseline" bans every flag, so its false bans equal the number of honest cases. The Judge's number is compared against that. |
| Review rate | Share of cases sent to a human. Costs moderator time; target about 12%. |
| Dev vs holdout | You edit prompts while looking at 50 "dev" cases. The other 150 ("holdout") are never looked at, so their numbers are honest. This prevents tuning the prompt to the test. |
Why replay recorded tool data? It is repeatable and cheap, and it measures the model and prompt instead of Arena's changing state. Why a holdout? Otherwise you would be marking your own homework.
8. The tests: how we know it works
Run everything with ./mvnw verify (no API key needed; Docker must be running, because the integration tests start Kafka and WireMock containers). It compiles, runs the tests, and fails if less than 85% of the lines of a module are executed by tests. Current numbers: 194 tests in core (97.2%), 77 unit and 12 integration tests in service (96.9%), 105 unit and 1 integration test in eval (96.2%).
Test helpers (shared as a "test-jar")
| Helper | What it is |
|---|---|
ScriptedGateway | A fake MessagesGateway. Queue model replies with .then(message) or failures with .thenFail(exception); afterwards read .requests() to see exactly what the code sent. |
TestMessages | Builds SDK Message objects from API-shaped JSON: toolUse(...), finalText(...), stoppedWith("max_tokens", ...), verdictJson(...). |
StubAnthropicServer | A local HTTP server impersonating the Claude API with scripted replies (429, 529, reset, slow...). |
FakeToolExecutor | Returns fixed results per tool and records every call. |
MutableClock | A clock tests can set and advance, so latency is a number you control. |
TestSettings | The default settings built in Java, shared by tests. |
What each test class proves
| Test class | Proves |
|---|---|
JudgeSettingsTest | Each validation rule rejects bad values (ban floor 0.80, confidences 0 to 1, positive durations, both models priced, prompt files present). |
VerdictParserTest | Valid JSON parses; lower-case enums work; extra text, fences, missing and extra fields, wrong types all fail as PARSE. |
VerdictValidatorTest | Every rule at its boundary. |
VerdictSchemaTest | The schema matches the contract; every object forbids extra properties. |
PromptLibraryTest, UserPromptBuilderTest | Cache marker only when caching is on; no dates or ids in cached text; no labels in prompts. |
ToolCatalogTest | Four tools, exact names, strict, player_id required, only two take limit. |
ToolExecutorGuardTest | Another player's id never reaches the backend; unknown tools, bad input and bad limits are error results. |
MemoizingToolExecutorTest | Same question asked once; a different limit asks again; exceptions are not cached. |
AgentLoopTest | The loop's behaviour in 21 scenarios (section 5.6). |
AnthropicResilienceTest | Retry-After parsing, which exceptions retry, config values, jitter bounds. |
ResilientMessagesGatewayTest | Real SDK client against the stub server: retries, breaker open and recovery, rate limiter, non-retryable errors. |
CostCalculatorTest | Hand-computed costs for both models, with and without cache tokens. |
ModelRouterTest, JudgeEngineTest | Every escalation reason, fresh Sonnet conversation, shared tool cache, baseline modes, costs and latency; the engine's entry points. |
DecisionPolicyTest | Every row of the table, the 0.90 boundary, the state-machine adjustments. |
JudgeConfigBindingTest | The real YAML files bind to the Java records; env var overrides work. |
SdkSmokeTest | (tag live, excluded from verify) One real round trip to Claude. Not yet run. |
How to read a test
Every test has three parts, often called arrange, act, assert. Here is a real one:
ModelRouterTest.java@Test
void lowConfidenceEscalatesToSonnetWhichGivesTheFinalVerdict() {
gateway.then(TestMessages.finalText(ban(0.78, 2))) // ARRANGE: Haiku says 0.78
.then(TestMessages.finalText(TestMessages.verdictJson("BAN", "AIMBOT", 0.96, false, 3))); // then Sonnet says 0.96
Investigation result = investigate(flag(SkillTier.GOLD)); // ACT
assertThat(result.verdict().confidence()).isEqualTo(0.96); // ASSERT
assertThat(result.attempts()).extracting(Attempt::model).containsExactly(HAIKU, SONNET);
assertThat(result.attempts()).extracting(Attempt::outcome).containsExactly(Outcome.ESCALATED, Outcome.FINAL);
assertThat(result.attempts().get(0).escalation()).isEqualTo(EscalationReason.LOW_CONFIDENCE);
assertThat(gateway.requests()).extracting(ModelRouterTest::modelOf).containsExactly(HAIKU, SONNET);
}
The test name is a sentence describing the behaviour. Reading just the method names of a test class tells you what the class is supposed to do, which is why you read tests right after the code.
Why the real API is never called in tests
Tests must be free, fast, repeatable and runnable in CI without secrets. A model's answers vary between runs and cost money. So every test replaces the model with a script. The one exception is SdkSmokeTest, tagged @Tag("live") and excluded from verify. It exists to confirm the real request shape is accepted. Be honest about this limit: the fakes only check my understanding of the API, which is exactly why the live test still matters.
9. Build files
The parent pom.xml
- Parent:
spring-boot-starter-parent 4.1.2, the same patch version as Arena and Telemetry, so versions of shared libraries agree. - Pinned versions:
anthropic-java 3.10.0,resilience4j 2.4.0(one property for all four Resilience4j libraries), JaCoCo 0.8.13. - Shared test libraries for every module: Spring's test starter (JUnit, AssertJ, Mockito), Testcontainers, Awaitility.
- JaCoCo measures which lines tests executed, and the build fails under 85% per module (
jacoco.minimum). The parent skips it (jacoco.skip=true) because it holds no code; each child switches it on. - Surefire runs unit tests; Failsafe runs integration tests named
*ITinverify. Thelivetests are excluded through the propertytest.excludedGroups(see section 11 for why a property).
The module POMs
| Module | Depends on | Notable |
|---|---|---|
judge-core | Anthropic SDK, Jackson 3, Resilience4j (retry, breaker, limiter), Micrometer core, slf4j-api | No Spring, no Kafka. Builds a test-jar so other modules reuse its test helpers. No spring-boot-maven-plugin: repackaging would make it unusable as a dependency. |
judge-service | core, Spring MVC, Kafka, Actuator, Prometheus, resilience4j-micrometer | Has the Spring Boot plugin, so it builds a runnable jar. |
judge-eval | core, Kafka, Spring RestClient | Runnable jar; no web server. |
Everyday commands
./mvnw verify # build, test, coverage check (all modules)
./mvnw -pl judge-core test # one module's tests
./mvnw -pl judge-core test -Dtest=AgentLoopTest # one test class
./mvnw -pl judge-core test -Dgroups=live -Dtest.excludedGroups=none # the live smoke test (needs ANTHROPIC_API_KEY)
On Windows PowerShell use .\mvnw.cmd instead of ./mvnw.
10. One flag, with real numbers
This worked example uses prices straight from judge-defaults.yml. The token counts are illustrative numbers I chose to show the arithmetic, not measurements. The exact method is the one in CostCalculator.
The flag: player sim_0042, tier GOLD, rule AIMBOT, 23 shots, 17 hits, 14 headshots (rate 0.8235), average reaction 61 ms.
Case A: Haiku is confident, no escalation
| Model call | What happens | input | cache read | output |
|---|---|---|---|---|
| 1 | Asks for get_player_profile and get_stat_percentiles | 450 | 3,000 | 90 |
| 2 | Asks for get_moderation_history | 1,400 | 3,000 | 80 |
| 3 | Answers: BAN, AIMBOT, confidence 0.95, 3 evidence items | 1,900 | 3,000 | 200 |
| Total | 3,750 | 9,000 | 370 |
Haiku cost = (3,750 * 0.10 + 9,000 * 0.01 + 370 * 0.50) / 1,000,000
= (375 + 90 + 185) / 1,000,000
= $0.000650
EscalationRule: certainty = max(0.95, 0.05) = 0.95, not below 0.85. Decision is BAN, but GOLD is not top tier, and 3 evidence items is at least 2. No escalation. Attempt outcomeFINAL.DecisionPolicy(player ACTIVE): BAN, 0.95 ≥ 0.90, no human review wanted, so row 2: sendBANto Arena with confidence 0.950 and reason[Judge v1, claude-haiku-5-5] ....
For comparison, imagine Sonnet doing the same work with no caching (every input token at full price, so 12,750 input tokens):
Sonnet cost, no cache = (12,750 * 2.00 + 370 * 10.00) / 1,000,000 = (25,500 + 3,700) / 1,000,000 = $0.029200
Haiku with cache = $0.000650 -> about 45 times cheaper for this flag
Sonnet with the cache would be (3,750*2.00 + 9,000*0.10 + 370*10.00) / 1,000,000 = $0.012100, so caching alone saves about 59% and routing the rest. This is exactly why the project plans separate runs (haiku-only, and routed with caching off) to split the saving into "routing" and "caching" parts. Treat these as arithmetic, not results.
Case B: Haiku is unsure, Sonnet is called
Same Haiku attempt, but it answers BAN with confidence 0.78.
- Certainty = max(0.78, 0.22) = 0.78, below 0.85: escalate with reason
LOW_CONFIDENCE. The Haiku attempt is recorded asESCALATED, reason "confidence 0.78 is below 0.85 in either direction", cost $0.000650 (still paid). - Sonnet starts a new conversation containing only the flag. It asks for the same tools, but they are answered from the shared cache: Arena is not called again.
- Say Sonnet uses input 2,350, cache write 3,000 (its own cache is cold), cache read 3,000, output 310:
Sonnet cost = (2,350 * 2.00 + 3,000 * 2.50 + 3,000 * 0.10 + 310 * 10.00) / 1,000,000 = (4,700 + 7,500 + 300 + 3,100) / 1,000,000 = $0.015600 - Sonnet answers BAN, 0.93, 3 evidence:
FINAL. Total for the flag: $0.000650 + $0.015600 = $0.016250. - Policy: 0.93 ≥ 0.90 so
BAN, reason[Judge v1, claude-sonnet-5-5] ...(the reason names the model that gave the final verdict).
Case C: a top-tier player
Same as A but the player is PRO. Haiku is confident (0.95) with 3 evidence items, yet the rule HIGH_TIER_BAN escalates anyway: pros legitimately look extreme, and banning one by mistake is costly, so a stronger second opinion is always taken.
Case D: everything fails
Haiku returns prose instead of JSON (INVALID, kind PARSE), Sonnet also fails. No verdict. Policy row 1: FLAG with human review, reason "Judge could not reach a verdict". The flag is not lost: a human sees it.
Case E: a whole run, scored optimal imaginary value
All numbers in this case are optimal imaginary values: what a good prompt version could reach on the 200 cases (120 cheaters, 80 non-cheaters). They are chosen so the arithmetic is consistent; they are not measured.
| Count (optimal imaginary value) | Cheaters (120) | Non-cheaters (80) |
|---|---|---|
| BAN | 112 | 4 |
| FLAG_FOR_REVIEW | 6 | 18 |
| NOTE or CLEAR | 2 | 58 |
banned = 112 + 4 = 116
precision = 112 / 116 = 96.6% (of those banned, how many were really cheating)
recall = 112 / 120 = 93.3% (of the cheaters, how many were banned)
false-ban cut = 1 - 4 / 80 = 95.0% (a rule-only baseline bans all 80, so it makes 80 false bans)
review rate = (6 + 18) / 200 = 12.0% (sent to a human moderator)
Notice what the 6 reviewed cheaters mean: they were not banned, so they count against recall, but a human will see them. That is the price of banning only when sure.
Case F: the cost split optimal imaginary value
The mean cost per verdict over 200 cases, from four runs of the same prompt version. All four values are optimal imaginary values, built from the same price list and token sizes as cases A and B.
| Run | Mean cost per verdict (optimal imaginary value) | Compared with the baseline |
|---|---|---|
| baseline (Sonnet only, no cache) | $0.0158 | - |
| routed, caching off | $0.0035 | 78% cheaper: this is the routing saving |
| routed, caching on | $0.0024 | 85% cheaper in total; 31% cheaper than routed without caching: this is the caching saving |
total saving = 1 - 0.0024 / 0.0158 = 84.8% (target: at least 65%)
routing saving = 1 - 0.0035 / 0.0158 = 77.8%
caching saving = 1 - 0.0024 / 0.0035 = 31.4%
The two parts do not add up to the total: they multiply (0.222 × 0.686 = 0.152 of the baseline cost remains, so 84.8% is saved).
11. Decisions and deviations from the written spec
An important habit: when reality disagrees with the plan, write down what you did and why. Each item below is marked // DECISION in the code or recorded in handoff.md. These are strong interview stories, because they show judgment.
| What | Spec said | What I did | Why |
|---|---|---|---|
| Low-confidence rule | escalate if confidence < 0.85 | escalate if max(c, 1-c) < 0.85 | The literal rule would escalate every CLEAR and WATCH, making the cost target unreachable (5.9). Still open for the owner to confirm. |
| Message construction | MessageParam.builder()... | MessageParam.ofUser(...) | The builder does not exist in SDK 3.10.0. Found with javap on the real jar. |
| JSON mapper access | JsonValue.jsonMapper() | ObjectMappers.jsonMapper() | Same: the spec's name does not exist. |
| Excluding the live test | <excludedGroups>live</excludedGroups> plus -Dgroups=live | exclusion through a test.excludedGroups property; live run passes -Dtest.excludedGroups=none | Surefire combines groups and excludedGroups with AND, so the spec's combination would run nothing. |
| Haiku-only mode | mode exists but no config key | new key routing.escalation-enabled | "Experiments change config, never code." |
limit range in tool schema | "integer 1 to 20" | range in the description, enforced by GuardedToolExecutor | Strict tool schemas do not support numeric range keywords. |
| The tool guard | described as rules | a decorator class, always applied inside AgentLoop | Safety by construction: no caller can forget it. |
| Failure cost | exceptions carry no data | exceptions carry AgentStats | A failed attempt still costs tokens; the cost record must be honest. |
| Settings binding | "bind to validated records" | validation in record constructors; Spring's Binder in the service | Keeps judge-core free of Spring; same style as Telemetry. |
| CLEAR note text | reason "starts with Cleared by Judge:" and also gets the [Judge ...] prefix | Cleared by Judge: [Judge v1, model] explanation | Satisfies both statements. |
| 408 and 409 from Claude | retry only 429, 5xx and network | followed the spec | The SDK would retry them by default; noted in handoff.md in case they appear in practice. |
12. Interview preparation
Interviewers for a project like this rarely ask you to recite code. They ask why, and then they follow up on your answer. The pattern that works: state the decision, give the reason, name the trade-off, and say how you checked it. Click each question to reveal a sample answer in plain words. Say it in your own words; do not memorise.
12.1 Your pitch
Adjust for what is true today. If you are interviewing before steps 8 to 12 are done, say "the core is built and tested; the service and evaluation are next". Never claim numbers you have not measured.
12.2 Design and architecture
Why does the model only recommend, and code decide the action?
Because a model's output is text produced by a probabilistic system, and the consequence here, banning someone, is serious and hard to undo. If the model could choose an HTTP call or a URL, a prompt injection or a plain mistake could do real damage. So the model returns a Verdict (data). DecisionPolicy is a pure function with seven rows that is trivially testable: a unit test loops confidence from 0 to 0.89 and asserts no BAN is ever produced. The trade-off is less flexibility: the model cannot do anything the policy does not allow, which is the point.
Why two models? Why does Sonnet start from scratch?
Cost and quality. Haiku is roughly twenty times cheaper per token, and most flags are easy. Sonnet is used only when Haiku is unsure, fails, or wants to ban a PLATINUM or PRO player, where a false ban hurts most. Sonnet gets a fresh conversation without Haiku's answer, because a model shown another model's conclusion tends to agree with it; an independent second opinion is worth more than a rubber stamp. The trade-off is that escalated flags cost both attempts. Both attempts share one tool cache so Arena is read once.
Why a threshold of 0.90 when Arena already enforces 0.80?
Defense in depth. Arena's floor protects the system from any buggy client. The Judge's stricter threshold reflects that a false ban is the worst outcome and a missed cheater is less bad, because the detector keeps watching. Two independent checks mean one bug is not enough. Startup validation also refuses a configured threshold below 0.80.
How do you make sure the model cannot look up other players?
GuardedToolExecutor checks every tool request: known tool name, object input, only allowed properties, and player_id must equal the flagged player, otherwise it returns an error result and never calls the backend. AgentLoop applies the guard itself, so no caller can skip it. It also canonicalises the input. A unit test asserts the backend's call list stays empty when another id is requested.
Why a hand-written loop instead of LangChain or Spring AI?
The agent loop is short, about 80 lines, and it is exactly where the hard parts live: several tool calls in one response must be answered in one message in order, limits on turns and tool calls, partial failures that still cost tokens, and different handling for model failures versus infrastructure failures. Owning it makes every behaviour testable and explainable and avoids a framework hiding retries or caching. The trade-off is I maintain more code, and I lose framework conveniences I did not need.
Where does state live? Do you have a database?
No database. The planned design uses the compacted Kafka topic judge-verdicts as the only state, replayed into memory at startup to know which flags are done. Fewer moving parts. The trade-off is memory proportional to the number of verdicts and a startup replay, fine at this scale.
What does "idempotent" mean for the Kafka listener and why do you need it?
Kafka delivers at least once: if the process dies before saving its position, the flag is delivered again. So handling a flag twice must be harmless. The design checks "already judged" first, pre-checks the player's status (a banned player is skipped), and Arena returns a conflict if the state already changed. The worst case is a duplicate NOTE in an audit trail. I chose at-least-once plus idempotent handling rather than trying to get exactly-once across Kafka and an HTTP API, which is not really achievable.
12.3 The LLM-specific questions
How does prompt caching work here, and what breaks it?
The API caches a prefix of the prompt, in the order tools, system text, then messages. I put everything identical across flags first: the four tool definitions, the role text, the rulebook, and one cache marker at the end of the rulebook. The flag and tool results come after, uncached. Cache reads cost about a tenth of normal input, so the roughly 3,000-token rulebook is nearly free after the first call. Anything that changes inside the cached part, such as a timestamp or id, invalidates it, so a unit test checks the system text has no dates or UUIDs. Also, the cached prefix must be at least 512 tokens on these models, and each model warms its own cache. I have not yet confirmed a cache hit on the live API.
What is structured output, and why validate again in code?
I send a JSON schema so the API constrains the answer to six fixed fields with allowed values. It guarantees the shape but not numeric ranges or lengths, which are only hints. So VerdictValidator enforces confidence in 0 to 1, 1 to 6 evidence items, text lengths, and the rule that BAN needs a cheat type. The parser is deliberately strict: extra text, markdown fences, missing or unknown fields are parse failures. Parse and validation failures are counted separately, so I can see which kind of problem a prompt version causes.
What happens if the model's answer is cut off or it refuses?
The stop reason tells me. Only end_turn carries a verdict. max_tokens, refusal or anything else becomes an InvalidVerdictException, the router escalates to Sonnet once, and if that also fails the policy sends the flag to human review instead of dropping it. The tokens spent on the failed attempt are still recorded and priced.
What if the model asks for several tools in one response?
All requested tools are executed and all results go back in one user message, in the same order, because the API requires every tool_use block to be answered in the next message. There is a test with two tools in one turn that checks order and message count. There is also a limit of 8 tool calls and 6 model turns per attempt so the loop cannot run forever or cost unboundedly.
How do you know the model is not just guessing?
That is what the evaluation harness is for (step 11: the code is built, but it has not run on real data yet): 200 labeled cases, with honest-but-flagged players as hard negatives, replayed through exactly the production code. Precision, recall and false bans are measured per prompt version, with a dev set used for tuning and a holdout set that nobody looks at while editing. Today I cannot give a measured accuracy, and I say so.
What is the "LOW_CONFIDENCE" decision you made?
The spec escalates when confidence is below 0.85. But confidence is the probability that the player cheats, so a confident innocent verdict has confidence near 0.05. Taken literally, every CLEAR and WATCH escalates and cost savings vanish. I escalate when the cheating probability is between 0.15 and 0.85, meaning max(c, 1-c) < 0.85. I documented it with a // DECISION comment, flagged it to the owner, and it is a one-line change to compare the literal rule in the evaluation.
How would you reduce cost further?
Measure first: the spec plans separate runs to split the saving into routing and caching. Then options: shorten the rulebook (tested as a prompt version), cache hits across more flags by keeping traffic steady inside the 5-minute cache window, return smaller tool results, lower the output token limit, and raise Haiku's confidence calibration so fewer flags escalate. The Batch API would help offline evaluation but not live moderation.
12.4 Resilience
Explain the order of retry, circuit breaker and rate limiter.
Retry is the outermost, so each attempt goes through the breaker and the limiter. A retried call therefore counts against the rate limit, and once the breaker opens, the next retry is refused immediately instead of hammering a failing service. If the breaker wrapped the retry it would see only the final outcome and miss the failing attempts.
Why maxRetries(0) on the SDK client?
The SDK retries by itself. If Resilience4j also retries, retries nest: roughly 3 times 4 attempts per failure, which multiplies load on an already struggling service. One retry layer, mine, is predictable and observable.
Which errors do you retry, and which do you not?
Retry: 429 (rate limited, honouring Retry-After), any 5xx including 529 overloaded, and network errors (timeouts, resets). Not retried inside the call: 400 (my request is wrong, it becomes a non-retryable exception and goes to the dead-letter topic), 401 and 403 (retrying in a tight loop will not fix a bad key; the flag is held and retried every few seconds so work resumes when someone fixes the key). Only 5xx and network errors count as failures for the circuit breaker, because 429s and 4xx do not mean the service is down. I verified the real SDK's exception mapping against a local stub server.
What is jitter and why add it?
A random variation on the wait between retries. Without it, many clients that failed together retry together and cause another spike (a thundering herd). Resilience4j's exponential random backoff spreads them out.
Can a flag be lost?
By design, no. Temporary failures (Claude down, Arena down, circuit open) throw exceptions that the Kafka error handler retries forever without saving the offset. Only a malformed flag or a request Claude rejects as invalid goes to the dead-letter topic. If both models fail to produce a verdict, the flag goes to a human instead of disappearing. (The Kafka part is planned, step 9; the exception types and the "no verdict becomes a review flag" policy row exist now.)
12.5 Java and code quality
Why records everywhere?
They are immutable, concise and give equals, hashCode and toString for free, which makes tests simple: assertThat(a).isEqualTo(b). Compact constructors let me validate on construction so an invalid object cannot exist. Immutability also makes sharing across virtual threads safe.
Why inject a Clock instead of calling Instant.now()?
So time is controllable in tests. The router measures attempt latency from the injected clock; in tests a MutableClock advances 100 ms per model call so I can assert exact latencies without sleeping. It also makes time-based logic (like "judged in the last 30 minutes") testable.
Why is judge-core free of Spring and Kafka?
So the decision path is testable with plain JUnit, fast and without containers, and so the service and the evaluation can both depend on exactly the same logic. The service adapts the world (Kafka, HTTP) to the core's interfaces.
What is the decorator pattern, where do you use it?
An object that implements an interface by wrapping another implementation of the same interface and adding behaviour. GuardedToolExecutor adds safety checks, MemoizingToolExecutor adds caching, and the gateway's rate limiter, breaker and retry are decorators around a call. Each concern is separate and independently testable, and they compose.
Why BigDecimal for cost?
Binary floating point cannot represent most decimals exactly, so sums of tiny prices drift. BigDecimal is exact; I round once to six decimals at the end.
12.6 Testing
How do you test code that calls an LLM?
At three levels. For the loop and routing, a scripted fake gateway plays back prepared model replies and records the requests, so every branch is tested deterministically and for free. For the resilience stack, the real SDK client talks to a local stub HTTP server that returns 429, 529, resets and slow responses. Later, service integration tests use real Kafka and a WireMock container. Only one tagged smoke test touches the real API, run by hand. And what the fakes cannot prove, namely that the real API accepts my request, is exactly what that smoke test and the evaluation are for.
What does 97.9% coverage actually tell you?
That tests executed that share of lines. It does not tell you the assertions are good. I treat it as a floor (the build enforces 85%) and rely on specific behaviour and boundary tests: confidence 0.90 versus 0.899, 6 versus 7 evidence items, exactly-one-user-message in Sonnet's first request.
12.7 Honesty and "what next" questions
What would you do differently / what is missing?
The evaluation harness is built but has never run on real data, so no accuracy is measured yet. The live smoke test has not been run, so the real API request shape is unverified. The synthetic dataset will have its own limits: Arena's statistics are snapshotted after the simulation rather than at flag time. Next steps are collecting the dataset, running six prompt versions and measuring. I would also want to review cases where the model said BAN on a PRO player.
Tell me about a bug or surprise you hit.
The spec's code sample for building a message used a builder that does not exist in the SDK version I pinned. I found it by inspecting the real jar with javap before building on it, switched to the union factory methods, and recorded the difference. Another: combining the spec's exclusion of the live test with the way I was told to run it would have selected no tests at all, because Surefire ANDs the two filters; I moved the exclusion into a property.
How would you explain this project to a non-technical person?
"A fast alarm says a player might be cheating. My system plays the careful referee: it looks at that player's whole history, decides whether to ban, watch or clear, and only does the drastic thing when it is very sure. When it is not sure, it hands the case to a human."
12.9 Result questions (answers use optimal imaginary values)
These answers quote numbers. Every number is an optimal imaginary value from section 12b, not a measurement. When you have run the evaluation, replace each number with your real one and keep the sentence structure.
What precision and recall did you get, and on which data?
Say: "On the 150 holdout cases, which I never looked at while editing prompts, the best version reached about 96.6% precision and 93.3% recall (optimal imaginary values). Precision is how many of the bans were real cheaters; recall is how many cheaters got banned. I report holdout because dev numbers are optimistic: I read the dev mistakes to write each prompt version."
Why is recall lower than precision, and is that a problem?
Say: "It is by design. Banning an innocent player is the worst outcome, so the system only bans above 0.90 confidence without a human-review request. About 6 of 120 cheaters went to the review queue instead of being banned (optimal imaginary value): they count against recall, but a moderator sees them. I report the review rate separately so nobody thinks those are lost."
How did you get 85% cheaper?
Say: "Two independent savings, measured with four runs of the same prompt. Routing: Haiku handles most flags and Sonnet only the unsure ones, which was about 78% cheaper than Sonnet alone (optimal imaginary value). Caching: the tools, system prompt and rulebook are identical on every call, so after the first call they are read from cache at a tenth of the price, which cut the cost of the routed setup by another 31% (optimal imaginary value). They multiply rather than add, so the total was about 85%, from $0.0158 to $0.0024 per verdict (optimal imaginary values)."
What did the prompt iterations change, and did they help?
Say: "Six versions, one change each, each motivated by reading the false bans and missed cheaters in the dev report. For example, requiring the match history before any aim-based ban, and stricter wording for pro players. Precision went from about 91.5% to 96.6% (optimal imaginary values). One version, a shorter rulebook, was meant to lower cost and left accuracy unchanged, which is a result too."
Is 100% valid output believable?
Say: "Two layers make it believable: the API is told the JSON schema and enforces the structure, and my parser then rejects anything else, including extra text or an unknown enum value. Over about 1,200 runs there were no parse failures (optimal imaginary value). I report parse failures and validation failures separately, because a value outside a limit (confidence 1.2) is a different problem from broken JSON. If one ever fails, the second model gets a turn, and if that fails too the player goes to a human."
What did the remaining false bans have in common?
Say: "That is the question the report answers: it lists every false ban with the evidence the model quoted. In this imaginary run, say all 4 were PRO players with extreme aim stats but normal reaction times (optimal imaginary value). The fix would be a stricter rule for that tier or requiring two independent signals. The honest approach is to read the list, not to hide it."
What was the cached prefix size, and why does it matter?
Say: "The cache only works above a minimum size (512 tokens for these models). The first call reported about 1,950 tokens written to the cache (optimal imaginary value), which is the tools, the system prompt and the rulebook, comfortably above the minimum. Later calls read it back. I confirm that by checking cacheCreationInputTokens on the first call and cacheReadInputTokens on the next."
What is the one number you are sure of?
Say: "The resilience result, because the test is deterministic: in ResilienceIT I send 20 flags while the simulated API answers HTTP 500 for 20 seconds. The circuit breaker opens, nothing is dropped or dead-lettered, and after the outage all 20 flags get a verdict. 20 of 20, measured, not imaginary."
12.8 A three-minute walk-through you can practise
- The problem: noisy detector, false bans are costly (20 seconds).
- The flow: flag, Haiku with tools, maybe Sonnet, policy, action (40 seconds). Draw the five boxes from section 4.
- Safety: model recommends, code acts; guard; two thresholds; nothing dropped (40 seconds).
- Cost: routing plus caching, measured per attempt in
BigDecimal(30 seconds). - Resilience: retry outermost, breaker, limiter,
maxRetries(0)(30 seconds). - Evaluation: labeled cases, dev vs holdout, same code path as production (20 seconds).
- Honest status and next steps (20 seconds).
12b. The results table optimal imaginary value
This is what COMPARISON.md would show after six prompt versions, if you had run everything. Every number is an optimal imaginary value: I chose values a well-tuned system could plausibly reach, consistent with the price list. Replace them with your own after running evals/README.md. The changes in the last column are the example changes the specification suggests; they are imaginary here because none of the versions v2 to v6 exists yet.
| Version | Change (imaginary) | Precision | Recall | False bans (of 80) |
|---|---|---|---|---|
| v1 | the starting prompt | 91.5% | 90.0% | 10 |
| v2 | stricter wording for PRO aim cases | 93.2% | 90.8% | 8 |
| v3 | require get_recent_matches before any aim-based ban | 94.0% | 91.7% | 7 |
| v4 | recalibrated confidence bands | 95.7% | 92.5% | 5 |
| v5 | shorter rulebook (cheaper, same accuracy) | 95.7% | 92.5% | 5 |
| v6 | evidence in a fixed order | 96.6% | 93.3% | 4 |
All values above are optimal imaginary values.
| Headline figure for v6, routed | Target | Value (optimal imaginary value, not measured) |
|---|---|---|
| Ban precision | at least 95% | 96.6% |
| Cheater recall | at least 92% | 93.3% |
| False bans fewer than rule-only | at least 80% | 95.0% (4 instead of 80) |
| Cost per verdict lower than Sonnet-only without cache | at least 65% | 84.8% ($0.0024 against $0.0158) |
| Schema-valid outputs | 100% over 1,200 or more runs | 100% over 1,200 runs (6 versions × 200 cases), 0 parse failures |
| Human review share | about 12% | 12.0% |
| Escalation rate | (no target) | 18% of cases reached Sonnet |
| Latency per verdict, p50 / p95 | (report only) | 6.1 s / 13.8 s |
| Size of the cached prefix (tools + system + rulebook) | at least 512 tokens | 1,950 tokens (first cacheCreationInputTokens) |
| Dropped verdicts under injected faults | 0 | 0 of 20: this one is measured (ResilienceIT, WireMock faults) |
13. How to practice reading code
Reading code is a skill you build by predicting, then checking. Try these in order. Each takes 10 to 20 minutes.
- Trace one test. Open
AgentLoopTest.oneToolRoundThenAVerdict. Before readingAgentLoop, write down what you expect: how many model calls, how many messages in the second request, who is the last message from. Then read the loop and check. - Break it on purpose. In
DecisionPolicychange>=to>in the auto-ban comparison. Run./mvnw -pl judge-core test -Dtest=DecisionPolicyTest. Which test fails, and why that one? (Answer: the 0.90 exact-boundary test.) Undo the change. This teaches you what tests protect. - Change a rule. Make
EscalationRule.certaintyreturnconfidence(the literal spec rule). RunModelRouterTest. Which tests fail and what does that tell you about the decision in section 5.9? - Follow the money. Use
CostCalculatorTest.sonnetWithAllFourKindsOfTokens: recompute 11,500 by hand from the prices injudge-defaults.yml. - Draw a sequence diagram on paper for "Haiku calls a tool, then answers": who calls whom, in order, from
ModelRouterdown to the gateway and back. Compare with section 4. - Read only the test names of
ModelRouterTestfrom top to bottom. Could you describe the router's behaviour from the names alone? That is the quality bar for test names. - Explain it aloud. Take the Q&A in section 12 and answer five of them without looking, then compare. Record yourself if you can.
A method for reading any unfamiliar file
- Read the class's one-paragraph comment, then its constructor: what does it need to do its job? Those are its dependencies.
- Find its public methods. Everything else is detail.
- Read the public method top to bottom once without stopping. Then again, stopping at each
ifandthrow: what situation does this handle? - Open its test and read the test names. Pick the most interesting test and trace it.
- Search for where it is used (your editor's "find usages"). Now you know its place in the picture.
14. Glossary
| Term | Meaning |
|---|---|
| Agent loop | Repeatedly call the model, run the tools it requests, feed results back, until it answers. |
| Aimbot, speedhack | Cheats: software that aims for the player; moving faster than the game allows. |
| At-least-once delivery | A message may arrive more than once, never zero times. Requires idempotent handling. |
| BigDecimal | Java's exact decimal number type, used for money. |
| Boundary test | A test at the exact edge of a rule (0.90 vs 0.899). |
| Cache read / write | Prompt-cache token kinds: write stores a prefix (slightly dearer), read reuses it (about a tenth of the price). |
| Circuit breaker | Stops calling a failing service for a while so it can recover; open, half-open, closed. |
| Compact constructor | A record's constructor block used to validate fields. |
| Dead-letter topic (DLT) | Where records go when they can never be processed. |
| Decorator | A wrapper implementing the same interface and adding behaviour. |
| Defense in depth | Several independent safeguards, so one failure is not enough to cause harm. |
| Dev / holdout | Cases you may tune on / cases you must not look at while tuning. |
| Escalation | Handing a flag from the cheap model to the strong model. |
| Fake / stub / mock | Test stand-ins: a simple working version / canned answers / records how it was used. |
| Flag | A Telemetry alert that a player looks suspicious in a short window (SuspiciousPlayer). |
| Haiku, Sonnet | Claude models: the small cheap one, and the larger stronger one. |
| Idempotent | Doing it twice has the same effect as doing it once. |
| Jackson | The JSON library (Jackson 3 here: tools.jackson). |
| Jitter | Random variation added to retry waits. |
| JSON schema | A description of the allowed shape of a JSON document. |
| Kafka topic / offset | A named durable log / a consumer's position in it. |
| Memoize | Cache a function's result by its inputs. |
| Precision / recall | Share of bans that were right / share of cheaters that were banned. |
| Prompt caching | The API remembers a prompt prefix so repeated text is cheap. |
| Rate limiter | Caps calls per time period. |
| Record | Java's immutable data class. |
| Resilience4j | Library providing retry, circuit breaker and rate limiter. |
| Stop reason | Why the model stopped: end_turn, tool_use, max_tokens, refusal. |
| Strict tool | A tool whose input schema the API enforces exactly. |
| System prompt | The standing instructions sent with every request. |
| Token | A chunk of text (about three quarters of a word); the unit of billing. |
| Tool / tool use | A function the model can ask your program to run. |
| Verdict | The model's structured recommendation: decision, cheat type, confidence, evidence, explanation, review flag. |
| WireMock / Testcontainers | A fake HTTP server in a container / a library that runs real services in Docker for tests (used from step 9). |