Fairplay Judge, explained from scratch

A guided tour of Project 3 for someone who has never built an LLM agent, never used Resilience4j and is not sure what a "record" is. Every idea is explained the first time it appears. Every code sample is copied from the real files (shortened where marked with ...). The last sections turn all of it into answers you can say out loud in an interview.

How to use this page Read sections 0 to 4 in order: they give you the whole picture. Then read section 5 one subsection at a time with the real file open next to it. Section 12 is the interview material; it only makes sense after you have read the rest. If a word is new, section 14 is a glossary.

0. Where the project stands

The project is built in 12 steps (the plan is in Fairplay Judge — Implementation Specification.md, section 17). Steps 1 and 3 to 11 are finished, and step 12 is finished except the parts that need an API key. Everything in this document marked built exists and is tested. Anything marked needs you cannot be done without your API key or a finished Telemetry run.

StepWhatStatus
1Maven skeleton, three modules, shared settings with validationdone
2SDK smoke test: one real call to Claude to check the request shapewritten and compiles; never run live (needs an API key)
3The verdict: data shape, JSON schema, strict parser, validatordone
4Prompts, the four tools, the guard, the per-flag cachedone
5The agent loopdone
6Resilience (retry, breaker, rate limit) and cost calculationdone
7Model routing, escalation rule, decision policy, JudgeEnginedone
8Arena side of the service (HTTP client, tools backed by Arena, applying actions)done
9Kafka side of the service (listener, publishing verdicts, error handling, metrics)done
10Fault injection tests (WireMock container)done
11Evaluation harness: build the 200-case dataset, replay it, score itcode done and tested; dataset not collected yet (needs a Telemetry run)
12Prompt versions v2 to v6, Docker, CI, Grafana, READMEDocker, Compose, CI, dashboard and README done; v2 to v6 need evaluation runs

Today's tests: 194 in judge-core (97.2% of lines executed by tests), 77 unit tests and 12 integration tests in judge-service (96.9%) and 105 unit tests and 1 integration test in judge-eval (96.2%).

How the numbers in this guide work (read this once) This guide is written as if you had already run everything with your API key: the Telemetry run, collect, all six prompt versions and the cost runs. You have not. Every result figure (precision, recall, cost per verdict, latency, review rate, token counts of a real run) is therefore marked optimal imaginary value: a value chosen to show what a good run would look like, picked so that it is consistent with the prices, the token arithmetic and the project targets. It is not a measurement. The only measured results are the test results (for example: in ResilienceIT, 20 of 20 flags were judged during a simulated outage). In an interview, say "the design aims at this" until you replace the imaginary values with your own, using evals/README.md. The same imaginary values are in the README's Results table, marked the same way.

1. The story: what is this project for?

Imagine an online shooting game. Some players cheat with an aimbot (software that aims for them) or a speedhack (moving faster than the game allows). The Fairplay system fights them with three projects:

Project 1: Arena

The game's backend. It stores players, matches, ratings and moderation history. It can ban a player. A web API.

Project 2: Telemetry

Watches a stream of in-game events (shots, positions) in real time and raises a flag when a player looks suspicious in a 5-second window. Fast, but noisy.

Project 3: Judge (this one)

Reads each flag, investigates the player with an AI model that can look things up in Arena, and decides: ban, watch, or clear.

Why not just ban every flag?

A 5-second window is tiny. A genuinely great player can look like a cheater for 5 seconds: a lucky streak, a headshot burst. If you banned every flag you would ban many innocent people, and banning an innocent player is the worst outcome. So someone has to look at the player's long-run record: hundreds of shots, many matches, past moderation history. That is a human moderator's job, and it is slow and expensive. The Judge does that investigation automatically, and sends only the genuinely unclear cases to a human.

An analogy

Think of a referee system in a sport. A fast camera system (Telemetry) flags a possibly illegal play. A junior reviewer (the cheap model, Claude Haiku) looks at the replay and the player's history. If the junior is confident, that is the answer. If the junior is unsure, or the player is a star, a senior reviewer (the strong, expensive model, Claude Sonnet) reviews it from scratch, without being told what the junior thought. Finally, a rules clerk (plain Java code, no AI) reads the reviewer's recommendation and decides what actually happens, using strict rules. The reviewers can only recommend. They can never press the "ban" button themselves.

suspicious-playersflag from Telemetry (Kafka topic)
→
Haiku investigatescheap, fast, calls tools
→
Sonnet, only if neededfresh second opinion
→
DecisionPolicyplain code picks ONE action
→
Arena + judge-verdictsaction applied, verdict published

Three promises the design keeps

  1. The model recommends, code acts. The AI returns a verdict (data). A plain Java class, DecisionPolicy, turns it into an Arena action. The model never chooses a web request or a URL. This is the single most important design idea; you will be asked about it.
  2. A ban needs strong confidence. Auto-ban only at confidence 0.90 or higher and with no "please ask a human" flag. Arena itself refuses service bans below 0.80 as a second safety net.
  3. A flag is never lost. If Claude or Arena is down, the flag waits and is retried. Only malformed flags go to a dead-letter topic.
Learning goals of the project The spec says the point is "a hand-written, measured agent loop": no LangChain, no Spring AI. You write the loop that talks to the model, runs tools and handles every failure yourself, and you measure cost and accuracy. That is exactly what makes it a strong portfolio piece, and exactly what interviewers will probe.

2. Ideas you need first

If you already know a topic, skip it. Each one is short on purpose.

2.1 Calling an LLM, and what a token is

An LLM (large language model) such as Claude is used through a web API. You send a request: the model name, some instructions (the system prompt), the conversation so far (a list of messages, each from the user or the assistant), and a limit on how long the answer may be. You get back a response containing the answer and a stop reason (why it stopped, for example "I finished" or "I hit your length limit").

The model reads and writes tokens, which are chunks of text, roughly three quarters of a word each. You pay per token, and output tokens cost more than input tokens. Prices in this project come from judge-defaults.yml in USD per million tokens, for example Haiku input $0.10 and Sonnet input $2.00. Haiku is the small, cheap, fast model; Sonnet is the larger, more capable, pricier one.

Calls fail: the service can be overloaded (HTTP 529), you can be rate-limited (429), the network can drop. A production system has to survive all of that. See 2.6.

2.2 Tools and the agent loop

A plain model only knows what is in its prompt. A tool (also called a function call) lets the model ask your program to look something up. You describe each tool to the model by name, description and input shape. The model can then answer not with text but with a request: "please run get_stat_percentiles with this player id". Your program runs it and sends the result back. The model continues. This repeat-until-done pattern is an agent loop:

send messages + tool list to the model
repeat:
    model replies
    if it asks for tools:   run them, append results as a new message, go around again
    if it gives an answer:  stop and use the answer
    if it hit a limit:      give up on this attempt

Important: the model never runs anything. It only writes a request. Your code decides whether to run it. That is why the "guard" in section 5.5 can refuse any request that is not about the flagged player.

2.3 Structured output and a JSON schema

You want the final answer as data, not prose: "BAN, confidence 0.94, these 3 facts". A JSON schema describes the allowed shape (field names, types, allowed words). You send the schema with the request and the API forces the model's answer to match it. This project's schema has six fields: decision, cheatType, confidence, evidence, explanation, needsHumanReview.

The API guarantees the shape but not every limit (for example "at most 600 characters" is only a hint). So the code validates again. Never trust output just because it was "structured".

2.4 Prompt caching

Every flag sends the same long instructions (the rulebook, about 3,000 tokens) before the part that changes (the flag). Re-sending thousands of identical tokens for every flag is wasteful. Prompt caching lets Anthropic remember a prefix of the prompt: the first request writes the cache (slightly more expensive), later requests read it (about ten times cheaper than normal input). You mark the end of the part to cache with a cache_control marker.

The cache works on a prefix, so the order matters: everything identical across flags first (tools, then system text, then the rulebook), then the changing flag last. Put anything that changes (a timestamp, a player id) inside the cached part and the cache never hits. Usage numbers come back in four kinds: normal input, cache write, cache read, output, and each has its own price. This is the main reason the project can be cheap.

2.5 Java you will meet

You do not need to be an expert, but you need to recognise these.

ThingWhat it meansExample in this project
recordA small immutable data holder. record Verdict(Decision decision, double confidence) automatically gets a constructor, getters (v.confidence()), equals and toString. The spec requires records for all data.Verdict, Attempt, ToolResult
compact constructorThe block inside a record with no parameter list. It runs on creation, so it is the place to reject bad values.JudgeSettings.Policy refuses a ban threshold below 0.80
enumA fixed list of allowed values.Decision { BAN, WATCH, CLEAR }
interfaceA promise of what methods exist, without saying how. Lets you swap implementations (real, fake).MessagesGateway, ToolExecutor
finalOn a class: cannot be extended. On a field: assigned once. The spec wants every field private final.almost every class
unchecked exceptionAn error you do not have to declare. Here, all project exceptions extend RuntimeException.InvalidVerdictException
Optional<T>"A value or nothing", instead of null.response.stopReason()
genericsTypes with a type in angle brackets: List<String> is a list of text. Read it as "a List of String".everywhere
lambdaA short inline function: (a, b) -> a + b. Supplier<T> is "a function with no input that returns a T".the gateway's decorators
decorator patternAn object that wraps another object of the same type and adds behaviour, then passes the call on.GuardedToolExecutor and MemoizingToolExecutor wrap a ToolExecutor
static factoryA static method that builds an object, with a meaningful name.JudgeEngine.create(...), ToolResult.ok(...)
dependency injectionA class receives what it needs through its constructor rather than creating it. Makes it testable, because tests can pass fakes.new AgentLoop(gateway, prompts, ...)

2.6 Retry, circuit breaker, rate limiter

Three small safety devices, all from the library Resilience4j:

The order matters and is a favourite interview question. See 5.7.

2.7 Maven, Spring Boot, Jackson, tests

2.8 Kafka in one minute (needed for steps 9 and later)

Kafka is a durable message log. A topic is a named log. Programs produce records into a topic and other programs consume them, each remembering how far it has read (the offset). If a consumer crashes before saving its offset, it re-reads the record: this is at-least-once delivery, so processing must be safe to repeat (idempotent). Telemetry writes flags to suspicious-players; the Judge reads it and writes judge-verdicts. None of the Kafka code exists yet (step 9), but the design decisions that depend on it appear throughout the core.

3. Map of the repository

One Maven project with three modules. Read the next tree top to bottom; "built" means the files exist today.

Project 3/ ├── pom.xml parent: versions, JaCoCo coverage rule, shared test libraries built ├── mvnw, mvnw.cmd, .mvn/ the Maven wrapper (copied from Telemetry) built ├── CLAUDE.md, todo.md, handoff.md guidance and progress notes for future work sessions built ├── Fairplay Judge — Implementation Specification.md the written plan (the "spec") ├── judge-core/ the brain. No Spring, no Kafka. built │ └── src/main/java/dev/fairplay/judge/core/ │ ├── JudgeSettings.java every setting, validated │ ├── JudgeEngine.java the single entry point: flag in, outcome out │ ├── JudgeJson.java the one shared JSON mapper │ ├── model/ Verdict, Decision, CheatType, SuspiciousPlayer, Evidence, Rule, SkillTier, │ │ TokenUsage, Attempt, Investigation, ArenaAction, ActionKind, JudgeAction, │ │ PlayerStatus, JudgeOutcome │ ├── prompt/ PromptLibrary, PromptVersion, UserPromptBuilder │ ├── tools/ ToolCatalog, ToolExecutor, ToolResult, GuardedToolExecutor, MemoizingToolExecutor │ ├── agent/ AgentLoop, MessagesGateway, ResilientMessagesGateway, AnthropicResilience, │ │ VerdictSchema, VerdictParser, VerdictValidator, the 3 exceptions, AgentStats, AgentRun │ ├── routing/ ModelRouter, EscalationRule, EscalationReason │ ├── policy/ DecisionPolicy │ └── cost/ PriceTable, CostCalculator │ └── src/main/resources/ │ ├── judge-defaults.yml the shared settings file │ └── prompts/v1/system.md, rulebook.md the instructions given to the model │ └── src/test/ 180 tests plus helpers shared with the other modules ├── judge-service/ the Spring Boot app: Kafka in, Arena out only config so far │ └── JudgeServiceApplication, config/JudgeProperties, config/JudgeSettingsConfig ├── judge-eval/ command-line app that scores prompt versions empty shell so far └── code_explanation/ this page

Reading order that works

  1. This page, sections 1 to 4.
  2. JudgeEngine.java: it is short and shows what the whole core does.
  3. model/: learn the nouns first.
  4. agent/AgentLoop.java: the heart.
  5. routing/ModelRouter.java and policy/DecisionPolicy.java.
  6. The rest (settings, tools, resilience, cost), when something earlier points to it.
  7. Always read a class's test right after it. Tests are examples with the answers written down.

4. The journey of one flag

Here is everything that happens to one flag, in order, with the class that does each step. The numbers match the layers in the code. Steps 1, 2 and 8 to 10 belong to the service, and steps 3 to 7 are in judge-core. Everything is built and tested.

  1. (service, planned) A SuspiciousPlayer JSON record arrives from the Kafka topic suspicious-players. The service checks Arena: is the player real, not already banned, not judged in the last 30 minutes? If any check fails the flag is recorded as SKIPPED and no model is called (free).
  2. (service, planned) The service calls JudgeEngine.judge(flag, arenaTools, status).
  3. JudgeEngine wraps the tools in one MemoizingToolExecutor (a per-flag cache) and hands everything to ModelRouter.
  4. ModelRouter runs attempt 1 with Haiku through AgentLoop.
  5. AgentLoop builds the request (cached system text, four tool definitions, the flag, the verdict schema), sends it through ResilientMessagesGateway, and loops: when the model asks for a tool, GuardedToolExecutor checks the request and the cache or Arena answers.
  6. When the model answers with text, VerdictParser and VerdictValidator check it strictly and produce a Verdict.
  7. EscalationRule asks: is Haiku unsure, or is this a ban on a top-tier player, or is a ban too thinly supported? If yes, ModelRouter runs attempt 2 with Sonnet from a blank conversation, sharing the same tool cache. Sonnet's answer is final.
  8. CostCalculator prices each attempt. The router records an Attempt per model call chain and returns an Investigation (final verdict or none, plus all attempts).
  9. DecisionPolicy turns the verdict and the player's status into exactly one ArenaAction.
  10. (service, planned) The service sends that action to Arena (always source: JUDGE), publishes a JudgeVerdictRecord to judge-verdicts, and only then lets Kafka mark the flag as done.
JudgeEngineentry point
→
ModelRouterwho runs, and when to escalate
→
AgentLoopone model, one conversation
→
ResilientMessagesGatewayretry, breaker, limiter
→
Claude APIover the internet

Each box only knows the box to its right. That is the whole point of the layering: you can test any layer by replacing the one to its right with a fake. AgentLoopTest replaces the gateway; ModelRouterTest replaces the gateway too, but exercises the router; ResilientMessagesGatewayTest replaces the internet with a tiny local web server.

5. judge-core, file by file

5.1 Settings: JudgeSettings and judge-defaults.yml

All tunable values (model names, limits, thresholds, prices) live in one YAML file, so an experiment means editing a value, never code. The same file is imported by the service and the evaluation, so they can never judge with different settings.

judge-core/src/main/resources/judge-defaults.yml (excerpt)
judge:
  prompt-version: ${JUDGE_PROMPT_VERSION:v1}      # environment variable, default v1
  models:
    triage: claude-haiku-5-5
    escalation: claude-sonnet-5-5
  limits:
    max-turns: 6          # model calls per attempt
    max-tool-calls: 8     # tool executions per attempt
    max-tokens: 1500      # output tokens per model call
  routing:
    triage-enabled: true
    escalation-enabled: true      # my addition, see section 11
    escalate-below-confidence: 0.85
    escalate-high-tier-bans: true
    min-evidence-for-ban: 2
  policy:
    auto-ban-min-confidence: 0.90
  pricing:                        # USD per million tokens
    "[claude-haiku-5-5]":  { input: 0.10, cache-write: 0.125, cache-read: 0.01, output: 0.50 }
    "[claude-sonnet-5-5]": { input: 2.00, cache-write: 2.50,  cache-read: 0.10, output: 10.00 }

The Java side is JudgeSettings, one record containing nested records. Look at how a bad config is rejected:

JudgeSettings.java (excerpt)
public record Policy(double autoBanMinConfidence, Duration recentVerdictWindow) {
    public Policy {                                    // compact constructor: runs on creation
        requireProbability(autoBanMinConfidence, "judge.policy.auto-ban-min-confidence");
        require(autoBanMinConfidence >= ARENA_BAN_CONFIDENCE_FLOOR,      // 0.80
                "judge.policy.auto-ban-min-confidence must be at least 0.8 (Arena's own floor)");
        requirePositive(recentVerdictWindow, "judge.policy.recent-verdict-window");
    }
}

Read it like this: you cannot even construct a Policy with an auto-ban threshold under 0.80. So a typo in YAML stops the application at startup instead of silently banning on weak evidence. The same file also checks that both configured models have a price, and that prompts/<version>/system.md and rulebook.md exist.

How does YAML become the record? In the service, JudgeSettingsConfig does one line: Binder.get(environment).bind("judge", JudgeSettings.class).get(). Spring's Binder maps max-turns to maxTurns and "30m" to a Duration. The core stays free of Spring. The service-only values (Kafka topic names, Arena URL) live in a second record, JudgeProperties.

Interview angle "Fail fast on bad config" and "keep the core framework-free" are both good, concrete engineering answers. The test JudgeConfigBindingTest binds the real YAML files and proves the defaults match the code's expectations.

5.2 The data shapes (package model)

Learn these nouns first; the rest of the code is verbs acting on them.

TypeWhat it isKey fields
SuspiciousPlayerThe flag from Telemetry. A copy of Telemetry's record; unknown JSON fields are ignored so Telemetry can add fields freely.flagId, playerId, username, skillTier, rule, score, detectedAt, evidence
EvidenceThe numbers behind a flag (shots, hits, headshot rate, reaction time, or speed samples).all nullable: aimbot flags fill some, speedhack others
VerdictThe model's structured answer.decision, cheatType, confidence, evidence (list), explanation, needsHumanReview
Decision, CheatTypeEnums. Each has a parse that upper-cases the text first, because models sometimes change capitalisation.BAN WATCH CLEAR; AIMBOT SPEEDHACK NONE
TokenUsageToken counts for one or many calls, in four kinds.input, cacheWrite, cacheRead, output; plus() adds two
AttemptOne model's try: which model, what happened, what it cost.model, outcome, stats, costUsd, latencyMs, reason
InvestigationThe models' work for one flag: final verdict (or none) plus every attempt.verdict, attempts; helpers finalModel(), totalCostUsd()
ArenaActionThe single action the policy chose.kind, label, reason, evidence, confidence, requiresHumanReview
ActionKind vs JudgeActionActionKind is what Arena receives: BAN FLAG CLEAR NOTE. JudgeAction is what the Judge records: BAN FLAG_FOR_REVIEW NOTE CLEAR SKIPPED. They differ on purpose, see 5.10.
PlayerStatusArena's account state.ACTIVE FLAGGED BANNED APPEALED REINSTATED
JudgeOutcomeWhat JudgeEngine returns: the investigation and the action.
Verdict.java
public record Verdict(
        Decision decision,
        CheatType cheatType,
        double confidence,
        List<String> evidence,
        String explanation,
        boolean needsHumanReview) {

    public Verdict {
        Objects.requireNonNull(decision, "decision");
        ...
        evidence = List.copyOf(evidence);     // an unmodifiable copy: nobody can change it later
    }
}

What does confidence mean? It is the model's probability that the player is cheating, from 0 to 1. A confident "innocent" verdict has confidence near 0.05, not 0.95. Keep this in mind; it explains one of the project's key decisions (5.9).

5.3 The verdict pipeline: schema, parser, validator

Three small classes make sure only a well-formed, in-range verdict ever leaves the model's output.

VerdictSchema: what we ask for

It holds the JSON schema as a Java text block and builds the SDK's OutputConfig from it. Every object in the schema has "additionalProperties": false and a fixed list of properties (no free-form maps). VerdictSchemaTest checks that.

VerdictParser: strict on purpose

VerdictParser.java (excerpt)
private final JsonMapper mapper = JsonMapper.builder()
        .enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS)   // "{...} thanks!" is rejected
        .build();

public Verdict parse(String text) {
    return validator.validate(parseUnvalidated(text));            // parse first, then validate
}
...
for (String field : FIELDS) {
    if (!root.has(field) || root.get(field).isNull()) {
        throw parseError("missing field " + field);
    }
}
for (String name : root.propertyNames()) {
    if (!FIELDS.contains(name)) throw parseError("unexpected field " + name);
}

Why so strict? The model is told "answer only with the JSON object". If it adds a sentence, wraps the JSON in a markdown fence, drops a field or invents one, that is a parse failure and the router will try Sonnet. Being strict makes failures visible and countable instead of silently accepting garbage. VerdictParserTest has a case for each.

VerdictValidator: the rules the schema cannot express

RuleWhy code must check it
confidence between 0 and 1The schema says "number"; ranges are only descriptions
1 to 6 evidence items, each 1 to 300 charactersFits Arena's limits and keeps tokens small
explanation 1 to 600 charactersSame
BAN needs a cheat type other than NONE; CLEAR needs NONEA BAN with no cheat type is self-contradictory

A boundary test checks each rule at the edge: 0 and 1 pass, -0.01 and 1.01 fail; 6 items pass, 7 fail. Boundary tests are where off-by-one bugs live.

InvalidVerdictException and its two kinds

Both failures throw InvalidVerdictException carrying a Kind: PARSE (not JSON, wrong shape, unknown enum word) or VALIDATION (well-formed but out of limits, or the model stopped for a reason other than finishing). They are counted separately in the metrics. The exception can also carry AgentStats: even a failed attempt used tokens and cost money, and the router needs that number.

5.4 Prompts: PromptLibrary and UserPromptBuilder

The model's instructions are plain Markdown files, not Java strings, so they can be edited and versioned. prompts/v1/system.md is the role and the working method ("always call get_moderation_history before deciding BAN"). prompts/v1/rulebook.md holds game facts (top sprint speed 7.0 units/s, reaction under 120 ms is not human) and how to weigh evidence. Later prompt versions (v2 to v6) will be copies with one change each, measured by the evaluation.

PromptLibrary.java (the important method)
public List<TextBlockParam> systemBlocks(boolean cachingEnabled) {
    TextBlockParam.Builder rulebook = TextBlockParam.builder().text(prompt.rulebook());
    if (cachingEnabled) {
        rulebook.cacheControl(CacheControlEphemeral.builder().build());   // the cache breakpoint
    }
    return List.of(TextBlockParam.builder().text(prompt.system()).build(), rulebook.build());
}
Position in the requestContentChanges per flag?Cached?
1The four tool definitionsnoyes (covered by the breakpoint below)
2System block 1: system.mdnoyes
3System block 2: rulebook.md with the cache markernoyes: the breakpoint ends here
4User message: the flag and the timeyesno
5+Assistant tool requests and tool resultsyesno

PromptLibraryTest protects this: it checks the rulebook block has cache control only when caching is on, and that the system text contains no date or UUID. Why a test for that? One accidental timestamp in the system text would silently turn off caching and multiply your bill.

UserPromptBuilder makes the part that does change. For a flag it returns text like:

Flagged at 2025-11-17T09:35:55.412Z. Investigate this player and return your verdict.

Flag:
{
  "flagId" : "7c1e...",
  "playerId" : "3f2a...",
  "rule" : "AIMBOT",
  ...
}

No label, no hint about whether the player is truly a cheater, ever appears in a prompt. UserPromptBuilderTest checks the words "cheater" and "honest" are absent.

5.5 Tools and the guard

ToolCatalog: four read-only tools

Tool nameReturns (from Arena)Inputs
get_player_profilestatus, country, account age, matches played, ratingplayer_id
get_stat_percentileslong-run stats and where the player ranks among all playersplayer_id
get_recent_matchesper-match stats, newest firstplayer_id, optional limit (1 to 20)
get_moderation_historypast moderation actions, newest firstplayer_id, optional limit

Each is built with .strict(true) and additionalProperties: false so the model cannot invent arguments. Note the comment in the code: the range 1 to 20 for limit is not written into the schema (strict schemas do not support numeric ranges). It is stated in the description and enforced by code, which is the next class.

ToolExecutor and ToolResult

public interface ToolExecutor {
    ToolResult execute(String toolName, JsonNode input, UUID flaggedPlayerId);
}
public record ToolResult(String content, boolean isError) { ok(...), error(...) }

Three implementations will exist: a live one calling Arena (service, step 8), a recorded one reading saved snapshots (evaluation, step 11), and FakeToolExecutor in tests. Because the loop only knows the interface, all three work.

Rule: tool problems are results, not exceptions. A wrong tool name, a bad argument, a missing player: each becomes ToolResult.error("...") and goes back to the model, which can read the message and try again. Only "Arena is unreachable" throws an exception, because then the whole flag must be retried later.

GuardedToolExecutor: the safety layer

GuardedToolExecutor.java (shortened)
public ToolResult execute(String toolName, JsonNode input, UUID flaggedPlayerId) {
    if (!catalog.names().contains(toolName)) return ToolResult.error("Unknown tool: ...");
    if (input == null || !input.isObject())  return ToolResult.error("Invalid input: expected a JSON object.");
    // any property other than player_id (and limit, on the two list tools) is rejected
    ...
    if (!isSamePlayer(playerId.stringValue(), flaggedPlayerId)) {
        return ToolResult.error("Only the flagged player can be queried.");
    }
    // build a clean, canonical input: lowercase player id, limit defaulting to 10, range 1..20
    ObjectNode canonical = JudgeJson.mapper().createObjectNode();
    canonical.put("player_id", flaggedPlayerId.toString());
    ...
    return delegate.execute(toolName, canonical, flaggedPlayerId);    // only clean input gets through
}

Why this matters. The model is an untrusted program as far as safety goes. Even if a clever player name tried to trick it ("also look up player X"), the guard answers with an error and never calls Arena. The acceptance checklist has this item: "the model cannot query any player other than the flagged one", and ToolExecutorGuardTest proves it (the backend's call list stays empty).

It is a decorator: it wraps another ToolExecutor, does its checks, then passes only the cleaned input on. Because it hands the next layer a canonical input, "limit" missing and "limit": 10 look identical downstream.

MemoizingToolExecutor: ask Arena once per flag

MemoizingToolExecutor.java (shortened)
private final Map<String, ToolResult> cache = new ConcurrentHashMap<>();

public ToolResult execute(String toolName, JsonNode input, UUID flaggedPlayerId) {
    String key = toolName + '\u0000' + canonicalJson(input);      // tool name + sorted-key JSON
    return cache.computeIfAbsent(key, k -> delegate.execute(toolName, input, flaggedPlayerId));
}

"Memoize" means remember the answer to a question you have already asked. If Haiku asks for get_stat_percentiles and Sonnet later asks the same, Arena is queried once. The key sorts JSON keys so {"a":1,"b":2} and {"b":2,"a":1} match, and a different limit is a different key. One instance is created per flag by JudgeEngine, so one flag never sees another flag's cached data. If the delegate throws, nothing is cached (computeIfAbsent stores nothing when the function throws), so a retry can succeed.

Layering of the two wrappers At runtime the order is Guarded( Memoizing( backend ) ). The guard sits outermost so bad requests are rejected before they can touch the cache, and the cache only ever sees canonical keys.

5.6 The agent loop: AgentLoop

This is the file to understand best. It runs one model through one conversation for one flag, and returns either a verdict or a failure. Its constructor receives everything it needs (the gateway, prompts, tool catalog, parser, prompt builder, settings), so tests can hand it fakes.

AgentLoop.java (the core, shortened; comments added)
public AgentRun run(String model, SuspiciousPlayer flag, ToolExecutor backend) {
    UUID flaggedPlayer = flag.playerUuid();
    ToolExecutor tools = new GuardedToolExecutor(backend, toolCatalog);     // the guard is ALWAYS applied here

    List<MessageParam> messages = new ArrayList<>();
    messages.add(MessageParam.ofUser(userPrompt.build(flag)));              // message 1: the flag

    for (int turn = 1; turn <= limits.maxTurns(); turn++) {                  // at most 6 model calls
        MessageCreateParams params = MessageCreateParams.builder()
                .model(model)
                .maxTokens(limits.maxTokens())
                .systemOfTextBlockParams(prompts.systemBlocks(settings.caching().enabled()))
                .tools(toolCatalog.toolUnions())
                .messages(List.copyOf(messages))                             // a snapshot, see below
                .outputConfig(VerdictSchema.outputConfig())
                .build();

        Message response = gateway.create(params);                           // the network call (resilient)
        usage = usage.plus(TokenUsage.fromSdk(response.usage()));
        messages.add(response.toParam());                                    // remember what the model said

        StopReason stop = response.stopReason().orElse(null);

        if (StopReason.TOOL_USE.equals(stop)) {
            List<ContentBlockParam> results = new ArrayList<>();
            for (ContentBlock block : response.content()) {                  // the model may ask for SEVERAL tools
                Optional<ToolUseBlock> use = block.toolUse();
                if (use.isEmpty()) continue;
                if (++toolCalls > limits.maxToolCalls()) throw new TurnLimitExceededException("tool calls");
                ToolResult result = tools.execute(use.get().name(), toJsonNode(use.get()._input()), flaggedPlayer);
                results.add(ContentBlockParam.ofToolResult(ToolResultBlockParam.builder()
                        .toolUseId(use.get().id()).content(result.content()).isError(result.isError()).build()));
            }
            messages.add(MessageParam.ofUserOfContentBlockParams(results));  // ALL results in ONE message, in order
            continue;                                                        // ask the model again
        }
        if (StopReason.END_TURN.equals(stop)) {
            String json = ...all text blocks joined...;
            return new AgentRun(verdictParser.parse(json), stats(...));      // parse + validate, or throw
        }
        throw new InvalidVerdictException(Kind.VALIDATION, "stop reason " + ...);   // max_tokens, refusal, ...
    }
    throw new TurnLimitExceededException("turns");                           // 6 calls and still no answer
}

Walk through it in plain words

  1. Set up. Wrap the tools in the guard. Start the conversation with one user message: the flag.
  2. Loop up to 6 times. Each time, build a full request. The conversation grows by appending to messages. List.copyOf sends a snapshot, so later additions cannot change a request already made. A test (earlierRequestsAreNotChangedByLaterTurns) exists purely for this.
  3. Send it through the gateway. Add the token counts to the running total. Append the assistant's reply to the conversation unchanged: the API requires that its tool_use blocks stay in the history before the results.
  4. Look at why it stopped.
    • tool_use: it wants tools. Run each (one response can contain several requests). Count them; past 8 total, give up. Pack all results into one user message, in the order requested, and loop.
    • end_turn: it is done. Join the text, parse and validate it, return.
    • anything else (max_tokens: answer cut off; refusal): no usable verdict, throw.
  5. Ran out of turns: throw TurnLimitExceededException.

Details worth knowing

What AgentLoopTest proves (21 tests)

It uses ScriptedGateway: a fake that plays back prepared model replies from a queue and records every request it received. TestMessages builds real SDK Message objects by parsing API-shaped JSON. So the tests walk the real loop code with no network. Cases include: one tool round then a verdict; two tools in one turn get two results in one message in order; a tool error comes back with isError; the model asking about another player gets the guard's error and the backend is never called; max_tokens, refusal and unparsable text all fail with the right kind; the turn limit and tool-call limit; token usage summed across turns with missing cache fields treated as 0; and the request itself carries the model, token limit, four tools, the schema and the cache marker.

5.7 Resilience: ResilientMessagesGateway and AnthropicResilience

MessagesGateway is a one-method interface: Message create(MessageCreateParams params). The loop only knows this interface, so tests can substitute ScriptedGateway. The production implementation wraps the real Anthropic SDK client in the three safety devices from 2.6.

ResilientMessagesGateway.java (shortened)
public Message create(MessageCreateParams params) {
    Supplier<Message> call    = () -> send(params);                          // the real HTTP call
    Supplier<Message> limited = RateLimiter.decorateSupplier(rateLimiter, call);
    Supplier<Message> guarded = CircuitBreaker.decorateSupplier(circuitBreaker, limited);
    Supplier<Message> retried = Retry.decorateSupplier(retry, guarded);       // retry is OUTERMOST
    try {
        return retried.get();
    } catch (BadRequestException e) {
        throw new NonRetryableJudgeException("anthropic rejected request", e);   // our request is wrong
    } catch (CallNotPermittedException | RequestNotPermitted | AnthropicServiceException
             | AnthropicIoException e) {
        throw new LlmUnavailableException(e);                                    // hold the flag, retry later
    }
}

Why retry is outermost

Read the nesting from the inside out: call is wrapped by the limiter, wrapped by the breaker, wrapped by the retry. When retried.get() runs, retry calls the breaker; the breaker calls the limiter; the limiter calls the network. Because retry is on the outside, every retry attempt passes through the breaker and the limiter again. So retried calls count toward the rate limit, and when the breaker opens, the next retry attempt is refused immediately (and an open breaker is not retried). If the order were reversed, a single retried call would count as one call to the breaker no matter how many attempts it took, hiding the outage.

The settings (AnthropicResilience)

DeviceSettingDetail
Retry4 attempts, 1 s start, x2, jitter 0.5, cap 20 sRetries only RateLimitException (429), InternalServerException (5xx, including 529 overloaded) and AnthropicIoException (network). A 429 with a Retry-After header waits at least that long (capped at 60 s). A 400 or 401 is not retried: waiting will not fix it.
Circuit breakerwindow of 20 calls, needs 10, opens at 50% failures, open 30 s, 3 trial callsCounts only 5xx and network errors as failures. 400, 401, 403 and 429 do not count: they do not mean the service is down.
Rate limiter120 calls per minute, wait up to 60 s for a slotOne limiter shared by all threads.
AnthropicResilience.java (the retry rule)
static boolean isRetryable(Throwable t) {
    return t instanceof RateLimitException || t instanceof InternalServerException
            || t instanceof AnthropicIoException;
}

Two details that interviewers like

How it was tested without the real API

ResilientMessagesGatewayTest starts a tiny local web server (StubAnthropicServer, built on the JDK's own HttpServer) that returns scripted replies: 429, 529, 500, a dropped connection, a slow answer, 400, 401. The real SDK client talks to it, so the tests prove how the SDK really maps HTTP errors to exceptions. It confirmed, for example, that 529 arrives as OverloadedException, a subclass of InternalServerException, so the retry rule covers it. Other tests prove the breaker opens after enough failures, makes the next call fail fast without sending a request, closes again after the service recovers, and is not opened by 429s.

5.8 Cost: PriceTable and CostCalculator

Every attempt is priced from config, so cost per verdict is a measured number, not an estimate. The formula (USD, with token counts divided by one million):

cost = (input * p_in + cacheWrite * p_cw + cacheRead * p_cr + output * p_out) / 1,000,000
CostCalculator.java
public BigDecimal cost(String model, TokenUsage usage) {
    JudgeSettings.Price p = prices.priceOf(model);
    BigDecimal total = p.input().multiply(BigDecimal.valueOf(usage.input()))
            .add(p.cacheWrite().multiply(BigDecimal.valueOf(usage.cacheWrite())))
            .add(p.cacheRead().multiply(BigDecimal.valueOf(usage.cacheRead())))
            .add(p.output().multiply(BigDecimal.valueOf(usage.output())));
    return total.divide(MILLION, 6, RoundingMode.HALF_UP);
}

Why BigDecimal and not double? Money must not accumulate floating-point error. 0.1 + 0.2 is not exactly 0.3 in double; BigDecimal is exact. Six decimal places because a Haiku call costs fractions of a cent.

Important detail: the API's input_tokens counts only the tokens after the last cache marker. Total input is cacheRead + cacheWrite + input, and each part has its own price. CostCalculatorTest checks hand-computed values for both models, including a case that rounds half up.

5.9 Routing: ModelRouter and EscalationRule

The idea in one paragraph

Haiku is about twenty times cheaper than Sonnet. Most flags are easy, so let Haiku handle everything first. Escalate to Sonnet only when Haiku is unsure, fails, or wants to ban a top-tier player (where a mistake is costliest). Sonnet starts a fresh conversation and does not see Haiku's answer, otherwise it would tend to just agree. Sonnet's verdict is final: no third attempt.

When does it escalate? (EscalationRule)

ReasonConditionWhere it comes from
INVALIDHaiku threw InvalidVerdictExceptionthe router's catch block
TURN_LIMITHaiku threw TurnLimitExceededExceptionthe router's catch block
LOW_CONFIDENCEHaiku is unsure (see the box below)EscalationRule
HIGH_TIER_BANdecision is BAN and the player is PLATINUM or PROEscalationRule
THIN_EVIDENCEdecision is BAN with fewer than 2 evidence itemsEscalationRule

The first match wins and is recorded on the attempt, which feeds the judge.escalations metric.

A decision you must be able to explain: how "low confidence" is read The spec says: escalate when confidence < 0.85. But confidence is the probability of cheating, and the prompt tells the model to answer below 0.50 when the player is probably innocent. Read literally, every CLEAR (confidence about 0.05) and every WATCH would escalate to Sonnet, and the "65% cheaper" goal would be impossible. So the code treats a verdict as unsure only when the cheating probability sits between 0.15 and 0.85:
static double certainty(double confidence) {
    return Math.max(confidence, 1.0 - confidence);     // how sure, whichever way it points
}
// escalate when certainty < 0.85
So BAN at 0.85 stays; BAN at 0.84 escalates; CLEAR at 0.15 stays; CLEAR at 0.16 escalates. This is a judgment call flagged with // DECISION and in handoff.md; switching to the literal rule is one line, and the evaluation can compare the two. In an interview, say exactly that: you found an ambiguity, you reasoned about cost, you documented it and made it testable.

The router's flow (ModelRouter.investigate)

if triage is disabled:             run ONLY Sonnet            (the cost baseline)
try:
    run = AgentLoop.run(Haiku)
    if escalation disabled or rule says "no":   record FINAL, done
    else:                                        record ESCALATED with the reason
catch InvalidVerdictException:     record INVALID   (and the reason INVALID)
catch TurnLimitExceededException:  record TURN_LIMIT
(if escalation is disabled here: no verdict at all)
run AgentLoop.run(Sonnet)  ->  FINAL (its verdict is the answer)  or  INVALID / TURN_LIMIT (no verdict)

Each attempt becomes an Attempt record: model, outcome (FINAL, ESCALATED, INVALID, TURN_LIMIT), token stats, cost (from CostCalculator), latency (from the injected Clock, never Instant.now(): a project rule that makes time controllable in tests) and a human-readable reason. The result is an Investigation.

Modes. routing.triage-enabled=false gives the Sonnet-only baseline used for the cost comparison. routing.escalation-enabled=false gives haiku-only. The second key was added by this project because the spec's evaluation needs "no escalation" but its config has no switch for it.

ModelRouterTest (24 tests) has one test per escalation reason, boundary tests for the confidence rule, the proof that Sonnet's first request has exactly one user message, proof that both attempts share one tool cache (the backend is hit once even though both models asked), and cost and latency checks with hand-computed numbers.

5.10 The decision policy: DecisionPolicy

This is plain Java with no AI. It is a pure function: the same inputs always give the same output, nothing is read from the outside world. That makes it trivial to test exhaustively, and it is the answer to "what if the model misbehaves?"

DecisionPolicy.java (the decision rows, simplified into pseudo-code)
if (v == null)
    return FLAG, human review, reason "Judge could not reach a verdict";              // row 1
if (v.decision() == Decision.BAN) {
    if (v.confidence() >= autoBanMinConfidence && !v.needsHumanReview())
        return BAN, with confidence (3 decimals);                                      // row 2
    return FLAG for human review, with confidence;                                     // row 3
}
if (v.decision() == Decision.WATCH) {
    if (v.needsHumanReview()) return FLAG for human review;                            // row 4
    return NOTE;                                                                       // row 5
}
// CLEAR
if (status == PlayerStatus.FLAGGED) return CLEAR;                                      // row 6
return NOTE "Cleared by Judge: ...";                                                   // row 7
#Final verdictArena requestRecorded as
1none (both attempts failed)FLAG, needs reviewFLAG_FOR_REVIEW
2BAN, confidence ≥ 0.90, no review wantedBANBAN
3any other BANFLAG, needs reviewFLAG_FOR_REVIEW
4WATCH with reviewFLAG, needs reviewFLAG_FOR_REVIEW
5WATCHNOTENOTE
6CLEAR and player is FLAGGEDCLEARCLEAR
7CLEARNOTENOTE

Arena's state machine

Arena only allows certain moves: FLAG only from ACTIVE or REINSTATED, CLEAR only from FLAGGED, BAN from ACTIVE, FLAGGED or REINSTATED, NOTE from anything. So after choosing, adjustForState fixes the request. If the policy wants to FLAG a player who is already FLAGGED, it sends a NOTE with requiresHumanReview true instead, because Arena's moderator queue shows each player's latest action. That is why there are two enums: the kind is NOTE (what Arena receives) but the label stays FLAG_FOR_REVIEW (what the Judge records), so the human-review rate is counted honestly.

Why 0.90, when Arena only needs 0.80?

Defense in depth. Arena's 0.80 floor protects against a bug in any client. The Judge's own 0.90 is stricter because a false ban is the worst outcome. Two independent safeguards means one bug is not enough to hurt an innocent player. noBanBelowTheArenaFloorEverIsProduced in DecisionPolicyTest loops confidence from 0.00 to 0.89 and asserts that no BAN is ever produced. The exact boundary is tested too: 0.90 bans, 0.899 flags for review, and 0.8996 does not auto-ban even though it rounds to 0.900 (the rule uses the raw number, rounding is only for what is sent).

5.11 JudgeEngine: the front door

JudgeEngine.java (shortened)
public static JudgeEngine create(JudgeSettings settings, MessagesGateway gateway, Clock clock) { ... }   // wires everything

public Investigation investigate(SuspiciousPlayer flag, ToolExecutor tools) {
    return router.investigate(flag, new MemoizingToolExecutor(tools));      // ONE cache per flag
}
public JudgeOutcome judge(SuspiciousPlayer flag, ToolExecutor tools) {                       // evaluation uses this
    return judge(flag, tools, PlayerStatus.ACTIVE);
}
public JudgeOutcome judge(SuspiciousPlayer flag, ToolExecutor tools, PlayerStatus status) {  // service uses this
    Investigation investigation = investigate(flag, tools);
    return new JudgeOutcome(investigation, decide(investigation, status));
}
public ArenaAction decide(Investigation investigation, PlayerStatus status) {                // re-run after an Arena conflict
    return policy.decide(investigation.verdict(), status, settings.promptVersion(), investigation.finalModel());
}

Both applications call this class. The service passes live Arena tools; the evaluation passes recorded ones. Everything else in the decision path is shared, so the evaluation measures exactly the code that runs in production. This is the answer to "how do you know your accuracy numbers are real?"

Notice decide is separate and does not call a model. If Arena answers "409 conflict" (another moderator acted in between), the service re-reads the player's status and calls decide again: no extra tokens spent.

5.12 Arena tools in the core: ArenaReader, ArenaToolExecutor

The four tools the model can call are answered by Arena, the game backend. Two programs need those answers: the live service (to judge real flags) and the evaluation's collect command (to record answers once, for later replay). If each wrote its own code to turn Arena's big responses into the small JSON the model sees, the recorded answers could differ from the live ones and the evaluation would measure the wrong thing. So that code lives once, in the core.

TypeWhat it is
ArenaReader (interface)Three questions you can ask Arena: the player, their recent matches, their moderation history. An empty result means "404, no such player". It does not say how to talk to Arena. The service answers with a fancy client (retry, circuit breaker); collect answers with a simple one. This is called dependency inversion: the core defines what it needs, and the outside world plugs in.
ArenaToolExecutorImplements ToolExecutor on top of an ArenaReader. It makes the compact results: the profile (username, status, account age in days, rating), the percentiles (with a note when the player has fewer than 5 matches), matches (headshot rate and accuracy computed to 3 decimals, never dividing by zero), history (reason cut to 200 characters). It drops private fields such as email and display name, which the model never needs.
ArenaUnavailableException"Arena could not be reached." The only exception that aborts a whole investigation: the flag is held and retried, never judged on missing data.
AttemptRecordOne model attempt flattened into plain numbers (model, turns, tokens, cost, latency). It appears on every judge-verdicts record and in every evaluation result, so it lives in the core and both applications reuse it.
Why the player is fetched only once The pre-check already calls GET /players/{id} to see whether the player is banned. The executor receives that response in its constructor, so get_player_profile and get_stat_percentiles are answered without a second call. Arena is read once per player for those two tools.

Where this appears in a test: ArenaToolExecutorTest checks that the profile has exactly the documented fields (and no email), that missing percentiles become null plus a note, that a match with zero kills has a headshot rate of 0 instead of a crash, and that an unreachable Arena throws.

6. judge-service built

The Spring Boot application that connects the core to the outside world: it reads flags from Kafka, asks the core for a verdict, acts in Arena and publishes the verdict. It is tested against a real Kafka broker and a WireMock container that pretends to be both Claude and Arena.

The classes

ClassWhat it does
ArenaClientLogs in as judge-agent, caches the token, logs in again once on a 401. Reads the player, matches and moderation history, and posts the moderation action. Has its own retry and circuit breaker. Implements the core's ArenaReader.
ArenaActionApplierSends the chosen action to Arena. On a 409 (another moderator acted first) it re-reads the status and re-runs engine.decide once; a second 409 is recorded as a NOTE with no Arena action.
PreChecksBefore any tokens are spent: unknown player or not role PLAYER, already BANNED or APPEALED. JudgeService adds the third check, judged in the last 30 minutes.
JudgeServiceOne flag from start to verdict record: pre-checks, JudgeEngine.judge, then ArenaActionApplier.
FlagListener@KafkaListener: parse the flag (malformed means dead letter), skip it if already judged, judge it, then publish the verdict and wait for the broker to acknowledge.
ProcessedFlags, ProcessedFlagsLoader, VerdictSnapshotReaderAt startup, replay the compacted judge-verdicts topic into memory, and only then start the listener. There is no database: that topic is the Judge's only state.
KafkaConfigDeclares the two topics the Judge writes, and the error handler: retry forever every 5 seconds for outages, dead letter only NonRetryableJudgeException.
JudgeMetricsPrometheus counters, timers and histograms: verdicts, tokens, cost, latency, escalations, invalid outputs.
AnthropicConfig, ArenaConfig, StartupChecksBuild the one SDK client (maxRetries(0)), the resilience stack and the engine. The service refuses to start if ANTHROPIC_API_KEY or the Arena password is missing, and says which, never the value.
application.ymlKafka consumer and producer settings, port 8083, Arena URL, topic names. Key line: max-poll-records: 1 with a long poll interval, because one flag can take 10 to 40 seconds of model time and Kafka must not think the consumer died.

The integration tests

Following one flag through the service, slowly

Imagine Telemetry just put this message on the topic suspicious-players: "player sim_0042 looks like an aimbot". Here is exactly what happens, in order.

  1. Kafka hands the message to FlagListener. It parses the JSON. If the text is not valid JSON, or has no flag id or player id, it throws NonRetryableJudgeException, and the error handler copies the message to the dead-letter topic. There is no point retrying garbage.
  2. "Have I judged this flag before?" ProcessedFlags is a map of flag ids already answered. If the id is there, the listener counts a duplicate and returns. This is what makes a restart safe.
  3. JudgeService asks Arena about the player (one call). Not a normal player, or already banned? Write a SKIPPED verdict and stop: no tokens spent. Judged within the last 30 minutes and then banned or flagged? Also skip.
  4. The engine investigates. Haiku calls tools (answered by ArenaToolExecutor), then returns a verdict. Maybe Sonnet takes a second look.
  5. DecisionPolicy picks one Arena action and ArenaActionApplier sends it with source: JUDGE. If Arena says 409 ("the player's state changed while you were thinking"), re-read the status and decide once more.
  6. Only now does the listener publish the verdict record to judge-verdicts and wait for the broker to say "stored". Then it remembers the flag id.
  7. Kafka's offset moves on. The offset is Kafka's bookmark. It is saved only after the method returns without an exception, so a crash anywhere above means the same flag comes back after the restart.
Why "publish after acting"? If we published first and then Arena was down, the topic would claim a ban that never happened. This way the worst case is the opposite: Arena acted but the record is missing, so the flag is judged again, and the pre-check or the 409 rule makes that harmless. This is the at-least-once trade-off explained below.

Why an outage does not lose flags

If Claude or Arena is unreachable after the retries, JudgeService throws. The listener does not catch it. The error handler waits 5 seconds and offers the same message again, forever, without moving the bookmark. Messages behind it in that partition wait. That is intended: it keeps order per player and drops nothing. ResilienceIT proves it: 20 flags during a simulated 20-second outage, none judged and none dead-lettered while it lasts, and all 20 judged after it ends.

The breaker, in numbers from the test

MomentWhat the circuit breaker does
First 10 calls fail with HTTP 500Still closed: it needs at least 10 calls in its window of 20 before judging the failure rate.
Failure rate reaches 50%Switches to open: further calls fail instantly without touching the network (the service is not hammered while it is sick).
After the open time (2 s in tests, 30 s in production)Half-open: lets 3 trial calls through. If they succeed it closes; if not, it opens again.

The exactly-once question

Kafka delivers at least once. If the service dies after sending the Arena action but before publishing the verdict, the flag is judged again. The second time the pre-check sees the player is already BANNED and records SKIPPED, or Arena answers 409 and the policy's retry rule handles it. The worst case is a duplicated NOTE in an audit trail, which is acceptable. This trade-off is meant to be written in the README: a good interview answer is "I chose at-least-once plus idempotent handling instead of chasing exactly-once".

7. judge-eval code built, no data yet

A command-line application that answers: "is this prompt actually good, and what does it cost?" It does not call Arena while judging; it replays recorded data. The code and its tests are finished. The dataset and the reports do not exist yet, because they need a finished Telemetry run and an API key (needs you, see evals/README.md).

  1. collect (DatasetCollector, CaseSampler): build a dataset of 200 labeled cases (about 120 cheaters, 80 honest-but-flagged players, including pros). The trick: run Telemetry once with loosened thresholds so strong honest players also get flagged, because a normal run produces too few hard negatives. For each case, store the flag and one saved answer for each of the four tools, made by the same ArenaToolExecutor that answers the model in production. Labels never appear in the flag or the saved data.
  2. run --prompt=v1 --mode=routed --subset=all (EvalRunner): judge every case with the production JudgeEngine, using RecordedToolExecutor in place of Arena. Modes: routed (normal), baseline (Sonnet only, no cache), haiku-only. It writes a .jsonl with every case, a .summary.json and a Markdown report (ReportWriter, Scorer).
  3. compare (ComparisonWriter): put all runs side by side in COMPARISON.md, with the cost saving split into routing and caching.

A run, step by step

  1. run --prompt=v1 --mode=routed --subset=all loads the 200 cases and builds the settings for the mode (EvalMode): routed keeps the shared defaults, baseline turns off Haiku and caching, haiku-only turns off escalation.
  2. EvalRunner judges every case on its own lightweight (virtual) thread, at most 4 at a time, all sharing one rate limiter. Each case gets a RecordedToolExecutor that answers from the case's saved snapshots, cutting the two lists to the requested size.
  3. The player is treated as ACTIVE and the service's pre-checks are skipped: every case is judged. Everything else (JudgeEngine, routing, DecisionPolicy) is the production code.
  4. A case where Claude stays unreachable is recorded as an error and left out of the scores. The report lists it so you can re-run.
  5. Scorer turns the 200 results into numbers, and ReportWriter writes three files: every case (.jsonl), the numbers (.summary.json) and a readable report (.md) that ends with every false ban and every missed cheater, with the evidence the model gave. That last list is what you read to write the next prompt version.
The nearest-rank percentile p95 of 200 costs: sort them, take rank ceil(0.95 * 200) = 190, so the 190th smallest value. No averaging between neighbours.

The scoring terms

TermMeaning
PrecisionOf the players the Judge banned, what share were real cheaters? (Low precision = innocent people banned.)
RecallOf all the cheaters, what share were banned? (Low recall = cheaters get away.)
False-ban reductionThe "baseline" bans every flag, so its false bans equal the number of honest cases. The Judge's number is compared against that.
Review rateShare of cases sent to a human. Costs moderator time; target about 12%.
Dev vs holdoutYou edit prompts while looking at 50 "dev" cases. The other 150 ("holdout") are never looked at, so their numbers are honest. This prevents tuning the prompt to the test.

Why replay recorded tool data? It is repeatable and cheap, and it measures the model and prompt instead of Arena's changing state. Why a holdout? Otherwise you would be marking your own homework.

8. The tests: how we know it works

Run everything with ./mvnw verify (no API key needed; Docker must be running, because the integration tests start Kafka and WireMock containers). It compiles, runs the tests, and fails if less than 85% of the lines of a module are executed by tests. Current numbers: 194 tests in core (97.2%), 77 unit and 12 integration tests in service (96.9%), 105 unit and 1 integration test in eval (96.2%).

Test helpers (shared as a "test-jar")

HelperWhat it is
ScriptedGatewayA fake MessagesGateway. Queue model replies with .then(message) or failures with .thenFail(exception); afterwards read .requests() to see exactly what the code sent.
TestMessagesBuilds SDK Message objects from API-shaped JSON: toolUse(...), finalText(...), stoppedWith("max_tokens", ...), verdictJson(...).
StubAnthropicServerA local HTTP server impersonating the Claude API with scripted replies (429, 529, reset, slow...).
FakeToolExecutorReturns fixed results per tool and records every call.
MutableClockA clock tests can set and advance, so latency is a number you control.
TestSettingsThe default settings built in Java, shared by tests.

What each test class proves

Test classProves
JudgeSettingsTestEach validation rule rejects bad values (ban floor 0.80, confidences 0 to 1, positive durations, both models priced, prompt files present).
VerdictParserTestValid JSON parses; lower-case enums work; extra text, fences, missing and extra fields, wrong types all fail as PARSE.
VerdictValidatorTestEvery rule at its boundary.
VerdictSchemaTestThe schema matches the contract; every object forbids extra properties.
PromptLibraryTest, UserPromptBuilderTestCache marker only when caching is on; no dates or ids in cached text; no labels in prompts.
ToolCatalogTestFour tools, exact names, strict, player_id required, only two take limit.
ToolExecutorGuardTestAnother player's id never reaches the backend; unknown tools, bad input and bad limits are error results.
MemoizingToolExecutorTestSame question asked once; a different limit asks again; exceptions are not cached.
AgentLoopTestThe loop's behaviour in 21 scenarios (section 5.6).
AnthropicResilienceTestRetry-After parsing, which exceptions retry, config values, jitter bounds.
ResilientMessagesGatewayTestReal SDK client against the stub server: retries, breaker open and recovery, rate limiter, non-retryable errors.
CostCalculatorTestHand-computed costs for both models, with and without cache tokens.
ModelRouterTest, JudgeEngineTestEvery escalation reason, fresh Sonnet conversation, shared tool cache, baseline modes, costs and latency; the engine's entry points.
DecisionPolicyTestEvery row of the table, the 0.90 boundary, the state-machine adjustments.
JudgeConfigBindingTestThe real YAML files bind to the Java records; env var overrides work.
SdkSmokeTest(tag live, excluded from verify) One real round trip to Claude. Not yet run.

How to read a test

Every test has three parts, often called arrange, act, assert. Here is a real one:

ModelRouterTest.java
@Test
void lowConfidenceEscalatesToSonnetWhichGivesTheFinalVerdict() {
    gateway.then(TestMessages.finalText(ban(0.78, 2)))                                        // ARRANGE: Haiku says 0.78
           .then(TestMessages.finalText(TestMessages.verdictJson("BAN", "AIMBOT", 0.96, false, 3)));  // then Sonnet says 0.96

    Investigation result = investigate(flag(SkillTier.GOLD));                                 // ACT

    assertThat(result.verdict().confidence()).isEqualTo(0.96);                                // ASSERT
    assertThat(result.attempts()).extracting(Attempt::model).containsExactly(HAIKU, SONNET);
    assertThat(result.attempts()).extracting(Attempt::outcome).containsExactly(Outcome.ESCALATED, Outcome.FINAL);
    assertThat(result.attempts().get(0).escalation()).isEqualTo(EscalationReason.LOW_CONFIDENCE);
    assertThat(gateway.requests()).extracting(ModelRouterTest::modelOf).containsExactly(HAIKU, SONNET);
}

The test name is a sentence describing the behaviour. Reading just the method names of a test class tells you what the class is supposed to do, which is why you read tests right after the code.

Why the real API is never called in tests

Tests must be free, fast, repeatable and runnable in CI without secrets. A model's answers vary between runs and cost money. So every test replaces the model with a script. The one exception is SdkSmokeTest, tagged @Tag("live") and excluded from verify. It exists to confirm the real request shape is accepted. Be honest about this limit: the fakes only check my understanding of the API, which is exactly why the live test still matters.

9. Build files

The parent pom.xml

The module POMs

ModuleDepends onNotable
judge-coreAnthropic SDK, Jackson 3, Resilience4j (retry, breaker, limiter), Micrometer core, slf4j-apiNo Spring, no Kafka. Builds a test-jar so other modules reuse its test helpers. No spring-boot-maven-plugin: repackaging would make it unusable as a dependency.
judge-servicecore, Spring MVC, Kafka, Actuator, Prometheus, resilience4j-micrometerHas the Spring Boot plugin, so it builds a runnable jar.
judge-evalcore, Kafka, Spring RestClientRunnable jar; no web server.

Everyday commands

./mvnw verify                                              # build, test, coverage check (all modules)
./mvnw -pl judge-core test                                 # one module's tests
./mvnw -pl judge-core test -Dtest=AgentLoopTest            # one test class
./mvnw -pl judge-core test -Dgroups=live -Dtest.excludedGroups=none   # the live smoke test (needs ANTHROPIC_API_KEY)

On Windows PowerShell use .\mvnw.cmd instead of ./mvnw.

10. One flag, with real numbers

This worked example uses prices straight from judge-defaults.yml. The token counts are illustrative numbers I chose to show the arithmetic, not measurements. The exact method is the one in CostCalculator.

The flag: player sim_0042, tier GOLD, rule AIMBOT, 23 shots, 17 hits, 14 headshots (rate 0.8235), average reaction 61 ms.

Case A: Haiku is confident, no escalation

Model callWhat happensinputcache readoutput
1Asks for get_player_profile and get_stat_percentiles4503,00090
2Asks for get_moderation_history1,4003,00080
3Answers: BAN, AIMBOT, confidence 0.95, 3 evidence items1,9003,000200
Total3,7509,000370
Haiku cost = (3,750 * 0.10  +  9,000 * 0.01  +  370 * 0.50) / 1,000,000
           = (375 + 90 + 185) / 1,000,000
           = $0.000650
  1. EscalationRule: certainty = max(0.95, 0.05) = 0.95, not below 0.85. Decision is BAN, but GOLD is not top tier, and 3 evidence items is at least 2. No escalation. Attempt outcome FINAL.
  2. DecisionPolicy (player ACTIVE): BAN, 0.95 ≥ 0.90, no human review wanted, so row 2: send BAN to Arena with confidence 0.950 and reason [Judge v1, claude-haiku-5-5] ....

For comparison, imagine Sonnet doing the same work with no caching (every input token at full price, so 12,750 input tokens):

Sonnet cost, no cache = (12,750 * 2.00  +  370 * 10.00) / 1,000,000 = (25,500 + 3,700) / 1,000,000 = $0.029200
Haiku with cache      = $0.000650     ->  about 45 times cheaper for this flag

Sonnet with the cache would be (3,750*2.00 + 9,000*0.10 + 370*10.00) / 1,000,000 = $0.012100, so caching alone saves about 59% and routing the rest. This is exactly why the project plans separate runs (haiku-only, and routed with caching off) to split the saving into "routing" and "caching" parts. Treat these as arithmetic, not results.

Case B: Haiku is unsure, Sonnet is called

Same Haiku attempt, but it answers BAN with confidence 0.78.

  1. Certainty = max(0.78, 0.22) = 0.78, below 0.85: escalate with reason LOW_CONFIDENCE. The Haiku attempt is recorded as ESCALATED, reason "confidence 0.78 is below 0.85 in either direction", cost $0.000650 (still paid).
  2. Sonnet starts a new conversation containing only the flag. It asks for the same tools, but they are answered from the shared cache: Arena is not called again.
  3. Say Sonnet uses input 2,350, cache write 3,000 (its own cache is cold), cache read 3,000, output 310:
    Sonnet cost = (2,350 * 2.00 + 3,000 * 2.50 + 3,000 * 0.10 + 310 * 10.00) / 1,000,000
                = (4,700 + 7,500 + 300 + 3,100) / 1,000,000 = $0.015600
  4. Sonnet answers BAN, 0.93, 3 evidence: FINAL. Total for the flag: $0.000650 + $0.015600 = $0.016250.
  5. Policy: 0.93 ≥ 0.90 so BAN, reason [Judge v1, claude-sonnet-5-5] ... (the reason names the model that gave the final verdict).

Case C: a top-tier player

Same as A but the player is PRO. Haiku is confident (0.95) with 3 evidence items, yet the rule HIGH_TIER_BAN escalates anyway: pros legitimately look extreme, and banning one by mistake is costly, so a stronger second opinion is always taken.

Case D: everything fails

Haiku returns prose instead of JSON (INVALID, kind PARSE), Sonnet also fails. No verdict. Policy row 1: FLAG with human review, reason "Judge could not reach a verdict". The flag is not lost: a human sees it.

Case E: a whole run, scored optimal imaginary value

All numbers in this case are optimal imaginary values: what a good prompt version could reach on the 200 cases (120 cheaters, 80 non-cheaters). They are chosen so the arithmetic is consistent; they are not measured.

Count (optimal imaginary value)Cheaters (120)Non-cheaters (80)
BAN1124
FLAG_FOR_REVIEW618
NOTE or CLEAR258
banned            = 112 + 4 = 116
precision         = 112 / 116        = 96.6%     (of those banned, how many were really cheating)
recall            = 112 / 120        = 93.3%     (of the cheaters, how many were banned)
false-ban cut     = 1 - 4 / 80       = 95.0%     (a rule-only baseline bans all 80, so it makes 80 false bans)
review rate       = (6 + 18) / 200   = 12.0%     (sent to a human moderator)

Notice what the 6 reviewed cheaters mean: they were not banned, so they count against recall, but a human will see them. That is the price of banning only when sure.

Case F: the cost split optimal imaginary value

The mean cost per verdict over 200 cases, from four runs of the same prompt version. All four values are optimal imaginary values, built from the same price list and token sizes as cases A and B.

RunMean cost per verdict (optimal imaginary value)Compared with the baseline
baseline (Sonnet only, no cache)$0.0158-
routed, caching off$0.003578% cheaper: this is the routing saving
routed, caching on$0.002485% cheaper in total; 31% cheaper than routed without caching: this is the caching saving
total saving   = 1 - 0.0024 / 0.0158 = 84.8%      (target: at least 65%)
routing saving = 1 - 0.0035 / 0.0158 = 77.8%
caching saving = 1 - 0.0024 / 0.0035 = 31.4%

The two parts do not add up to the total: they multiply (0.222 × 0.686 = 0.152 of the baseline cost remains, so 84.8% is saved).

11. Decisions and deviations from the written spec

An important habit: when reality disagrees with the plan, write down what you did and why. Each item below is marked // DECISION in the code or recorded in handoff.md. These are strong interview stories, because they show judgment.

WhatSpec saidWhat I didWhy
Low-confidence ruleescalate if confidence < 0.85escalate if max(c, 1-c) < 0.85The literal rule would escalate every CLEAR and WATCH, making the cost target unreachable (5.9). Still open for the owner to confirm.
Message constructionMessageParam.builder()...MessageParam.ofUser(...)The builder does not exist in SDK 3.10.0. Found with javap on the real jar.
JSON mapper accessJsonValue.jsonMapper()ObjectMappers.jsonMapper()Same: the spec's name does not exist.
Excluding the live test<excludedGroups>live</excludedGroups> plus -Dgroups=liveexclusion through a test.excludedGroups property; live run passes -Dtest.excludedGroups=noneSurefire combines groups and excludedGroups with AND, so the spec's combination would run nothing.
Haiku-only modemode exists but no config keynew key routing.escalation-enabled"Experiments change config, never code."
limit range in tool schema"integer 1 to 20"range in the description, enforced by GuardedToolExecutorStrict tool schemas do not support numeric range keywords.
The tool guarddescribed as rulesa decorator class, always applied inside AgentLoopSafety by construction: no caller can forget it.
Failure costexceptions carry no dataexceptions carry AgentStatsA failed attempt still costs tokens; the cost record must be honest.
Settings binding"bind to validated records"validation in record constructors; Spring's Binder in the serviceKeeps judge-core free of Spring; same style as Telemetry.
CLEAR note textreason "starts with Cleared by Judge:" and also gets the [Judge ...] prefixCleared by Judge: [Judge v1, model] explanationSatisfies both statements.
408 and 409 from Clauderetry only 429, 5xx and networkfollowed the specThe SDK would retry them by default; noted in handoff.md in case they appear in practice.

12. Interview preparation

Interviewers for a project like this rarely ask you to recite code. They ask why, and then they follow up on your answer. The pattern that works: state the decision, give the reason, name the trade-off, and say how you checked it. Click each question to reveal a sample answer in plain words. Say it in your own words; do not memorise.

12.1 Your pitch

"The Judge is an LLM moderation agent in Java. It takes flags from a real-time cheat detector, investigates the flagged player through four read-only tools backed by the game's API, and returns a schema-checked verdict: ban, watch or clear, with cited evidence and a confidence. The key design rule is that the model only recommends; a plain Java policy chooses the single action, with a stricter ban threshold than the backend demands. A cheap model triages every flag and a stronger one gives an independent second opinion when needed, with prompt caching to cut cost. I wrote the agent loop by hand, with retries, a circuit breaker and a rate limiter, and I built an evaluation harness to measure precision, recall and cost per prompt version."

Adjust for what is true today. If you are interviewing before steps 8 to 12 are done, say "the core is built and tested; the service and evaluation are next". Never claim numbers you have not measured.

12.2 Design and architecture

Why does the model only recommend, and code decide the action?

Because a model's output is text produced by a probabilistic system, and the consequence here, banning someone, is serious and hard to undo. If the model could choose an HTTP call or a URL, a prompt injection or a plain mistake could do real damage. So the model returns a Verdict (data). DecisionPolicy is a pure function with seven rows that is trivially testable: a unit test loops confidence from 0 to 0.89 and asserts no BAN is ever produced. The trade-off is less flexibility: the model cannot do anything the policy does not allow, which is the point.

Why two models? Why does Sonnet start from scratch?

Cost and quality. Haiku is roughly twenty times cheaper per token, and most flags are easy. Sonnet is used only when Haiku is unsure, fails, or wants to ban a PLATINUM or PRO player, where a false ban hurts most. Sonnet gets a fresh conversation without Haiku's answer, because a model shown another model's conclusion tends to agree with it; an independent second opinion is worth more than a rubber stamp. The trade-off is that escalated flags cost both attempts. Both attempts share one tool cache so Arena is read once.

Why a threshold of 0.90 when Arena already enforces 0.80?

Defense in depth. Arena's floor protects the system from any buggy client. The Judge's stricter threshold reflects that a false ban is the worst outcome and a missed cheater is less bad, because the detector keeps watching. Two independent checks mean one bug is not enough. Startup validation also refuses a configured threshold below 0.80.

How do you make sure the model cannot look up other players?

GuardedToolExecutor checks every tool request: known tool name, object input, only allowed properties, and player_id must equal the flagged player, otherwise it returns an error result and never calls the backend. AgentLoop applies the guard itself, so no caller can skip it. It also canonicalises the input. A unit test asserts the backend's call list stays empty when another id is requested.

Why a hand-written loop instead of LangChain or Spring AI?

The agent loop is short, about 80 lines, and it is exactly where the hard parts live: several tool calls in one response must be answered in one message in order, limits on turns and tool calls, partial failures that still cost tokens, and different handling for model failures versus infrastructure failures. Owning it makes every behaviour testable and explainable and avoids a framework hiding retries or caching. The trade-off is I maintain more code, and I lose framework conveniences I did not need.

Where does state live? Do you have a database?

No database. The planned design uses the compacted Kafka topic judge-verdicts as the only state, replayed into memory at startup to know which flags are done. Fewer moving parts. The trade-off is memory proportional to the number of verdicts and a startup replay, fine at this scale.

What does "idempotent" mean for the Kafka listener and why do you need it?

Kafka delivers at least once: if the process dies before saving its position, the flag is delivered again. So handling a flag twice must be harmless. The design checks "already judged" first, pre-checks the player's status (a banned player is skipped), and Arena returns a conflict if the state already changed. The worst case is a duplicate NOTE in an audit trail. I chose at-least-once plus idempotent handling rather than trying to get exactly-once across Kafka and an HTTP API, which is not really achievable.

12.3 The LLM-specific questions

How does prompt caching work here, and what breaks it?

The API caches a prefix of the prompt, in the order tools, system text, then messages. I put everything identical across flags first: the four tool definitions, the role text, the rulebook, and one cache marker at the end of the rulebook. The flag and tool results come after, uncached. Cache reads cost about a tenth of normal input, so the roughly 3,000-token rulebook is nearly free after the first call. Anything that changes inside the cached part, such as a timestamp or id, invalidates it, so a unit test checks the system text has no dates or UUIDs. Also, the cached prefix must be at least 512 tokens on these models, and each model warms its own cache. I have not yet confirmed a cache hit on the live API.

What is structured output, and why validate again in code?

I send a JSON schema so the API constrains the answer to six fixed fields with allowed values. It guarantees the shape but not numeric ranges or lengths, which are only hints. So VerdictValidator enforces confidence in 0 to 1, 1 to 6 evidence items, text lengths, and the rule that BAN needs a cheat type. The parser is deliberately strict: extra text, markdown fences, missing or unknown fields are parse failures. Parse and validation failures are counted separately, so I can see which kind of problem a prompt version causes.

What happens if the model's answer is cut off or it refuses?

The stop reason tells me. Only end_turn carries a verdict. max_tokens, refusal or anything else becomes an InvalidVerdictException, the router escalates to Sonnet once, and if that also fails the policy sends the flag to human review instead of dropping it. The tokens spent on the failed attempt are still recorded and priced.

What if the model asks for several tools in one response?

All requested tools are executed and all results go back in one user message, in the same order, because the API requires every tool_use block to be answered in the next message. There is a test with two tools in one turn that checks order and message count. There is also a limit of 8 tool calls and 6 model turns per attempt so the loop cannot run forever or cost unboundedly.

How do you know the model is not just guessing?

That is what the evaluation harness is for (step 11: the code is built, but it has not run on real data yet): 200 labeled cases, with honest-but-flagged players as hard negatives, replayed through exactly the production code. Precision, recall and false bans are measured per prompt version, with a dev set used for tuning and a holdout set that nobody looks at while editing. Today I cannot give a measured accuracy, and I say so.

What is the "LOW_CONFIDENCE" decision you made?

The spec escalates when confidence is below 0.85. But confidence is the probability that the player cheats, so a confident innocent verdict has confidence near 0.05. Taken literally, every CLEAR and WATCH escalates and cost savings vanish. I escalate when the cheating probability is between 0.15 and 0.85, meaning max(c, 1-c) < 0.85. I documented it with a // DECISION comment, flagged it to the owner, and it is a one-line change to compare the literal rule in the evaluation.

How would you reduce cost further?

Measure first: the spec plans separate runs to split the saving into routing and caching. Then options: shorten the rulebook (tested as a prompt version), cache hits across more flags by keeping traffic steady inside the 5-minute cache window, return smaller tool results, lower the output token limit, and raise Haiku's confidence calibration so fewer flags escalate. The Batch API would help offline evaluation but not live moderation.

12.4 Resilience

Explain the order of retry, circuit breaker and rate limiter.

Retry is the outermost, so each attempt goes through the breaker and the limiter. A retried call therefore counts against the rate limit, and once the breaker opens, the next retry is refused immediately instead of hammering a failing service. If the breaker wrapped the retry it would see only the final outcome and miss the failing attempts.

Why maxRetries(0) on the SDK client?

The SDK retries by itself. If Resilience4j also retries, retries nest: roughly 3 times 4 attempts per failure, which multiplies load on an already struggling service. One retry layer, mine, is predictable and observable.

Which errors do you retry, and which do you not?

Retry: 429 (rate limited, honouring Retry-After), any 5xx including 529 overloaded, and network errors (timeouts, resets). Not retried inside the call: 400 (my request is wrong, it becomes a non-retryable exception and goes to the dead-letter topic), 401 and 403 (retrying in a tight loop will not fix a bad key; the flag is held and retried every few seconds so work resumes when someone fixes the key). Only 5xx and network errors count as failures for the circuit breaker, because 429s and 4xx do not mean the service is down. I verified the real SDK's exception mapping against a local stub server.

What is jitter and why add it?

A random variation on the wait between retries. Without it, many clients that failed together retry together and cause another spike (a thundering herd). Resilience4j's exponential random backoff spreads them out.

Can a flag be lost?

By design, no. Temporary failures (Claude down, Arena down, circuit open) throw exceptions that the Kafka error handler retries forever without saving the offset. Only a malformed flag or a request Claude rejects as invalid goes to the dead-letter topic. If both models fail to produce a verdict, the flag goes to a human instead of disappearing. (The Kafka part is planned, step 9; the exception types and the "no verdict becomes a review flag" policy row exist now.)

12.5 Java and code quality

Why records everywhere?

They are immutable, concise and give equals, hashCode and toString for free, which makes tests simple: assertThat(a).isEqualTo(b). Compact constructors let me validate on construction so an invalid object cannot exist. Immutability also makes sharing across virtual threads safe.

Why inject a Clock instead of calling Instant.now()?

So time is controllable in tests. The router measures attempt latency from the injected clock; in tests a MutableClock advances 100 ms per model call so I can assert exact latencies without sleeping. It also makes time-based logic (like "judged in the last 30 minutes") testable.

Why is judge-core free of Spring and Kafka?

So the decision path is testable with plain JUnit, fast and without containers, and so the service and the evaluation can both depend on exactly the same logic. The service adapts the world (Kafka, HTTP) to the core's interfaces.

What is the decorator pattern, where do you use it?

An object that implements an interface by wrapping another implementation of the same interface and adding behaviour. GuardedToolExecutor adds safety checks, MemoizingToolExecutor adds caching, and the gateway's rate limiter, breaker and retry are decorators around a call. Each concern is separate and independently testable, and they compose.

Why BigDecimal for cost?

Binary floating point cannot represent most decimals exactly, so sums of tiny prices drift. BigDecimal is exact; I round once to six decimals at the end.

12.6 Testing

How do you test code that calls an LLM?

At three levels. For the loop and routing, a scripted fake gateway plays back prepared model replies and records the requests, so every branch is tested deterministically and for free. For the resilience stack, the real SDK client talks to a local stub HTTP server that returns 429, 529, resets and slow responses. Later, service integration tests use real Kafka and a WireMock container. Only one tagged smoke test touches the real API, run by hand. And what the fakes cannot prove, namely that the real API accepts my request, is exactly what that smoke test and the evaluation are for.

What does 97.9% coverage actually tell you?

That tests executed that share of lines. It does not tell you the assertions are good. I treat it as a floor (the build enforces 85%) and rely on specific behaviour and boundary tests: confidence 0.90 versus 0.899, 6 versus 7 evidence items, exactly-one-user-message in Sonnet's first request.

12.7 Honesty and "what next" questions

What would you do differently / what is missing?

The evaluation harness is built but has never run on real data, so no accuracy is measured yet. The live smoke test has not been run, so the real API request shape is unverified. The synthetic dataset will have its own limits: Arena's statistics are snapshotted after the simulation rather than at flag time. Next steps are collecting the dataset, running six prompt versions and measuring. I would also want to review cases where the model said BAN on a PRO player.

Tell me about a bug or surprise you hit.

The spec's code sample for building a message used a builder that does not exist in the SDK version I pinned. I found it by inspecting the real jar with javap before building on it, switched to the union factory methods, and recorded the difference. Another: combining the spec's exclusion of the live test with the way I was told to run it would have selected no tests at all, because Surefire ANDs the two filters; I moved the exclusion into a property.

How would you explain this project to a non-technical person?

"A fast alarm says a player might be cheating. My system plays the careful referee: it looks at that player's whole history, decides whether to ban, watch or clear, and only does the drastic thing when it is very sure. When it is not sure, it hands the case to a human."

12.9 Result questions (answers use optimal imaginary values)

These answers quote numbers. Every number is an optimal imaginary value from section 12b, not a measurement. When you have run the evaluation, replace each number with your real one and keep the sentence structure.

What precision and recall did you get, and on which data?

Say: "On the 150 holdout cases, which I never looked at while editing prompts, the best version reached about 96.6% precision and 93.3% recall (optimal imaginary values). Precision is how many of the bans were real cheaters; recall is how many cheaters got banned. I report holdout because dev numbers are optimistic: I read the dev mistakes to write each prompt version."

Why is recall lower than precision, and is that a problem?

Say: "It is by design. Banning an innocent player is the worst outcome, so the system only bans above 0.90 confidence without a human-review request. About 6 of 120 cheaters went to the review queue instead of being banned (optimal imaginary value): they count against recall, but a moderator sees them. I report the review rate separately so nobody thinks those are lost."

How did you get 85% cheaper?

Say: "Two independent savings, measured with four runs of the same prompt. Routing: Haiku handles most flags and Sonnet only the unsure ones, which was about 78% cheaper than Sonnet alone (optimal imaginary value). Caching: the tools, system prompt and rulebook are identical on every call, so after the first call they are read from cache at a tenth of the price, which cut the cost of the routed setup by another 31% (optimal imaginary value). They multiply rather than add, so the total was about 85%, from $0.0158 to $0.0024 per verdict (optimal imaginary values)."

What did the prompt iterations change, and did they help?

Say: "Six versions, one change each, each motivated by reading the false bans and missed cheaters in the dev report. For example, requiring the match history before any aim-based ban, and stricter wording for pro players. Precision went from about 91.5% to 96.6% (optimal imaginary values). One version, a shorter rulebook, was meant to lower cost and left accuracy unchanged, which is a result too."

Is 100% valid output believable?

Say: "Two layers make it believable: the API is told the JSON schema and enforces the structure, and my parser then rejects anything else, including extra text or an unknown enum value. Over about 1,200 runs there were no parse failures (optimal imaginary value). I report parse failures and validation failures separately, because a value outside a limit (confidence 1.2) is a different problem from broken JSON. If one ever fails, the second model gets a turn, and if that fails too the player goes to a human."

What did the remaining false bans have in common?

Say: "That is the question the report answers: it lists every false ban with the evidence the model quoted. In this imaginary run, say all 4 were PRO players with extreme aim stats but normal reaction times (optimal imaginary value). The fix would be a stricter rule for that tier or requiring two independent signals. The honest approach is to read the list, not to hide it."

What was the cached prefix size, and why does it matter?

Say: "The cache only works above a minimum size (512 tokens for these models). The first call reported about 1,950 tokens written to the cache (optimal imaginary value), which is the tools, the system prompt and the rulebook, comfortably above the minimum. Later calls read it back. I confirm that by checking cacheCreationInputTokens on the first call and cacheReadInputTokens on the next."

What is the one number you are sure of?

Say: "The resilience result, because the test is deterministic: in ResilienceIT I send 20 flags while the simulated API answers HTTP 500 for 20 seconds. The circuit breaker opens, nothing is dropped or dead-lettered, and after the outage all 20 flags get a verdict. 20 of 20, measured, not imaginary."

12.8 A three-minute walk-through you can practise

  1. The problem: noisy detector, false bans are costly (20 seconds).
  2. The flow: flag, Haiku with tools, maybe Sonnet, policy, action (40 seconds). Draw the five boxes from section 4.
  3. Safety: model recommends, code acts; guard; two thresholds; nothing dropped (40 seconds).
  4. Cost: routing plus caching, measured per attempt in BigDecimal (30 seconds).
  5. Resilience: retry outermost, breaker, limiter, maxRetries(0) (30 seconds).
  6. Evaluation: labeled cases, dev vs holdout, same code path as production (20 seconds).
  7. Honest status and next steps (20 seconds).

12b. The results table optimal imaginary value

This is what COMPARISON.md would show after six prompt versions, if you had run everything. Every number is an optimal imaginary value: I chose values a well-tuned system could plausibly reach, consistent with the price list. Replace them with your own after running evals/README.md. The changes in the last column are the example changes the specification suggests; they are imaginary here because none of the versions v2 to v6 exists yet.

VersionChange (imaginary)PrecisionRecallFalse bans (of 80)
v1the starting prompt91.5%90.0%10
v2stricter wording for PRO aim cases93.2%90.8%8
v3require get_recent_matches before any aim-based ban94.0%91.7%7
v4recalibrated confidence bands95.7%92.5%5
v5shorter rulebook (cheaper, same accuracy)95.7%92.5%5
v6evidence in a fixed order96.6%93.3%4

All values above are optimal imaginary values.

Headline figure for v6, routedTargetValue (optimal imaginary value, not measured)
Ban precisionat least 95%96.6%
Cheater recallat least 92%93.3%
False bans fewer than rule-onlyat least 80%95.0% (4 instead of 80)
Cost per verdict lower than Sonnet-only without cacheat least 65%84.8% ($0.0024 against $0.0158)
Schema-valid outputs100% over 1,200 or more runs100% over 1,200 runs (6 versions × 200 cases), 0 parse failures
Human review shareabout 12%12.0%
Escalation rate(no target)18% of cases reached Sonnet
Latency per verdict, p50 / p95(report only)6.1 s / 13.8 s
Size of the cached prefix (tools + system + rulebook)at least 512 tokens1,950 tokens (first cacheCreationInputTokens)
Dropped verdicts under injected faults00 of 20: this one is measured (ResilienceIT, WireMock faults)
One honest detail about "optimal" These values hit every target on purpose. A real run may miss some: that is normal, and the spec says the measured number, not the target, goes on a resume. If recall comes out at 88%, you say 88% and explain what the missed cheaters in the report have in common.

13. How to practice reading code

Reading code is a skill you build by predicting, then checking. Try these in order. Each takes 10 to 20 minutes.

  1. Trace one test. Open AgentLoopTest.oneToolRoundThenAVerdict. Before reading AgentLoop, write down what you expect: how many model calls, how many messages in the second request, who is the last message from. Then read the loop and check.
  2. Break it on purpose. In DecisionPolicy change >= to > in the auto-ban comparison. Run ./mvnw -pl judge-core test -Dtest=DecisionPolicyTest. Which test fails, and why that one? (Answer: the 0.90 exact-boundary test.) Undo the change. This teaches you what tests protect.
  3. Change a rule. Make EscalationRule.certainty return confidence (the literal spec rule). Run ModelRouterTest. Which tests fail and what does that tell you about the decision in section 5.9?
  4. Follow the money. Use CostCalculatorTest.sonnetWithAllFourKindsOfTokens: recompute 11,500 by hand from the prices in judge-defaults.yml.
  5. Draw a sequence diagram on paper for "Haiku calls a tool, then answers": who calls whom, in order, from ModelRouter down to the gateway and back. Compare with section 4.
  6. Read only the test names of ModelRouterTest from top to bottom. Could you describe the router's behaviour from the names alone? That is the quality bar for test names.
  7. Explain it aloud. Take the Q&A in section 12 and answer five of them without looking, then compare. Record yourself if you can.

A method for reading any unfamiliar file

  1. Read the class's one-paragraph comment, then its constructor: what does it need to do its job? Those are its dependencies.
  2. Find its public methods. Everything else is detail.
  3. Read the public method top to bottom once without stopping. Then again, stopping at each if and throw: what situation does this handle?
  4. Open its test and read the test names. Pick the most interesting test and trace it.
  5. Search for where it is used (your editor's "find usages"). Now you know its place in the picture.

14. Glossary

TermMeaning
Agent loopRepeatedly call the model, run the tools it requests, feed results back, until it answers.
Aimbot, speedhackCheats: software that aims for the player; moving faster than the game allows.
At-least-once deliveryA message may arrive more than once, never zero times. Requires idempotent handling.
BigDecimalJava's exact decimal number type, used for money.
Boundary testA test at the exact edge of a rule (0.90 vs 0.899).
Cache read / writePrompt-cache token kinds: write stores a prefix (slightly dearer), read reuses it (about a tenth of the price).
Circuit breakerStops calling a failing service for a while so it can recover; open, half-open, closed.
Compact constructorA record's constructor block used to validate fields.
Dead-letter topic (DLT)Where records go when they can never be processed.
DecoratorA wrapper implementing the same interface and adding behaviour.
Defense in depthSeveral independent safeguards, so one failure is not enough to cause harm.
Dev / holdoutCases you may tune on / cases you must not look at while tuning.
EscalationHanding a flag from the cheap model to the strong model.
Fake / stub / mockTest stand-ins: a simple working version / canned answers / records how it was used.
FlagA Telemetry alert that a player looks suspicious in a short window (SuspiciousPlayer).
Haiku, SonnetClaude models: the small cheap one, and the larger stronger one.
IdempotentDoing it twice has the same effect as doing it once.
JacksonThe JSON library (Jackson 3 here: tools.jackson).
JitterRandom variation added to retry waits.
JSON schemaA description of the allowed shape of a JSON document.
Kafka topic / offsetA named durable log / a consumer's position in it.
MemoizeCache a function's result by its inputs.
Precision / recallShare of bans that were right / share of cheaters that were banned.
Prompt cachingThe API remembers a prompt prefix so repeated text is cheap.
Rate limiterCaps calls per time period.
RecordJava's immutable data class.
Resilience4jLibrary providing retry, circuit breaker and rate limiter.
Stop reasonWhy the model stopped: end_turn, tool_use, max_tokens, refusal.
Strict toolA tool whose input schema the API enforces exactly.
System promptThe standing instructions sent with every request.
TokenA chunk of text (about three quarters of a word); the unit of billing.
Tool / tool useA function the model can ask your program to run.
VerdictThe model's structured recommendation: decision, cheat type, confidence, evidence, explanation, review flag.
WireMock / TestcontainersA fake HTTP server in a container / a library that runs real services in Docker for tests (used from step 9).