Fairplay Telemetry, explained from scratch

A guided tour of Project 2 for someone who has never touched Kafka, Kafka Streams or Spring Boot. Every idea is explained the first time it appears, and every code sample is taken from the real files in this repository.

0. Where the project stands

The project was built in 12 steps (the plan comes from the written specification in the project root). All twelve are finished and committed to git. Everything marked built exists and is tested. The one thing that is described but not done is a live run against a real Arena (Project 1), see the table below.

StepWhatStatus
1Maven skeleton, four modules, Docker Compose with a Kafka brokerdone
2telemetry-common: shared data shapes, topic names, JSON conversion, test helpersdone
3Simulator core: the fake players, how they behave, how they generate eventsdone
4 to 6The detector's brain: the Kafka Streams topologydone
7The detector as a real Spring Boot app, tested against a real Kafkadone
8Simulator app: the engine that actually sends events to Kafkadone
9Verifier: scores the detector's accuracy and checks nothing was lostdone
10Dockerfile, three detector copies, Prometheus, Grafana, CI, the chaos scriptdone
11Arena integration (registering players and sending finished matches to Project 1)done (tested against a stub Arena)
12Measured benchmark runs and the READMEdone

Feature by feature

FeatureStateEvidence
Parse events, dead-letter bad onesbuilt and testedDeadLetterTopologyTest (7 tests), DetectorEndToEndIT
Integrity tracking (count, duplicates, gaps)built and testedIntegrityTopologyTest (12), ExactlyOnceIT on a real broker, and the measured run
Aimbot and speedhack windows, scoringbuilt and testedAimbotTopologyTest (10), SpeedhackTopologyTest (7), DetectionRulesTest (15)
Profile join and cooldownbuilt and testedProfileJoinTopologyTest (5), the cooldown tests inside the aimbot and speedhack classes
Spring wiring, health, metricsbuilt and testedWiringUnitTest (7). Prometheus now really scrapes all three detectors (checked in the measured run)
Simulator: core, engine, publisher, manifest, ground-truth publishingbuilt and tested99 unit tests and SimulatorIT (2, real Kafka). A real run sustained about 24,900 events per second
Verifierbuilt and tested46 unit tests and TopicSnapshotReaderIT (5, real Kafka)
Arena bridge and player registrationbuilt, stub-testedArenaBridgeIT (6, real Kafka plus a stub Arena) and the client and registrar tests. Not run against a real Arena.
Docker Compose, Prometheus, Grafana, CIbuilt and runThe full stack was started and measured. The CI workflow file exists but has not run on GitHub yet
Chaos script, benchmarks, READMEdonescripts/chaos-kill.sh, scripts/measured-run.sh, and the numbers in README.md

Real test numbers (run on 2026-11-19)

I ran ./mvnw verify for the whole project with Docker Desktop running. It finished with BUILD SUCCESS.

ModuleUnit testsIntegration tests (real Kafka in Docker)Line coverage (JaCoCo)
telemetry-common241 (KafkaTestContainerIT)100% (98 of 98 lines)
telemetry-simulator992 (SimulatorIT)99.2% (589 of 594)
telemetry-detector1009 (DetectorEndToEndIT 2, ExactlyOnceIT 1, ArenaBridgeIT 6)100% (397 of 397)
telemetry-verifier465 (TopicSnapshotReaderIT)100% (358 of 358)
Total26917286 tests, 0 failures

The build's gate is 85% line coverage per module. *Application classes (the main methods) are excluded from the count. Coverage here means "this line ran during a test", not "this line is correct". A real example of the difference: every verifier unit test passed while the verifier could not start inside Spring (see section 5d).

The measured results are in section 5g Unlike the first version of this guide, the speed and accuracy numbers are now measured: about 24,900 events per second sustained, 100% recall, 0.11% false positives, zero lost or duplicated events after killing one of three detectors, and a failover of 14.4 seconds. The failover missed its 10-second target, and that is reported as measured.

1. The story: what is this project for?

Imagine an online shooting game with thousands of players. A small number of them cheat:

The game's servers produce a constant stream of tiny facts: "player X fired a shot, it hit, it was a headshot, they reacted in 60 ms", "player X moved 0.4 units". Those facts are called events. This project is a system that watches the events as they happen and shouts "player X looks like a cheater" within a few seconds.

There is no real game here, so a program called the simulator pretends to be the game: 1,000 fake players, 5% of them secretly cheating, producing about 25,000 events per second. That is the test bench.

An everyday analogy

Think of a bank's fraud team watching card payments. Every card produces a stream of small payments. The team does not judge each payment alone. It looks at the last few minutes of each card and asks "does this pattern look like a thief?" It also numbers the receipts, so a missing or doubled receipt is noticed straight away.

Map it across: the cards are players, the payments are events, "the last few minutes" is a 5-second window, the fraud team is the detector, and the numbered receipts are the seq counters.

Simulatorfake players and cheaters, about 25,000 events a second
→
Kafkathe pipe that stores every event, in order per player
→
Detectorwatches 5-second windows, counts events, raises flags
→
Flagsa suspect list for the Judge (project 3), plus counts for the verifier

The design promises

The whole project is organised around five promises. Each has a test or a mechanism that keeps it, which is what makes it more than a slogan.

PromiseHow it is keptWhat guards it
1. The detector never peeks at the answer sheet.The simulator writes who really cheats to cheater-ground-truth. Only the verifier may read it. If the detector could, its accuracy score would be meaningless.TopologyShapeTest.neverMentionsTheGroundTruthTopic
2. Exactly-once is proven, not just claimed.Every event carries a per-player seq. The detector counts what arrived, what arrived twice and what is missing.IntegrityTopologyTest, and ExactlyOnceIT which even re-sends one event on purpose to show a duplicate would be caught. Not yet proven across a crash at full speed. That is step 12.
3. Bad data never stops the pipeline.A record that cannot be read is parked in match-events.DLT with the reason, and processing continues.DeadLetterTopologyTest.validEventsAfterBadOnesAreStillProcessedAndNeverDeadLettered
4. The contracts are frozen.Topic names, JSON field names, enum values, config keys and metric names are never renamed. The flag format is designed as the contract with the Judge.JsonSerdeTest.omitsNullFieldsAndKeepsContractFieldNames. I did not check the Judge's own code against the format while writing this guide.
5. The logic can be tested without Spring and without waiting.The detector's wiring diagram is plain Kafka Streams code. Time comes from an injected Clock, so a test can jump 11 minutes instantly.Every *TopologyTest uses TopologyFixture and MutableClock

Where this fits

Project 1: Arena

A separate web service that stores players and finished matches. Telemetry sends it match results, but this is optional and switched off by default.

Project 2: Telemetry (this one)

Generates events, detects cheaters in real time, and measures how well it did.

Project 3: Judge

A future AI agent that reads Telemetry's suspect list and decides who to ban. Telemetry only flags; it never bans.

The four programs in this repository

ModulePlain-English jobStatus
telemetry-commonA shared library. Defines what an event looks like, the names of the Kafka topics, and how to turn objects into JSON text and back.built
telemetry-simulatorPretends to be the game. Makes players, decides who cheats, produces events.built
telemetry-detectorThe star. Reads events, spots cheaters, publishes flags, and keeps count of every event so loss can be proven.built (Arena bridge not yet)
telemetry-verifierThe examiner. After a run it compares the detector's flags with the simulator's secret answer sheet and prints a grade.built

2. Ideas you need first

You can skip anything you already know. Each idea is short and used later.

An event, and JSON

An event is one small record of something that happened. Ours look like this when written as JSON, a plain-text format where data is written as "name": value pairs inside curly braces:

{ "eventId": "0b9c...", "runId": "r-9f2c41", "playerId": "3f2a...", "matchId": "sim-r-9f2c41-000017",
  "seq": 1532, "type": "SHOT", "ts": 1763372123456,
  "hit": true, "headshot": false, "reactionMs": 231 }

Two fields deserve a note. ts is the time of the event as epoch milliseconds: the number of milliseconds since 1 January 1970, which is how computers store points in time. seq is a per-player counter: a player's first event has seq 1, the next 2, then 3, with no gaps. That counter is the secret behind the "nothing was lost" proof later.

There are three kinds of event: SHOT (fired a weapon), POSITION (moved a little), and KILL (killed someone).

Java records

A record is a short way to write a Java class that only holds values. This one line:

public record PlayerProfile(
        String playerId,
        String username,
        String country,
        SkillTier skillTier,
        long registeredAt) {
}

gives you a class with those five fields, a constructor, getters (profile.username()), and correct equals/toString, all automatically. Records cannot be changed after creation, which makes them safe to pass around. The project rule is that every event, DTO (a data-carrying object) and stored state value is a record.

More Java you will meet

Beyond records (above), these Java features appear again and again. Each example is a real line from the project.

Two different things called "Record" A Java record (lower case keyword) is the short data class described above. Kafka Streams also has a class named Record<K, V> (in org.apache.kafka.streams.processor.api) which is one key-value entry flowing through a processor. When you see Record<String, MatchEvent> in a processor, it is the Kafka one.

Kafka

Apache Kafka is a very fast, durable message pipe. Think of it as a set of append-only logbooks that many programs can write to and read from independently.

WordMeaning here
TopicA named logbook, for example match-events. Producers add entries to the end; entries are kept for a while (24 hours for events) and are never edited.
Producer / consumerA program that writes to a topic / a program that reads from it. They never talk to each other directly, only through the topic.
Record, key, valueOne entry. The value is our JSON; the key is a label, here always the player's id.
PartitionA topic is split into lanes so many readers can work in parallel. Records with the same key always go to the same partition, and within a partition order is guaranteed. Our key is the player id, so each player's events stay in order. match-events has 6 partitions.
OffsetThe position number of a record within a partition (0, 1, 2, ...). A reader remembers its offset so it can resume after a crash.
BrokerThe Kafka server. We run exactly one, in Docker, in "KRaft" mode (Kafka 4 removed the old ZooKeeper helper).
Compacted topicA topic that keeps only the latest record per key, like a phone book rather than a diary. Player profiles work this way.
Dead-letter topic (DLT)A "returns desk". A record that cannot be processed is parked there with the reason, so one bad record cannot stop the pipeline.

Kafka Streams

Kafka Streams is a Java library for processing topics as they fill up: read records, transform them, count things, join with other data, and write the results to new topics. The pieces you will meet:

Exactly-once

When a program crashes halfway through a job, restarting it can lead to some events being processed twice (duplicates) or skipped (losses). Kafka Streams has a setting called exactly_once_v2 that makes "read input, update the state store, write the output, remember how far I got" happen as one atomic transaction: it either all happens or none of it does. This project does not just claim that; it checks it with the seq counters (see Step B). So far that check has run on 10,000 events with no crash; the kill-a-detector test is step 12.

Spring Boot and Maven

Testing ideas used here

Language and framework things you will meet

ThingWhat it isExample from this project
recordAn immutable data class in one linePlayerProfile, MatchEvent, AimStats
Compact constructorChecks or tidies values when a record is createdDeadLetter (caps the error text), DetectionProperties (refuses nonsense settings)
sealed interfaceA closed list of allowed implementationsParseResult is either Parsed or Failed
Switch expressionA switch that returns a valueMatchEvent.isValid(), KafkaStreamsHealthIndicator.health()
Method reference X::yShorthand for a lambda that calls one methodAimStats::empty in TelemetryTopology
@SpringBootApplicationMarks the class where the program starts and Spring scans for beansDetectorApplication
@ConfigurationProperties("telemetry.detection")Copies a block of application.yml into a typed objectDetectionProperties. @ConfigurationPropertiesScan finds it
@Configuration and @BeanA method whose return value Spring creates once and sharesClockConfig.clock(), the NewTopic beans in TopicsConfig
@ComponentA class Spring creates itselfStreamsWiring, MicrometerTopologyMetrics, KafkaStreamsHealthIndicator
@Import(TopicsConfig.class)Pulls in a configuration class that lives outside the scanned packageDetectorApplication, SimulatorApplication
@EnableKafkaStreamsTells Spring Boot to create and run a Kafka Streams clientDetectorApplication
${NAME:default} in YAMLRead an environment variable, or use the default after the colon${PROCESSING_GUARANTEE:exactly_once_v2}
10m, 5s in YAMLSpring turns these into a Java Durationcooldown: 10m
Processor<KIn,VIn,KOut,VOut>Kafka Streams interface for custom per-record codeIntegrityProcessor, CooldownProcessor, ParseProcessor
@JsonInclude(NON_NULL), @JsonIgnoreJackson annotations that shape the JSONMatchEvent (null fields are left out; isValid() is not written)
@SpringBootTestStarts the real Spring application inside a testExactlyOnceIT
@DynamicPropertySourceSets a configuration value at test timePoints spring.kafka.bootstrap-servers at the Docker broker
@DirtiesContextThrow away the Spring context after the test classBoth detector ITs, so the next one starts clean
JUnit 5 @TestMarks a test methodEvery *Test class
AssertJ assertThat(x).isEqualTo(y)Readable assertionsEvery test
Awaitility await().atMost(...)Keeps retrying a check until it passes or time runs outExactlyOnceIT waits for stats to arrive
Testcontainers KafkaContainerStarts a throwaway Kafka in Docker for a testKafkaTestContainer

3. Map of the repository

Project 2/ ├── pom.xml parent build: Spring Boot 4.1.2, JaCoCo (85% gate), failsafe (runs *IT tests) [built] ├── mvnw, mvnw.cmd, .mvn/ the Maven wrapper [built] ├── compose.yaml Docker: broker, Kafka UI, 3 detector replicas, Prometheus, Grafana; profiles sim, tools [built] ├── Dockerfile, .dockerignore one Dockerfile for all three apps (--build-arg MODULE=...) [built] ├── .env.example settings template (copy to .env, which git ignores) [built] ├── CLAUDE.md, handoff.md, todo.md notes for the next working session. handoff/todo are the live status ├── Fairplay Telemetry — Implementation Specification.md the written plan (1,310 lines) ├── Code Explanation/ you are here ├── ops/ prometheus.yml, grafana/provisioning (data source, dashboard loader), the dashboard [built] ├── scripts/ chaos-kill.sh (kill one detector, time the failover), measured-run.sh [built] ├── .github/workflows/ci.yml runs ./mvnw verify on every push [built] ├── reports/ verifier output (git-ignored) │ ├── telemetry-common/ shared library [built] │ └── src/main/java/dev/fairplay/telemetry/common/ │ ├── Topics.java, TopicsConfig.java, TopicsProperties.java │ ├── json/ JsonSerde.java, JsonSerdes.java │ └── model/ MatchEvent, PlayerProfile, SuspiciousPlayer, IntegrityStats, DeadLetter, ... (16 files) │ └── src/test/.../support/ MutableClock, TestEvents, KafkaTestContainer, StubArenaServer (shared as a test-jar) │ ├── telemetry-simulator/ the fake game [built] │ └── src/main/java/dev/fairplay/telemetry/simulator/ │ ├── SimulatorApplication.java, SimulatorProperties.java, ClockConfig.java │ ├── population/ BehaviorProfile, CheaterAssignment, PlayerState, PopulationFactory │ ├── match/ MatchRoom, MatchSummaryBuilder │ ├── engine/ EventFactory, MalformedInjector, ShardWorker, SimulationEngine │ ├── output/ EventPublisher, ManifestTracker │ └── arena/ ArenaProperties, ArenaClient, ArenaPlayerRegistrar (only when telemetry.arena.enabled) │ ├── telemetry-detector/ the cheat detector [built] │ └── src/main/java/dev/fairplay/telemetry/detector/ │ ├── DetectorApplication, DetectionProperties │ ├── ClockConfig, StreamsWiring, StreamsRuntimeConfig, MicrometerTopologyMetrics, KafkaStreamsHealthIndicator │ ├── topology/ TelemetryTopology, EventParser, ParseProcessor, ParseResult, IntegrityProcessor, │ │ AimStats, SpeedStats, DetectionRules, FlagCandidate, CooldownProcessor, TopologyMetrics │ └── bridge/ ArenaProperties, ArenaMatchClient, ArenaBridgeListener, BridgeErrorHandlingConfig, │ RetryableArenaException, NonRetryableArenaException (the optional Arena bridge) │ └── telemetry-verifier/ the examiner [built] └── VerifierApplication, VerifierProperties, VerifierRunner, TopicSnapshotReader, DetectionEvaluator, DetectionReport, IntegrityChecker, IntegrityReport, ReportWriter Every module also has src/test/java/... mirroring the main packages.

Size today: 124 Java files, about 4,100 lines of production code and 5,600 lines of tests (counted with wc -l). There are 286 automated tests: 269 unit tests and 17 integration tests.

Recommended reading order

  1. common/model/MatchEvent.java and SuspiciousPlayer.java: the two shapes everything is about.
  2. detector/topology/DetectionRules.java and its test DetectionRulesTest: the thresholds, in plain Java, with no Kafka.
  3. detector/topology/AimStats.java and SpeedStats.java: what a window remembers.
  4. detector/topology/IntegrityProcessor.java: the smallest processor, and the clever one.
  5. detector/topology/TelemetryTopology.java: the wiring diagram, once you know every piece it connects.
  6. detector/src/test/.../TopologyFixture.java then AimbotTopologyTest: how a test drives the whole thing.
  7. simulator/engine/EventFactory.java: where events come from.
  8. Last, the Spring layer in detector/*.java and the application.yml files.

4. The journey of one event

Before reading any code, here is the whole system as a picture. Purple boxes are Kafka topics; blue boxes are programs or steps.

Simulatormakes 1,000 players and plays them in matches, about 25,000 events a second
→
match-eventsevery shot, move and kill; key = player id; 6 partitions
→
Detector (Kafka Streams)parse, count, window, join, de-duplicate
Detector
→
suspicious-playersthe flags, for the Judge integrity-statsper-player counts, for the verifier match-events.DLTbad records with the reason
player-profilesusername, tier (the "phone book")
→
Detectorlooks the suspect up so the flag has a name
 
cheater-ground-truththe simulator's secret answer sheet. The detector never reads it.
→
Verifiercompares flags with the answer sheet

The nine topics, all declared in code by TopicsConfig:

TopicPartitionsKeptWritten byHolds
match-events624 hsimulatorthe raw gameplay events
match-events.DLT67 ddetectorrecords that could not be parsed, with the reason
player-profiles6compactedsimulatorlatest username and skill tier per player
suspicious-players67 ddetectorthe flags (the contract with the Judge)
integrity-stats6compacteddetectorper-player count, duplicates, gaps
match-summaries37 dsimulatorfinished matches, shaped for Arena
match-summaries.DLT37 dArena bridgematches Arena rejected
cheater-ground-truth1compactedsimulatorwho really cheats (verifier only)
simulation-manifest1compactedsimulatorhow many events were really sent per player

The numbered steps, with the class that does each one

This is the path of a single SHOT event. The tag says whether the code for that step exists today.

  1. A game tick creates the event. EventFactory.tick(...) decides whether the player fires, whether it hits, how fast they reacted, and takes the next seq from PlayerState.nextSeq(). built
  2. The loop that calls it every 50 ms (ShardWorker, four of them, started by SimulationEngine) and the publisher that sends the bytes to Kafka (EventPublisher, which also counts every acknowledged send in the ManifestTracker). built
  3. Sometimes a garbage record is sent too (MalformedInjector.maybeNext, 0.1% of the time). built The ShardWorker sends each bad record through EventPublisher.publishMalformed; it is not counted in the manifest.
  4. Kafka stores the record. Its key is the player id, so it lands in one of the 6 partitions of match-events, in order with that player's other events. built (topic declared by TopicsConfig)
  5. The detector reads the raw bytes. TelemetryTopology.addTo declares builder.stream(MATCH_EVENTS, ...ByteArray()). built
  6. Parse. ParseProcessor calls EventParser.parse. A good event moves on. A bad one becomes a DeadLetter and is written to match-events.DLT. built
  7. Count. IntegrityProcessor.process compares seq with the last one for this player and updates the seq-tracker store. Every 5 seconds emitAll writes every player's counts to integrity-stats. built
  8. Window. The same raw bytes are parsed a second time with mapValues (to avoid a hidden repartition, see the trap). Only SHOTs continue, grouped by player into 5-second windows that start every second. AimStats.add folds each shot in. built
  9. Rule check. DetectionRules.aimbotTriggered tests the window's stats. If true, FlagCandidate.aimbot packages it with a score. built
  10. Enrich. The candidate is merged with speedhack candidates, re-keyed through the named flags repartition step, and left-joined to the player-profiles table with FlagCandidate.withProfile. built
  11. De-duplicate and publish. CooldownProcessor drops repeats within 10 minutes and writes one SuspiciousPlayer to suspicious-players. built
  12. Somebody reads the flag. The Judge (project 3) reads suspicious-players. The verifier reads suspicious-players, integrity-stats, the manifest and the ground truth to grade the run (section 5d). built

5a. telemetry-common: the shared vocabulary built

Both the simulator and the detector must agree on exactly what an event looks like and what the topics are called. Putting that in one library means they cannot disagree. This module has no main method; it is just a dependency.

Topic names and creation

Topics is a list of constants, so no other file ever types "match-events" by hand:

common/Topics.java
public static final String MATCH_EVENTS = "match-events";
public static final String MATCH_EVENTS_DLT = "match-events.DLT";
public static final String PLAYER_PROFILES = "player-profiles";
public static final String SUSPICIOUS_PLAYERS = "suspicious-players";
...
public static final int PLAYER_PARTITIONS = 6;

TopicsConfig turns each row of the table above into a Spring bean of type NewTopic. When an application starts, Spring's KafkaAdmin notices these beans and creates any topic that does not exist yet, so nobody has to create topics by hand. TopicsProperties supplies the replication factor (how many broker copies; 1 for our single broker) from telemetry.topics.replication-factor.

Why the apps use @Import(TopicsConfig.class) Spring only scans for beans inside the application's own package. TopicsConfig lives in the common package, so SimulatorApplication and DetectorApplication import it explicitly.

Turning objects into JSON and back: JsonSerde

Kafka only stores bytes. A serializer turns a Java object into bytes; a deserializer does the reverse; a serde is the pair. JsonSerde<T> does this with the Jackson 3 library, for any record type:

common/json/JsonSerde.java
public static final JsonMapper MAPPER = JsonMapper.builder()
        .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
        .build();

@Override
public byte[] serialize(String topic, T value) {
    if (value == null) {
        return null;
    }
    try {
        return MAPPER.writeValueAsBytes(value);
    } catch (RuntimeException e) {
        throw new SerializationException("Cannot serialize " + type.getSimpleName(), e);
    }
}

Notes: unknown JSON fields are ignored (so a newer producer does not break an older consumer); a null value stays null (Kafka uses null as "delete this key"); any failure is wrapped in Kafka's SerializationException. JsonSerdes offers shortcuts like JsonSerdes.matchEvent() and the generic JsonSerdes.of(AimStats.class) for types that live outside this module.

The data shapes (the model package)

RecordWhat it represents
MatchEventOne gameplay event. Fields that do not apply to its type are null and left out of the JSON.
PlayerProfileplayerId, username, country, SkillTier (BRONZE to PRO), registeredAt.
SuspiciousPlayerThe flag: who, which Rule (AIMBOT or SPEEDHACK), a score 0.5 to 1.0, the time window, and Evidence. This is the contract with the Judge; its field names must never change.
EvidenceThe numbers behind a flag. Aimbot fills shots/hits/headshotHits/headshotRate/avgReactionMs; speedhack fills movingSamples/overspeedSamples/overspeedRatio/maxSpeed.
IntegrityStatsPer player: count, lastSeq, duplicates, gaps, updatedAt.
DeadLettersourceTopic, partition, offset, key, the original bytes as Base64 text, the error (capped at 500 characters), receivedAt.
GroundTruth, CheatType, SegmentThe answer sheet: is the player HONEST, PRO or a CHEATER, and which cheat from AIMBOT, AIMBOT_SUBTLE, SPEEDHACK, SPEEDHACK_SUBTLE.
ManifestEntry, RunInfoWhat the simulator really sent, for the verifier.
MatchSummary, ParticipantSummaryA finished match, shaped like Arena's API body.

MatchEvent.isValid(): what counts as a good event

common/model/MatchEvent.java
@JsonIgnore
public boolean isValid() {
    if (type == null || seq < 1 || ts <= 0 || !isUuid(playerId)) {
        return false;
    }
    if (isBlank(eventId) || isBlank(runId) || isBlank(matchId)) {
        return false;
    }
    return switch (type) {
        case SHOT     -> hit != null && headshot != null && reactionMs != null;
        case POSITION -> dx != null && dy != null && dtMs != null && dtMs > 0;
        case KILL     -> !isBlank(victimId) && headshot != null;
    };
}

The detector treats anything failing this check exactly like unreadable garbage: it goes to the dead-letter topic. @JsonIgnore stops Jackson from writing a bogus "valid" field into the JSON. The extra dtMs > 0 check is our own addition, because a movement speed is distance divided by time and dividing by zero would produce infinite speed.

Test helpers shared with other modules

Packaged as a "test-jar" so the simulator and detector tests can reuse them:

Why it is written this way, pitfalls, and what the tests prove

FileWhy this wayPitfallTests that prove it
Topics, TopicsConfigNames live in one place, so a typo cannot create a topic nobody reads. NewTopic beans mean the apps create their own topics (the broker has auto-creation switched off).Each app must @Import(TopicsConfig.class), or no topic is created.TopicsConfigTest.declaresTheNineTopicsWithPartitionsAndCleanupPolicies (also checks the 24 h and 7 d retention and the compacted ones)
JsonSerdeOur own class, because Spring's built-in one uses Jackson 2 and this project uses Jackson 3. Unknown JSON fields are ignored, so a newer producer cannot break an older reader.Jackson 3 will serialize new Object() without complaint (empty beans no longer fail), so the test for "serialization fails" needs a value with a getter that throws.JsonSerdeTest (9 tests): round trips, unknownFieldsAreIgnored, unserializableValueThrowsSerializationException, null handling
MatchEvent.isValid()One definition of "a good event", shared by the dead-letter path and the window path.UUID.fromString alone accepts odd shortened text like 1-1-1-1-1, so isUuid compares the parsed result with the input.MatchEventTest (7 tests), e.g. playerIdMustBeAUuid, positionNeedsDxDyAndPositiveDt
DeadLetterThe 500-character cap keeps a dead-letter record small even when a parser error message is long.The cap is applied in the constructor, so even a record read back from JSON is capped.ModelHelpersTest.deadLetterCapsTheErrorAt500Characters
SuspiciousPlayer, EvidenceThe flag format is the contract with the Judge, so the field names are fixed and unused evidence fields are left out of the JSON.Never rename a field. The aimbot and speedhack evidence share one record, so half of it is always null.JsonSerdeTest.omitsNullFieldsAndKeepsContractFieldNames
MutableClock, TestEvents, KafkaTestContainerShared test helpers, packaged as a test-jar so the other modules can reuse them.The Kafka image tag must match the Kafka version Boot resolves (4.2.2). Update both together.KafkaTestContainerIT proves Docker and Testcontainers work at all.

5b. telemetry-simulator: the fake game built

The simulator's job is to produce believable traffic, including known cheaters, and to remember the truth. Everything is driven by a seed, a starting number for the random-number generator: the same seed yields the same players and the same behavior every time, which makes experiments repeatable.

The engine (step 8) is described at the end of this section The parts below (settings, population, behavior, event generation, match summaries, bad-data injection) are pure logic with no Kafka. After them comes the part that runs it all: the worker threads, the publisher and the manifest.

Settings: SimulatorProperties

Loaded from application.yml under telemetry.sim: seed: 42, players: 1000, duration: 5m, match-size: 10, match-duration: 120s, tick-hz: 20 (20 game ticks per second), shots-per-second: 4.5, kill-chance-per-hit: 0.25, shard-threads: 4, cheater-ratio: 0.05, pro-ratio: 0.02, malformed-ratio: 0.001, and the window in which cheating may start (10 to 60 seconds in).

It is a record whose constructor checks the rules and refuses to start if one is broken. Example rules: ratios must be between 0 and 1; cheat-start-max must be below duration; players must divide evenly into rooms across worker threads (1000 = 10 × 4 × 25).

Who is who: CheaterAssignment

Shuffles the numbers 0 to 999 with the seed, then hands out roles in order: the first 50 become cheaters (20 AIMBOT, 10 AIMBOT_SUBTLE, 12 SPEEDHACK, 8 SPEEDHACK_SUBTLE), the next 20 become PROs, the remaining 930 are HONEST.

Why "pros"? A pro is an honest player who is genuinely excellent: high accuracy and fast reactions. A few of them will occasionally look like a subtle aimbot. That is deliberate: they are the realistic false alarms the Judge must learn to clear.

How each player behaves: BehaviorProfile

Each player gets a profile drawn once from ranges, e.g. an honest player's hit chance is a random number between 0.20 and 0.45 and typical reaction time is about 240 ms. A cheater has two profiles: the normal one, and a cheat one they switch to at a random moment. Aimbots change only the aiming numbers; speedhackers change only movement.

hit chanceheadshot chance (when hit)reaction (ms)
Honest0.20 to 0.450.25 to 0.55about 240
Pro0.40 to 0.550.55 to 0.68155 to 185
Aimbot0.60 to 0.900.75 to 0.9540 to 80
Subtle aimbot0.45 to 0.600.60 to 0.70110 to 135

A speedhacker still draws a base speed of 5 to 7 units per second, but that speed is then multiplied by 1.5 to 2.0 (roughly 7.5 to 14), and a subtle one by only 1.12 to 1.20 (roughly 5.6 to 8.4, so some samples stay under the detector's 7.5 limit). An honest player's speed is 2 to 7. A player's skill tier (BRONZE...PRO) is derived from their base headshot chance, so a cheater's tier does not give them away.

Building the population: PopulationFactory and PlayerState

For player i: the id is UUID.nameUUIDFromBytes(seed + ":" + i) (a UUID computed from text, so it is always the same for the same seed), the username is sim_0001, sim_0002..., the country comes from the Datafaker fake-data library, and the profiles and the cheat start time come from one seeded random generator. The result is a list of PlayerState records:

simulator/population/PlayerState.java
public BehaviorProfile profileAt(long nowMs) {
    return isCheatingAt(nowMs) ? cheat : base;
}
public boolean isCheatingAt(long nowMs) {
    return cheatStartsAt != null && nowMs >= cheatStartsAt;
}
public long nextSeq() {
    return seq.incrementAndGet();
}

That is the whole "cheater switches on" mechanism: before cheatStartsAt the player behaves honestly; from that millisecond on, the cheat profile applies. nextSeq() hands out 1, 2, 3, ... for each event of that player.

One tick of the game: EventFactory

Every tick (1/20 s = 50 ms), each player produces, in this order:

  1. Exactly one POSITION: with probability moveChance they move at a random speed (times the speedhack multiplier if cheating and started), in a random direction, with a little measurement noise; otherwise they stand still (dx = dy = 0).
  2. Zero or more SHOTs: the count follows a Poisson distribution with mean 4.5/20 = 0.225 per tick (about 4.5 shots a second). Each shot is a hit with probability hitChance; a hit is a headshot with probability headshotChance; the reaction time is drawn from a normal (bell-curve) distribution around the player's mean, never below a floor.
  3. At most one KILL after each hit: with probability 0.25 a random other player in the room dies.
simulator/engine/EventFactory.java (the shot part)
boolean hit = rng.nextDouble() < aim.hitChance();
boolean headshot = hit && rng.nextDouble() < aim.headshotChance();
int reactionMs = (int) Math.max(aim.reactionFloor(),
        Math.round(aim.reactionMean() + aim.reactionSd() * rng.nextGaussian()));
room.recordShot(player, hit, reactionMs);
long shotSeq = player.nextSeq();
sink.accept(new MatchEvent(eventId(player.playerId(), shotSeq), runId, player.playerId(),
        room.externalId(), shotSeq, EventType.SHOT, now, hit, headshot, reactionMs, ...));

With 1,000 players that adds up to 20,000 POSITION + 4,500 SHOT + about 400 KILL events per second, the "25,000 per second" target. All events in one tick share one timestamp from the injected clock. Event ids are computed from run, player and seq (not random), so everything stays a function of the seed.

Matches: MatchRoom and MatchSummaryBuilder

Players are grouped into rooms of 10. A MatchRoom keeps each player's running totals (kills, deaths, headshot kills, shots fired, shots hit, total reaction time). When a match ends, MatchSummaryBuilder ranks players (most kills first, then fewest deaths, then player id) and produces a MatchSummary shaped exactly like the JSON Arena accepts. It guarantees Arena's rules: placements are exactly 1..10, headshots never exceed kills, hits never exceed shots, and the average reaction is at least 1 even for a player who never fired.

Bad data on purpose: MalformedInjector

A detector that only sees perfect data is untested. With probability 0.1% per event, the injector adds a broken record: (1) truncated JSON {"broken":, (2) valid JSON missing required fields, or (3) 16 random bytes (the first forced to 0xFF so it can never accidentally be valid JSON). The detector must route these to the dead-letter topic and carry on.

Why it is written this way, pitfalls, and what the tests prove

FileWhy this wayPitfallTests that prove it
SimulatorPropertiesThe constructor checks every rule, so a bad setting stops the program at startup, not ten minutes into a run. Extra rules were added beyond the spec: match-size at least 2, 1000 % tick-hz == 0, kill-chance-per-hit above 0, cheater-ratio + pro-ratio at most 1.With the defaults, players must be a multiple of 40 (10 × 4). Changing SIM_PLAYERS to 1,010 stops the app.SimulatorPropertiesTest (8), e.g. startupFailsWhenPlayersIsNotAMultipleOfMatchSizeTimesShards
CheaterAssignmentShuffle the indices with the seed, then deal roles in order. The same seed always gives the same answer sheet. With 1,000 players: 50 cheaters (50 × 20 / 100 = 10 subtle aimbots, 12 speedhackers, 8 subtle speedhackers, and the remaining 20 plain aimbots), 20 pros, 930 honest.Percentages are rounded down, and any rounding remainder goes to plain AIMBOT.CheaterAssignmentTest.thousandPlayersGiveTheSpecCounts, sameSeedGivesTheSameAssignment, roundingRemainderGoesToAimbot
PopulationFactory, PlayerStateAll randomness comes from a seeded generator, so a run is repeatable. Player ids are derived from the seed and index, so they repeat across runs.That repetition is why docker compose down -v is needed before each measured run: old integrity counters would otherwise carry over. Also, PlayerState holds an AtomicLong, so comparing two players with equals compares the counter by identity. Tests compare a projection instead.PopulationFactoryTest (9), e.g. sameSeedGivesTheSamePopulationIncludingProfilesAndStartTimes, profileSwitchesExactlyAtCheatStart
BehaviorProfileOne table of ranges per kind of player, drawn once. A cheater's cheat profile changes only its own family (aim or movement), so an aimbot still moves like an honest player.Subtle speedhackers multiply their speed by only 1.12 to 1.20. From a base of 5 to 7 units/s that is 5.6 to 8.4, and the detector's limit is 7.5, so some of their samples are legitimately below it.BehaviorProfileTest (9), one test per profile kind plus skillTierBoundaries
EventFactoryOne tick of one player. The shot count is a Poisson draw (Knuth's algorithm), so shots arrive randomly rather than on a metronome.All events of a tick share one timestamp. Event ids are name-based UUIDs of run, player and seq, not random, to keep the simulator a function of the seed.EventFactoryTest (12), e.g. shotRateMatchesShotsPerSecond, aimbotProfileSwitchesAtCheatStart, sameSeedAndClockReproduceTheSameEvents
MatchRoom, MatchSummaryBuilderThe room is touched by exactly one worker thread, so it needs no locks. The builder guarantees every rule Arena's API enforces.PlayerTotals is a small mutable class, a deliberate exception to "everything is a record", because it is updated thousands of times per second and never leaves the thread.MatchSummaryBuilderTest (11), especially aSimulatedMatchSatisfiesEveryArenaInvariant
MalformedInjectorA detector fed only perfect data is untested.The 16 random bytes could by chance form valid JSON, so the first byte is forced to 0xFF, which can never begin valid text.MalformedInjectorTest.noPayloadParsesIntoAValidEvent, allThreeKindsAreProducedInRotation

Running it for real: the engine built

ClassWhat it doesWhy this way
EventPublisherThe only class that talks to Kafka. publish(event) turns the event into JSON bytes and sends it to match-events with the player id as the key and the event time as the record timestamp. When Kafka acknowledges, it tells the manifest; when a send fails, it counts a failure and logs it. It never retries by hand.The producer is idempotent, so Kafka already retries safely. Metrics objects are created once in the constructor, because creating one per event at 25,000 a second would cost real time.
ManifestTrackerPer player: how many sends were acknowledged, how many failed, and the highest acknowledged seq. At the end it becomes one ManifestEntry per player.Every counter is a LongAdder/AtomicLong, because Kafka's I/O thread updates them while four worker threads keep sending. finalSeq is the last seq generated, not the last acknowledged, so a failed send shows up as finalSeq > acked.
ShardWorkerOne of 4 worker threads. It owns a fixed set of rooms (so no data is shared with other threads), has its own SplittableRandom(seed + k) and its own MalformedInjector, and every 50 ms plays one tick for each of its players. When a match is over it publishes the summaries and reshuffles its players into new rooms.A task given to scheduleAtFixedRate that throws is silently never run again, which would freeze a quarter of the players with no visible error. So run() catches everything, logs it and lets the next tick happen. All rooms of one shard start and end together, which keeps the reshuffle trivial.
SimulationEngineStarts when the app is ready, on a thread named simulation-main so the app and its actuator stay responsive. It builds the population, publishes one profile and one ground-truth record per player, starts the workers, logs events per second every 10 s, and after duration stops the workers and flushes. Then it publishes the manifest and RunInfo (key __run__) and logs SIMULATION COMPLETE. The app keeps running afterwards so Prometheus can still scrape it.Each shard gets its own single-thread executor, so a shard never moves between threads. telemetry.sim.autostart=false switches the automatic start off for tests. With Arena enabled it registers the players first (section 5e).

Measured on a laptop: about 24,900 events per second for the whole 5 minutes, 7,466,329 events acknowledged, 0 failed sends and 7,503 malformed records (0.1%).

Test classWhat it proves
ManifestTrackerTest (4)Counts per player, highest acknowledged seq ignoring out-of-order acks, and entries using the generated seq
EventPublisherTest (10)The record's topic, key and timestamp; acknowledged, failed and thrown sends all counted correctly; malformed records counted but kept out of the manifest; every other topic and key
ShardWorkerTest (6)Rooms dealt round-robin; gapless seqs; a finished match is published and replaced; bad records follow the ratio; a failing tick does not escape
SimulationEngineTest (4)A failed setup send fails the run instead of hanging it; stopping interrupts a run; with Arena enabled every record uses Arena's ids
SimulatorIT (2, real Kafka)40 players for 6 seconds: every player's events arrive as seq 1..N in order, the manifest agrees (acked == finalSeq, 0 failed), 40 profiles and 40 ground-truth records, every shard finished a match, and the dashboard metrics exist

5c. telemetry-detector: the brain built

This is the centerpiece, and the part worth understanding deeply. It is split in two on purpose:

The topology package

Pure Kafka Streams logic. No Spring. It receives plain objects (settings, a clock, a metrics counter) and builds the wiring diagram. Because it needs no Spring, tests can run it in memory in milliseconds.

The Spring layer

A thin outer shell that gives the topology its real clock, real settings and real Kafka connection, plus health checks and metrics.

Settings: DetectionProperties

detector/src/main/resources/application.yml (the interesting part)
telemetry:
  detection:
    detector-version: "1.0.0"
    cooldown: 10m
    integrity-emit-interval: 5s
    aimbot:
      window-size: 5s
      advance: 1s
      grace: 2s
      min-hits: 12
      max-avg-reaction-ms: 140
      min-headshot-rate: 0.62
    speedhack:
      window-size: 5s
      advance: 1s
      grace: 2s
      min-moving-samples: 40
      moving-speed-floor: 0.5
      speed-limit: 7.5
      min-overspeed-ratio: 0.15

The same file configures Kafka Streams itself: processing.guarantee: exactly_once_v2, two stream threads, one standby replica, a 500 ms commit interval, and session.timeout.ms: 6000. That last one matters for later: if a detector instance is killed, the cluster only notices after the session timeout, so the default of 45 seconds would make a "recover in under 10 seconds" goal impossible.

The topology, one screen

TelemetryTopology.addTo(builder) in four moves. Read this once, then the sub-sections explain each move.

match-eventsraw bytes
→
A. ParseParseProcessor
good → continue
bad → match-events.DLT
→
B. IntegrityIntegrityProcessor
→ integrity-stats
match-eventsraw bytes (parsed again)
→
C. Windowsaimbot (SHOT) and speedhack (POSITION), 5 s hopping, rules check
→
D. Flagsmerge → repartition "flags" → join profiles → cooldown → suspicious-players

Step A: parse, and send bad records to the dead-letter topic

The detector reads match-events as raw bytes, not as ready-made objects. Why? A normal deserializer that hits garbage throws an exception inside Kafka's own plumbing, where handling it is awkward. By parsing inside our own code, a bad record is just another branch of the diagram.

EventParser is the single definition of a good record:

detector/topology/EventParser.java
static Outcome parse(String key, byte[] bytes) {
    if (bytes == null) {
        return new Outcome(null, "null payload");
    }
    try {
        MatchEvent event = JsonSerde.MAPPER.readValue(bytes, MatchEvent.class);
        if (event == null) {
            return new Outcome(null, "payload is JSON null");
        }
        if (!event.isValid()) {
            return new Outcome(null, "invalid event: a required field is missing or malformed");
        }
        if (!event.playerId().equals(key)) {
            return new Outcome(null, "key mismatch: record key '" + key + "' is not playerId '"
                    + event.playerId() + "'");
        }
        return new Outcome(event, null);
    } catch (RuntimeException e) {
        return new Outcome(null, e.getMessage() != null ? e.getMessage() : e.getClass().getSimpleName());
    }
}

Four things make a record bad: no payload, unreadable JSON, a failed isValid(), or a record key that is not the event's own playerId (which would break the "same key, same partition" promise). The key check is important: if a wrong-key event sneaked through, a player's events could land in different partitions and arrive out of order.

ParseProcessor wraps this in Kafka Streams' Processor interface. For a bad record it builds a DeadLetter using context.recordMetadata() (which tells it the topic, partition and offset of the record it is looking at) and forwards a ParseResult.Failed; for a good one it forwards ParseResult.Parsed. ParseResult is a Java sealed interface with exactly those two record types, so the compiler knows there are no other possibilities. Then the topology splits the stream in two:

Map<String, KStream<String, ParseResult>> outcome = parsed.split(Named.as("parse-"))
        .branch((key, result) -> result instanceof ParseResult.Failed, Branched.as("failed"))
        .defaultBranch(Branched.as("ok"));

outcome.get("parse-failed")
        .mapValues(result -> ((ParseResult.Failed) result).deadLetter())
        .to(Topics.MATCH_EVENTS_DLT, Produced.with(Serdes.String(), JsonSerdes.deadLetter()));

Operators can open Kafka UI and see exactly what failed and why, with the original bytes preserved as Base64 text.

Step B: integrity tracking, the exactly-once check

This is the clever bit. Because every player's events carry seq numbers 1, 2, 3 with no gaps, the detector can audit itself. For each player it remembers the last seq seen and applies three rules (the real code in IntegrityProcessor.apply):

SituationMeaningUpdate
seq == lastSeq + 1the next one, in ordercount + 1, lastSeq = seq
seq <= lastSeqseen before: a duplicateduplicates + 1
seq > lastSeq + 1something is missing: a gapgaps + (seq - lastSeq - 1), count + 1, lastSeq = seq

If the system is truly exactly-once, then for every player at the end: count equals the number of events sent, duplicates is 0, gaps is 0. The verifier (step 9) will compare against what the simulator says it sent.

The counts live in a state store named seq-tracker. A timer (context.schedule(interval, WALL_CLOCK_TIME, ...)) fires every 5 seconds and re-emits every player's stats to integrity-stats, not just those that changed. This is deliberate: after a failover the new machine has no memory of what changed, and the reader only needs the latest value per player, so repeating is safe and simple.

Step C: the windows, where cheaters are caught

For aimbots, the topology takes only SHOT events, groups them by player, and slices time into 5-second windows that start every second. For each (player, window) it keeps an AimStats value:

detector/topology/AimStats.java
public record AimStats(int shots, int hits, int headshotHits, long sumReactionMs) {
    ...
    public AimStats add(MatchEvent shot) {
        return new AimStats(
                shots + 1,
                hits + (Boolean.TRUE.equals(shot.hit()) ? 1 : 0),
                headshotHits + (Boolean.TRUE.equals(shot.headshot()) ? 1 : 0),
                sumReactionMs + (shot.reactionMs() == null ? 0 : shot.reactionMs()));
    }
    ...
    public double avgReactionMs() {
        return shots == 0 ? 0.0 : (double) sumReactionMs / shots;
    }
    ...
    public double headshotRate() {
        return hits == 0 ? 0.0 : (double) headshotHits / hits;
    }
}

Each new shot produces a new AimStats (records are immutable) which Kafka Streams stores. Speedhack detection is the same shape using POSITION events and SpeedStats, where speed = sqrt(dx² + dy²) / (dtMs / 1000); a sample counts as moving if speed is at least 0.5, and as overspeed if it is moving and above 7.5. Idle samples count toward nothing, so a player who stands still for most of the window is not diluted into looking innocent.

The rules, in DetectionRules

RuleFires when ALL are true (boundaries inclusive)Score
AIMBOThits ≥ 12, average reaction ≤ 140 ms, headshot rate ≥ 0.620.5 + 0.25·c((rate-0.62)/(0.95-0.62)) + 0.25·c((140-avg)/(140-30))
SPEEDHACKmoving samples ≥ 40, overspeed ratio ≥ 0.150.5 + 0.5·c((ratio-0.15)/(1-0.15))

c(x) means "clamp to between 0 and 1". So a window that only just crosses the line scores 0.5, and one far beyond scores 1.0. Scores are rounded to 3 decimals and rates to 4, using round-half-up. "Inclusive" matters: exactly 12 hits triggers, 11 does not; exactly 140 ms triggers, 141 does not. These edge cases are individually tested.

detector/topology/TelemetryTopology.java (aimbot branch)
KStream<String, FlagCandidate> aimFlags = windowEvents
        .filter((key, event) -> event.type() == EventType.SHOT)
        .groupByKey(Grouped.with(Serdes.String(), JsonSerdes.matchEvent()))
        .windowedBy(TimeWindows.ofSizeAndGrace(aim.windowSize(), aim.grace()).advanceBy(aim.advance()))
        .aggregate(AimStats::empty, (key, event, stats) -> stats.add(event),
                Materialized.<String, AimStats, WindowStore<Bytes, byte[]>>as(AIMBOT_WINDOWS)
                        .withKeySerde(Serdes.String())
                        .withValueSerde(JsonSerdes.of(AimStats.class)))
        .toStream()
        .filter((windowedKey, stats) -> stats != null && rules.aimbotTriggered(stats))
        .map((windowedKey, stats) -> KeyValue.pair(windowedKey.key(),
                FlagCandidate.aimbot(windowedKey.key(), windowedKey.window(), stats, rules)));

Read it top to bottom: keep only shots, group by player, cut into hopping windows (grace of 2 s means events arriving up to 2 s late are still counted), fold every shot into the running AimStats, turn the updates back into a stream, keep those that trip the rule, and wrap them as FlagCandidates.

Why 5-second windows advancing every second? A flag needs enough evidence (12 hits). An aimbot lands about 3 hits per second, so it needs roughly 3.5 s of cheating. Because a new window starts every second, within one second of the cheat starting there is a window containing only cheating shots. A single 10-second window would mix honest and cheating shots and respond more slowly.

Step D: join with the profile, then de-duplicate

  1. Merge the aimbot and speedhack candidates into one stream.
  2. Repartition it (write to and re-read from an internal topic that Kafka Streams names <application-id>-flags-repartition, because the step is named flags). Changing the key from a "windowed key" back to a plain player id counts as a key change, and Kafka Streams needs the data re-sorted into partitions before it can join with another topic. Flags are rare, so this costs almost nothing.
  3. Left join with the player-profiles table: attach username and skillTier. "Left" means: if there is no profile, still emit the flag, just with those two fields empty.
  4. Cooldown: CooldownProcessor removes repeats.

Why repeats exist: a hopping window re-evaluates on every new event, and five windows overlap each event. An aimbot would be flagged dozens of times a minute. The cooldown keeps a state store flag-cooldown mapping playerId|rule to the time of the last published flag. A new candidate within 10 minutes of the last is dropped; otherwise it is stamped with a new flagId (random UUID) and detectedAt and published to suspicious-players.

detector/topology/CooldownProcessor.java
String storeKey = candidate.playerId() + "|" + candidate.rule();
long now = clock.millis();
Long lastEmitted = store.get(storeKey);
if (lastEmitted != null && now - lastEmitted < cooldown.toMillis()) {
    return;                       // seen recently: drop it
}
store.put(storeKey, now);
metrics.flagged(candidate.rule());
...
context.forward(record.withValue(flag));

Because the key includes the rule, an aimbot flag never hides a speedhack flag for the same player.

The Spring layer (step 7)

ClassWhat it does
DetectorApplicationThe entry point. @EnableKafkaStreams tells Spring Boot to create and run a Kafka Streams client from the spring.kafka.streams.* settings. @ConfigurationPropertiesScan loads DetectionProperties.
ClockConfigProvides the real Clock (UTC system time) as a bean.
StreamsWiringThree lines: when Spring creates it, it receives the StreamsBuilder, the settings, the clock and the metrics, and calls new TelemetryTopology(...).addTo(builder). This is the only point where Spring meets the topology.
MicrometerTopologyMetricsImplements TopologyMetrics with counters: telemetry.detect.events.parsed, ...dead_lettered, and telemetry.detect.flags tagged by rule. These feed Prometheus/Grafana later. They can over-count after a crash-retry, so they are for dashboards only; exact numbers come from the verifier.
KafkaStreamsHealthIndicatorFeeds the /actuator/health page: UP when the client is RUNNING, OUT_OF_SERVICE while CREATED or REBALANCING (or not yet created), DOWN otherwise.
StreamsRuntimeConfigIf a stream thread crashes, replace it instead of killing the instance; and log every state change ("Kafka Streams state transition from REBALANCING to RUNNING"). The log lines are a second way to time a failover.

The "repartition trap" (a deliberate departure from the written spec)

The spec's sample code parses with a process(...) step and then groups that result into windows. When we printed the finished topology to check it, it contained two extra internal topics: aimbot-windows-repartition and speedhack-windows-repartition. The reason: Kafka Streams cannot tell whether a custom processor changed the key, so after one it plays safe and re-writes every record through a new topic before grouping. That would push every shot and movement event (about 24,000 per second) through the broker a second time.

The fix, in TelemetryTopology: the window branches do not use the process output. They re-parse the raw stream with mapValues, which promises "I do not change keys", using the very same EventParser so both paths accept exactly the same records:

KStream<String, MatchEvent> windowEvents = raw
        .mapValues((key, bytes) -> EventParser.parse(key, bytes).event())
        .filter((key, event) -> event != null);

The price is parsing each event twice (cheap). Two tests guard it: TopologyShapeTest fails if a *-windows-repartition topic returns, and DetectorEndToEndIT checks the real broker's topic list.

Pitfalls, and what the tests prove

TopicThe catchGuarded by
The cooldown uses the wall clock"Now" in CooldownProcessor is the injected Clock, not the event's own time. If you replay yesterday's events, the cooldown counts real minutes, not game minutes. This follows the spec on purpose.AimbotTopologyTest.aSecondBurstInsideTheCooldownDoesNotFlagAgain (30 s later: silent) and aBurstAfterTheCooldownFlagsAgain (11 min later: flags)
Flags are emitted on every window updateA window re-fires on each new event once it crosses the threshold, and five windows overlap each event. Without the cooldown one aimbot would produce dozens of flags a minute.The cooldown tests above; DetectorEndToEndIT asserts exactly one flag for the cheater on a real broker
One cooldown per player and ruleThe store key is playerId|rule, so an aimbot flag never hides a speedhack flag.SpeedhackTopologyTest.anAimbotFlagDoesNotHideASpeedhackFlag, AimbotTopologyTest.theCooldownIsPerPlayer
flagId is a random UUIDThat is fine in the detector (the "everything is seeded" rule is about the simulator). Under exactly_once_v2 a flag written inside a transaction that later aborts is never visible to a read_committed reader, so readers of suspicious-players must use that setting or they could see a flag that was rolled back.Nothing tests the abort case directly. The rule is written down in CLAUDE.md.
Integrity treats "late" as "duplicate"By the three rules in Step B, a seq at or below the last seen one counts as a duplicate. It relies on Kafka keeping one player's records in order inside a partition, which the key guarantees.IntegrityTopologyTest.applyCoversEveryCase
Metrics can over-countA retried transaction can count an event twice, so the Micrometer counters are for dashboards only. Exact numbers come from the verifier.Stated in the class comment of MicrometerTopologyMetrics
Every event is parsed twiceThe price of the repartition fix. It is cheap CPU work, but it is not zero. I have not measured it.DeadLetterTopologyTest and AimbotTopologyTest.invalidEventsAreDeadLetteredAndNeverCountedInTheWindows prove both parse paths accept exactly the same records
The detector's own endpoints are openapplication.yml exposes health,info,metrics,prometheus and the module has no security dependency, so anyone who can reach port 8082 can read them. Fine on a laptop, a gap anywhere else. The broker and Kafka UI are also unauthenticated and unencrypted.Known gap, not tested
Integration tests do not check exact evidence numbersThe production cache (64 MB) emits window results at commit time, so the first flag can come from a different window than in the in-memory test. The ITs assert who is flagged and how often.DetectorEndToEndIT filters everything by its own random player ids

5d. telemetry-verifier: the examiner built

The verifier is a one-shot command-line program, not a server. You start it, it reads some topics, writes a Markdown report into reports/, prints it, and exits with a number that scripts and CI can check. It has two modes, and it exists so that claims like "caught 98% of cheaters" and "lost no events" come from a measurement, not from hope.

cheater-ground-truththe answer sheet
+
suspicious-playersthe detector's flags
→
DetectionEvaluatorpure math: recall, false positives, latency
→
detection-<runId>.md
simulation-manifestwhat the simulator really sent
+
integrity-statswhat the detector counted
→
IntegrityCheckerpure comparison, per player
→
integrity-<runId>.md

The classes

ClassWhat it doesWorth knowing
TopicSnapshotReaderReads a whole topic as it stands now. A plain consumer with no group id, read_committed, no auto-commit. It writes down each partition's end offset once, then polls until it has reached them (or read-timeout passes). readLatest keeps the last value per key and a null value (a tombstone) removes the key; readAll keeps every record.read_committed matters: the detector writes in Kafka transactions, and a record of an aborted transaction must never be counted. TopicSnapshotReaderIT proves it with a real aborted transaction.
DetectionEvaluatorTakes the run's RunInfo, the ground truth and every flag, and produces a DetectionReport. A cheater is detected when at least one flag arrives at or after the moment their cheat started; the latency is the earliest such flag minus that moment. A honest or pro player with any flag is a false positive.Flags before the run started, and flags for players not in this run's ground truth, are ignored. Percentiles use "nearest rank" (sort, take index ceil(p/100 × n) - 1). Any ratio with a zero denominator is 0.0, never NaN.
IntegrityCheckerFor each player in the manifest, compares it with that player's IntegrityStats. A player passes only if stats exist, the simulator reported no failed sends, and duplicates = 0, gaps = 0, lastSeq = finalSeq and count = finalSeq.It lists at most 20 failing players (with every reason) but counts them all.
ReportWriterRenders a report as Markdown and writes reports/detection-<runId>.md or integrity-<runId>.md. The first lines are the run id, start and end times and the mode.Plain string building, so a report is easy to diff and paste into the README.
VerifierRunnerParses --mode, --min-recall, --max-fpr and --settle, runs the mode, and remembers the exit code. It is both a CommandLineRunner and an ExitCodeGenerator, so VerifierApplication.main can end with System.exit(SpringApplication.exit(context)).Exit codes: 0 fine, 1 integrity failed, 2 a threshold was missed, 3 the simulation has not finished (no __run__ record), 4 bad arguments. Code 4 is our own addition.
VerifierPropertiesreports-dir, read-timeout and settle-poll-interval, validated in the constructor like the other property records.settle-poll-interval is not in the written spec; it lets tests poll in milliseconds instead of 5 seconds.

Why --settle exists

The detector publishes integrity totals every 5 seconds. When a run has just ended, the last totals may not have arrived yet, so a check at that instant could fail for no real reason. With --settle=60s the verifier re-reads integrity-stats every 5 seconds until every player passes or 60 seconds have gone by, then reports the last attempt.

A bug that only a real run found VerifierRunner has two constructors: a public one for Spring and a package-private one that lets tests inject a clock. Spring cannot choose between two constructors, so the first run in Docker died with "No default constructor found", although every unit test was green (they built the class by hand). The fix was @Autowired on the public one, and VerifierContextTest now starts the whole application context so that kind of wiring mistake fails in mvn verify.

What the tests prove

Test classWhat it proves
DetectionEvaluatorTest (13)A hand-built scenario of 7 players: recall 0.75, false-positive rate, precision, a wrong first rule not counted as correct, an early flag reported but not counted as detection, a missed cheater listed by username, flags before the run ignored, stale ground truth of another run ignored, nearest-rank percentiles for n = 1, 2 and 10.
IntegrityCheckerTest (10)The pass case and each failure reason on its own (duplicates, gaps, wrong last seq, wrong count, failed sends, missing stats), several reasons at once, the 20-failure cap, and an empty manifest not counting as a pass.
ReportWriterTest (7)The Markdown of both reports, the file name and directory creation, and an unwritable directory being an error.
VerifierRunnerTest (12)Every exit code, thresholds met and missed, a settle loop that needs three reads, a settle that gives up, and an interrupt during settling.
VerifierContextTest (1)The whole application starts the way the jar starts it.
TopicSnapshotReaderIT (5, real Kafka)Latest value per key, tombstones, typed reads, an empty topic returning at once, and an aborted transaction staying invisible.

5e. The Arena integration built

Arena is Project 1, the game platform. With Arena enabled, the fake players become real accounts in Arena, and every finished match is recorded there. It is off by default (telemetry.arena.enabled=false): Telemetry runs fully on its own without it. It has two halves, one in each app.

In the simulator: registering the players

ClassWhat it does
arena/ArenaPropertiesThe connection settings. Its constructor refuses to start the app when Arena is enabled and service-password is blank. The password comes from TELEMETRY_SERVICE_PASSWORD, never from code.
arena/ArenaClientUses Spring's RestClient. It logs in as telemetry-sim lazily and caches the token (logging in again once on any 401). register(player) posts to /api/v1/auth/register; on 409 USERNAME_TAKEN it looks the player up with GET /api/v1/players?q=... and takes the exact, case-insensitive username match.
arena/ArenaPlayerRegistrarRegisters all players on virtual threads (cheap threads that are fine to block), limited by a semaphore of 16, because Arena hashes passwords slowly (bcrypt). It saves {"sim_0001": "<arena uuid>", ...} to sim-players.json, and a later run skips players already in that file. If any player cannot be registered the run does not start; the ones that worked are still saved.
SimulationEngine (changed)With a registrar present it asks Arena for ids before the run clock starts, then builds the real population and swaps each player's made-up id for Arena's with PlayerState.withPlayerId. So profiles, ground truth, events and match summaries all carry Arena's ids.
Why register before the clock starts Cheat start times are offsets from runStartedAt (10 to 60 seconds in). Registering 1,000 players can take a minute, so starting the clock first would let cheats begin before any event was sent. The engine therefore builds a throwaway population only to learn the usernames (they do not depend on the clock), registers, starts the clock, and builds the real one.

In the detector: the bridge

match-summaries3 partitions
→
ArenaBridgeListenerreads a String, parses it
→
ArenaMatchClientPOST /api/v1/matches
→
Arena
DefaultErrorHandlerretry with backoff, or give up
→
match-summaries.DLTsame partition number, original JSON plus Spring's exception headers
Arena answersThe client doesThen
201counts telemetry.bridge.matches.posted{outcome=created}done
200counts {outcome=replayed}: the match was already stored, this was a retrydone
401logs in again and retries once inside the same calla second 401 is non-retryable
429, any 5xx, connection refused, timeoutthrows RetryableArenaExceptionretried with exponential backoff (1 s, 2 s, 4 s), then dead-lettered
400, 404, 409, 422throws NonRetryableArenaException carrying Arena's code and detaildead-lettered at once

Delivery is at-least-once (a record can be sent twice if the app dies at the wrong moment), but Arena treats a match's externalId as idempotent (a repeat returns 200 and changes nothing), so the end result is exactly-once. That is the same trick, "make the receiver tolerate repeats", used for the detector's integrity proof, only here it is Arena's job.

Expected dead letters Once the Judge (Project 3) bans a cheater, Arena answers 422 PLAYER_NOT_ELIGIBLE for any match that includes them, and that summary lands in match-summaries.DLT. That is correct behavior, not a bug.
A small trap in the written spec @KafkaListener(id = "arena-bridge") makes the listener id the consumer group id too, which would silently override the group-id: telemetry-arena-bridge in the YAML. The listener sets idIsGroup = false so the YAML value applies.

What the tests prove

Test classWhat it proves
StubArenaServer (in common's test-jar)Not a test: a fake Arena built on the JDK's own HttpServer, answering from scripted queues of responses and recording every request. No extra dependency.
ArenaClientTest (6), ArenaPlayerRegistrarTest (5), ArenaPropertiesTest (3)Registration body and id, the 409 lookup with an exact match, token caching, one re-login on a 401, partial saves on failure, rerun skipping mapped players, a blank password failing startup.
ArenaMatchClientTest (12)Every row of the status table above, a refused connection being retryable, a login rejected versus a login that fails with 503.
ArenaBridgeIT (6, real Kafka plus the stub)201: one call, no dead letter. 503, 503, 201: three calls, no dead letter. 503 forever: exactly three calls, then the dead-letter topic with Spring's headers. 422: one call, then the dead-letter topic. Garbage JSON: zero calls, dead-lettered. 401 then 201: a fresh token is used and the match still arrives.

Real Arena: the client's request and response shapes were checked against Arena's source code in ../Project 1/arena (register, login, /players?q=, /matches, the problem JSON with code and detail). A live end-to-end run against a running Arena (which needs its Postgres and Redis) is described in the README but was not part of the measured runs.

5f. Delivery: Docker, Prometheus, Grafana, CI, scripts built

FileWhat it is
DockerfileOne file for all three apps: docker build --build-arg MODULE=telemetry-detector .. A build stage compiles the module with the Maven wrapper (a cache mount keeps downloaded dependencies between builds), and a slim Java 21 runtime stage runs the jar as a non-root user with -XX:MaxRAMPercentage=75.
.dockerignoreKeeps target/, reports, data, .git and .env out of the build context (so the password file is never copied into an image).
compose.yamlThe whole stack. Default: Kafka, Kafka UI, 3 detector replicas (deploy.replicas: 3), Prometheus and Grafana. Behind profiles: sim (the simulator) and tools (the verifier), so each run starts on purpose. The detectors, simulator and verifier are capped at 1 GB each so the stack fits in Docker Desktop's VM on a 16 GB laptop.
ops/prometheus.ymlScrapes the simulator by name and finds the 3 detector replicas by DNS: Compose resolves the service name telemetry-detector to one address per replica.
ops/grafana/...A provisioned Prometheus data source (uid prometheus) and the "Fairplay Telemetry" dashboard with eight panels: events produced/sec, events processed/sec, stream process rate by instance, consumer lag, flags per minute by rule, dead letters per minute, cheaters active and bridge outcomes. The Kafka client metric names in it were read from a live /actuator/prometheus (kafka_stream_thread_process_rate, kafka_consumer_fetch_manager_records_lag_max), not guessed.
.github/workflows/ci.ymlRuns ./mvnw -B verify on Ubuntu (which has Docker, so the Testcontainers tests run unchanged) and uploads the JaCoCo reports.
scripts/chaos-kill.shKills one detector replica with docker kill, waits until both survivors log Kafka Streams' own "State transition from REBALANCING to RUNNING", prints the elapsed milliseconds, and shows the consumer group so you can check that all 6 partitions have an owner.
scripts/measured-run.shOne whole measured run: fresh stack, the simulation, a chaos kill about 90 s in, then both verifier modes. Used for both the exactly-once and the at-least-once runs.

Things the real run taught us

5g. The measured results measured

These come from real runs on one laptop (AMD Ryzen 5 3600, 16 GB RAM, Docker Desktop with a 12-CPU, 7.7 GB VM). 1,000 players, 5 minutes, seed 42, three detector replicas, one killed with docker kill about 90 seconds in. Each configuration was run once.

ClaimTargetMeasured (exactly-once run)Met?
Events per second25,000about 24,900 produced, steady for the whole 5 minutes (24,556 to 25,204 in every 10-second window); 7,466,329 events, 0 failed sendsyes
Partitions of match-events66yes
Median detection latency, obvious cheatersunder 5 saimbot 3.9 s, speedhack 0.9 syes
Recallat least 98%100% (50 of 50)yes
False-positive rateunder 1%0.11% (1 PRO player of 950 honest and pro)yes
Lost or duplicated events after killing 1 of 3 detectors0 and 00 and 0: 7,466,329 expected, 7,466,329 counted, all 1,000 players passyes
Failoverunder 10 s14.4 sno
Consumer lag within about 12,500 and flatflat15,000 to 20,000 before the kill, a peak near 380,000 after it, drained in about 2 minutesno
Test coverageat least 85%100%, 99.2%, 100%, 100% (common, simulator, detector, verifier)yes

Exactly-once against at-least-once

Same seed, same kill about 90 seconds in, only the processing guarantee changed. The third run also disabled the state-store cache, which is the condition under which the specification expects an at-least-once detector to show duplicates.

RunGuaranteeEvents expected = countedDuplicatesGapsFailoverIntegrity exit code
r-49fe8fexactly_once_v27,466,3290014.4 s0
r-df576cat_least_once7,465,7950016.7 s0
r-8bfb5eat_least_once, cache disabled7,467,5290014.5 s0
The comparison did not show what the plan predicted The plan said an at-least-once detector would count duplicates after a kill, because the state store can be ahead of the committed offsets when an instance dies. Neither at-least-once run showed any: 0 duplicates and 0 gaps, and the integrity check passed. The detectors really did run at_least_once (checked in their logged configuration). I did not find the reason. A likely candidate is that a duplicate needs the instance to die in the short window between a state-store write reaching the changelog and the offset commit, and that window was never hit; I did not verify that. The honest conclusion is narrow: these runs prove the exactly-once run lost and duplicated nothing, and ExactlyOnceIT proves the checker does catch a duplicate when one is injected, but they do not show exactly-once beating at-least-once. One more thing came out of the third run: with the cache off, the detector's median flag time rose from about 3 seconds to about 30 seconds, because writing every update to the changelog made it fall behind.

Detection report of the exactly-once run

Cheat typeCheatersDetectedp50p95max
AIMBOT20203.9 s7.6 s10.9 s
AIMBOT_SUBTLE10106.6 s21.4 s21.4 s
SPEEDHACK12120.9 s2.5 s2.5 s
SPEEDHACK_SUBTLE882.0 s5.7 s5.7 s
All50503.0 s17.6 s21.4 s

All 50 first flags used the correct rule and none came before the cheat started. The single false positive was sim_0815, a PRO player, flagged by the aimbot rule with score 0.538 (12 hits, headshot rate 0.667, reaction 139 ms). That is the realistic borderline case the Judge is meant to clear: 1 of 20 pros (5%) against 0 of 930 ordinary players.

The failover, from the survivors' logs

  1. +0 s: docker kill on one of three detectors.
  2. about +5 s: the group coordinator notices (the 6-second session timeout) and the first rebalance starts.
  3. +5 to +14 s: two more rebalances follow back to back, because Kafka Streams moves tasks in steps, each step requesting another rebalance.
  4. +14.4 s: both survivors log State transition from REBALANCING to RUNNING. All 6 match-events partitions are owned by the two survivors.
  5. afterwards: a backlog of about 14 seconds of input (about 380,000 events) drains over roughly 2 minutes, because two instances have only about 5,000 events a second of headroom over the 25,000 a second arriving.

6. What is left, and what was learned all 12 steps done

Everything in the original plan now exists. This section lists what is honestly not finished, and the lessons the real runs taught.

Not done

ItemWhyWhat it would take
A live run against a real ArenaThe bridge and the registrar were tested against StubArenaServer, and their request shapes were checked against Arena's source code. Arena needs Postgres and Redis, which were not started.Start Project 1, set ARENA_ENABLED=true, ARENA_BASE_URL and TELEMETRY_SERVICE_PASSWORD, run the simulator, then check GET /api/v1/players/{id}/matches
Failover under 10 secondsMeasured 14.4 s (exactly-once run) and 16.7 s (at-least-once run). About 5 s is the session timeout; the rest is three rebalances in a row as Kafka Streams moves tasks in steps.Tune max.warmup.replicas, acceptable.recovery.lag and the heartbeat, or try Kafka 4.2's new Streams rebalance protocol; re-measure with scripts/measured-run.sh
Lag flat after a failoverTwo surviving instances have only about 5,000 events/s of headroom over the 25,000/s input, so a 14-second outage left a backlog of about 380,000 events that took about 2 minutes to drain.More stream threads or more CPU per detector
CI on GitHubThe workflow file exists and runs the same ./mvnw verify that passes locally, but it has not run on GitHub yet.Push and look at the Actions tab
Avro and Schema RegistryAn optional stretch in the spec.A schema-registry service and an .avsc for MatchEvent

Targets, now measured

Target (spec section 15)Result
25,000 events per second sustainedabout 24,900 produced and about 25,000 processed, steady for the 5 minutes (see section 5g)
Consumer lag flat and within about 12,500missed: 15,000 to 20,000 before the kill, a peak of about 380,000 after it
Median detection latency under 5 seconds for obvious cheatersmet: aimbot 3.9 s, speedhack 0.9 s
Recall at least 98% and false-positive rate under 1%met: 100% and 0.11%
Zero lost and zero duplicated events after killing 1 of 3 detectorsmet: 7,466,329 expected and counted, 0 duplicates, 0 gaps
Failover under 10 secondsmissed: 14.4 s

Try it (Docker Desktop must be running)

# from the project folder
./mvnw verify                                   # build everything and run every test (about 4 to 5 minutes)

cp .env.example .env
docker compose up -d --build                    # broker, Kafka UI, 3 detectors, Prometheus, Grafana
docker compose --profile sim up -d telemetry-simulator    # the 5-minute run
docker compose --profile tools run --rm telemetry-verifier --mode=detection
docker compose --profile tools run --rm telemetry-verifier --mode=integrity --settle=60s
bash scripts/chaos-kill.sh                      # about 90 s into a run
docker compose --profile sim --profile tools down -v    # reset before the next run

Kafka UI is on localhost:8090, Grafana on localhost:3000 (admin/admin) and Prometheus on localhost:9090. On Windows PowerShell use .\mvnw.cmd instead of ./mvnw.

7. The tests: how we know it works

There are three layers, from fast and small to slow and realistic:

1. Plain unit tests

One class, no Kafka, milliseconds. Example: DetectionRulesTest.

2. Topology tests

The whole detector wiring runs in memory inside TopologyTestDriver, with a fake clock. Seconds each, because of the on-disk stores. Example: AimbotTopologyTest.

3. Integration tests (*IT)

The real Spring app against a real Kafka broker in Docker. Slowest, but they see things the in-memory driver cannot: real partitions, real internal topics, real transactions.

How to run them

./mvnw verify                                  # everything, including the *IT classes. Needs Docker Desktop running.
./mvnw -pl telemetry-detector -am test -Dtest=AimbotTopologyTest -Dsurefire.failIfNoSpecifiedTests=false
                                               # one unit test class. -am builds telemetry-common first.
./mvnw verify -DskipITs                        # unit tests and the coverage gate only, no Docker needed

Unit tests run under Maven's surefire plugin and *IT classes under failsafe. The parent pom.xml lists failsafe explicitly: Spring Boot only manages its version, and before that line was added every *IT was silently skipped.

I ran the full verify on 2026-11-19: 269 unit and 17 integration tests passed, and the 85% coverage gate was met in all four modules (see section 0).

The helper classes

HelperWhereWhat it gives a test
MutableClockcommon (test-jar)A Clock a test can set() and advance()
TestEventscommon (test-jar)One-line builders such as TestEvents.shot(player, seq, ts, hit, headshot, reactionMs), and player(1) for a fixed UUID
KafkaTestContainercommon (test-jar)One real apache/kafka-native:4.2.2 broker per test JVM, started once
TopologyFixturedetector testsThe real topology inside a TopologyTestDriver, a MutableClock, and ready-made input and output topics (fixture.events, fixture.flags, fixture.deadLetters, fixture.integrity). advanceTime(...) moves both the fake clock and the driver's wall clock.
TestDetection.defaults()detector testsThe standard thresholds as a DetectionProperties
KafkaItdetector testsPlain-client helpers for ITs: produce events and read a whole topic with read_committed
TestSimPropssimulator testsA standard SimulatorProperties, with the seed, player count and malformed ratio adjustable
StubArenaServercommon (test-jar)A fake Arena on the JDK HttpServer: scripted responses per route, every request recorded, a login that hands out token-1, token-2, ...

Test class to what it proves

Test classTestsWhat it proves
telemetry-common
JsonSerdeTest9Every event type and contract record survives JSON and back; null fields are omitted; field names are the contract; unknown fields are ignored; failures become SerializationException
MatchEventTest7isValid() per event type, boundary cases, strict UUID check
ModelHelpersTest4Dead-letter error cap at 500 characters, IntegrityStats.empty, the evidence factories
TopicsConfigTest4Nine topics with the right partitions, cleanup policy and retention; replication factor from settings
KafkaTestContainerIT1Docker and Testcontainers can start a Kafka broker that answers
telemetry-simulator
SimulatorPropertiesTest8The yml defaults bind and validate; environment overrides work; every broken rule stops startup
CheaterAssignmentTest61,000 players give 50 cheaters in the right mix and 20 pros; same seed, same answer
PopulationFactoryTest9Id and username formulas; population repeats for a seed; profiles switch exactly at the cheat start; seq counts up without gaps
BehaviorProfileTest9Every random draw stays inside its table range; tier boundaries
EventFactoryTest12Every generated event is valid; seq grows by exactly 1; one POSITION first per tick; shot rate matches the setting; aimbot and speedhack switch on at the right moment; same seed and clock give the same events
MatchSummaryBuilderTest11Placements 1..N, ranking and tie-breaks, Arena's invariants, ISO-8601 times
MalformedInjectorTest5Three kinds in rotation; none ever parses as a valid event; rate follows the ratio
telemetry-detector
DetectionRulesTest15Inclusive boundaries (12 hits yes, 11 no; 140 ms yes, 141 no; 0.62 yes) and the score formulas, plus HALF_UP rounding
StatsTest8AimStats and SpeedStats accumulate correctly, are immutable, and survive the JSON used by the window stores
DeadLetterTopologyTest7Garbage, invalid and wrong-key records reach the DLT with reason, offset and Base64 payload; good records after bad ones still flow
IntegrityTopologyTest12Count, duplicates and gaps; every player re-emitted every interval; bad records never reach the tracker
AimbotTopologyTest10An obvious burst gives exactly one flag with exact evidence; honest play never flags; cooldown; slow, few-headshot or too-few-hit bursts do not flag; bad records are not counted in windows
SpeedhackTopologyTest7The same for speedhacks, including idle samples not diluting the ratio
ProfileJoinTopologyTest5Username and tier are attached; unknown player still flagged with nulls; latest profile wins
TopologyShapeTest5What the topology reads and writes; no *-windows-repartition topic; never touches the ground-truth topic
DetectionPropertiesTest7The yml defaults bind; broken settings fail startup
WiringUnitTest7Metrics counters, every health state, the dead-thread handler, the clock bean
TopologyMetricsTest1The no-op metrics object does nothing and does not crash
DetectorEndToEndIT2On a real broker: one flag for the cheater (with profile fields), none for the honest player, one dead letter; topic partition counts; no window repartition topic
ExactlyOnceIT120 players × 500 events end at count 500, 0 duplicates, 0 gaps; a re-sent event is counted as exactly one duplicate
Added in steps 8 to 11
ManifestTrackerTest, EventPublisherTest, ShardWorkerTest, SimulationEngineTest, PlayerStateTest25The simulator engine: see section 5b
SimulatorIT2Gapless in-order seqs per player on a real broker, and the manifest agreeing with the topic
ArenaClientTest, ArenaPlayerRegistrarTest, ArenaPropertiesTest14The simulator's Arena registration: see section 5e
ArenaMatchClientTest, ArenaBridgeUnitTest16The bridge's HTTP client and listener parsing
ArenaBridgeIT6Retries, dead letters and re-login on a real broker with a stub Arena
DetectionEvaluatorTest, IntegrityCheckerTest, ReportWriterTest, VerifierRunnerTest, VerifierPropertiesTest, VerifierContextTest46The verifier: see section 5d
TopicSnapshotReaderIT5Latest value per key, tombstones, read_committed against an aborted transaction

One real test, taken apart

AimbotTopologyTest.anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence. It first needs a helper that sends a burst of shots:

detector/src/test/.../AimbotTopologyTest.java
private long burst(String player, long firstSeq, long startTs, int count, long spacingMs, int headshots,
                   int reactionMs) {
    for (int i = 0; i < count; i++) {
        boolean headshot = i >= count - headshots;
        fixture.send(TestEvents.shot(player, firstSeq + i, startTs + i * spacingMs, true, headshot, reactionMs));
    }
    return firstSeq + count;
}

Then the test itself:

@Test
void anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence() {
    // 15 shots over 2.8 s, all hits, 13 headshots (the last 13), 60 ms reaction.
    burst(P, 1, T0, 15, 200, 13, 60);                          // ARRANGE + ACT: feed the topology

    List<SuspiciousPlayer> flags = fixture.flags.readValuesToList();   // ACT: read what it wrote

    assertThat(flags).hasSize(1);                              // ASSERT: exactly one flag
    SuspiciousPlayer flag = flags.get(0);
    assertThat(flag.rule()).isEqualTo(Rule.AIMBOT);
    ...
    assertThat(flag.detectedAt()).isEqualTo(T0);
    assertThat(flag.windowStart()).isEqualTo(T0 - 2000);
    assertThat(flag.windowEnd()).isEqualTo(T0 + 3000);
    assertThat(flag.evidence()).isEqualTo(Evidence.aimbot(12, 12, 10, 0.8333, 60.0));
    // 0.5 + 0.25 x (0.8333 - 0.62) / 0.33 + 0.25 x (140 - 60) / 110
    assertThat(flag.score()).isEqualTo(0.843);
    ...
}
PhaseWhat happens
ArrangeThe class creates a TopologyFixture (the real topology, in memory, clock frozen at START = 2026-11-17T10:00:00Z). T0 is that moment in milliseconds. No broker, no Spring.
Actburst(...) pipes 15 shots into match-events, 200 ms apart, each with its own event time. Then the test reads the suspicious-players output topic.
AssertOne flag only (so the cooldown works). The window is the earliest one containing shot 12, [T0−2 s, T0+3 s). The evidence is 12 shots, 12 hits, 10 headshots, rate 0.8333, average reaction 60. The score matches the formula worked out in the comment. detectedAt is T0 because the fake clock never moved.

The same numbers are walked through step by step in section 9.

Why external services are faked (and when they are not)

8. Build and configuration

The module poms and the configuration files, one by one

FileWhat is in it
pom.xml (parent)Parent is spring-boot-starter-parent 4.1.2 (the same patch as Arena). java.version is 21. jacoco.skip is true by default (the parent has no code) and each child module switches it to false. jacoco.minimum is 0.85. Test libraries shared by every module: Spring Boot test starter, spring-boot-starter-kafka-test, Testcontainers (testcontainers-junit-jupiter, testcontainers-kafka) and Awaitility. The JaCoCo agent ignores *Application classes.
telemetry-common/pom.xmlKafka starter, kafka-streams, and Jackson 3 (tools.jackson.core:jackson-databind). A plugin packages the test helpers as a test-jar so other modules can use them.
telemetry-simulator/pom.xmlCommon, web and actuator starters, Kafka starter, validation, Prometheus registry, and Datafaker (fake country codes).
telemetry-detector/pom.xmlCommon, web, Kafka starter, kafka-streams, actuator, validation, Prometheus registry (runtime), and for tests kafka-streams-test-utils (that is where TopologyTestDriver lives) plus the common test-jar. There is no security dependency.
telemetry-verifier/pom.xmlCommon, Kafka starter, validation. No web starter: it is a command-line app, spring.main.web-application-type: none.
detector/.../application.ymlStreams settings (exactly_once_v2, 2 threads, 1 standby, 500 ms commit, 64 MB cache, session.timeout.ms: 6000, heartbeat.interval.ms: 2000), server port 8082, Actuator exposure, the detection thresholds, the telemetry.arena.* keys, and consumer and producer blocks that are used only by the Arena bridge (its consumer group telemetry-arena-bridge and the producer for dead letters).
simulator/.../application.ymlProducer settings (acks: all, idempotence on, lz4 compression, linger.ms: 10), port 8081, the telemetry.sim.* numbers, and the Arena keys. EventPublisher sends through this producer.
compose.yamlOne Kafka broker (apache/kafka:${KAFKA_VERSION}, KRaft, KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false", health-checked), Kafka UI on port 8090, the three detector replicas, the simulator and verifier behind profiles, Prometheus and Grafana.
.env.exampleDefaults for KAFKA_VERSION, SIM_SEED, SIM_PLAYERS, PROCESSING_GUARANTEE, ARENA_ENABLED=false, and a placeholder TELEMETRY_SERVICE_PASSWORD=change-me. Copy to .env (git-ignored), never commit real secrets.
.gitignoreIgnores target/, .env, reports/, data/ and sim-players.json.

9. One cheater, from first shot to flag

These are the real numbers of the scenario in AimbotTopologyTest.anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence, so every figure below is checked by a passing test. Let T0 be a moment aligned to a whole second. A cheater fires 15 shots, one every 200 ms, all hits, the last 13 headshots, each with a 60 ms reaction.

shot #123456789101112131415
time after T00.00.20.40.60.81.01.21.41.61.82.02.22.42.62.8
resulthithitHSHSHSHSHSHSHSHSHSHSHSHSHS
  1. Each shot arrives on match-events, keyed by the player id, and is parsed (valid, key matches).
  2. Each shot is added to five windows. The windows are 5 s long and start on whole seconds, so shot 12 (at 2.2 s) falls in windows starting at T0−2 s, T0−1 s, T0, T0+1 s and T0+2 s.
  3. Until shot 12 arrives, no window has 12 hits, so nothing fires. When shot 12 arrives, the earliest window, [T0−2 s, T0+3 s), holds shots 1 to 12: shots 12, hits 12, headshotHits 10, rate 10/12 = 0.8333, average reaction 60.
  4. The rule: hits 12 ≥ 12 ✓, 60 ≤ 140 ✓, 0.8333 ≥ 0.62 ✓. It fires.
  5. Score: 0.5 + 0.25 × (0.8333−0.62)/0.33 + 0.25 × (140−60)/110 = 0.5 + 0.1616 + 0.1818 = 0.843.
  6. The other overlapping windows also fire as more shots arrive. They flow into the flags repartition, then the profile join, then the cooldown, which lets the first through and drops the rest.
  7. One SuspiciousPlayer record lands on suspicious-players: rule AIMBOT, score 0.843, window [T0−2 s, T0+3 s), evidence {shots 12, hits 12, headshotHits 10, headshotRate 0.8333, avgReactionMs 60.0}, and the username/tier if a profile exists.
Detection latency The flag appears when the 12th hit arrives, about 2.2 s after the burst began here. For a real aimbot at ~3.4 hits per second (0.75 hit chance × 4.5 shots a second) that is about 3.5 s, plus up to 0.5 s of commit interval, which is where the "median under 5 seconds" target comes from. That is arithmetic from the settings. It has not been measured.

10. Decisions and deviations from the written spec

Where the spec was silent or wrong, the code marks the choice with a comment starting // DECISION: (there are 9 of them in the source). The table also includes things recorded in handoff.md.

WhatWhat the plan saidWhat was doneWhy
Window branchesSpec section 9's snippet groups the output of the process(...) parse step into windowsThe windows re-parse the raw stream with mapValues and the same EventParserprocess marks its output "key may have changed", so Kafka Streams would add aimbot-windows-repartition and speedhack-windows-repartition topics and push all ~24,000 events/s through the broker again. Found by printing topology.describe(). Guarded by TopologyShapeTest.
Kafka versionSpec assumed Kafka 4.1.x4.2.2 everywhere (images, Testcontainers, libraries)Spring Boot 4.1.2 manages kafka-clients 4.2.2. Update the image tag and KAFKA_VERSION together if Boot changes.
Config validationSpec section 6: "all counts, Hz, durations and thresholds are @Positive"Checks in the compact constructors of SimulatorProperties, DetectionProperties, TopicsPropertiesHibernate Validator has no @Positive for Duration, some rules compare two fields, and common then needs no validation starter.
Extra simulator rulesNot listedmatch-size ≥ 2, 1000 % tick-hz == 0, kill-chance-per-hit > 0, cheater-ratio + pro-ratio ≤ 1A kill needs a victim; dtMs must be a whole number; a zero chance means no kills ever; the roles must fit in the population.
dtMs > 0 for POSITIONNot listedAdded to MatchEvent.isValid()Speed is distance divided by time; zero would divide by zero.
Segment and RunInfo typesUsed by the spec but never definedPlain enum and record added in commonThe spec needs them.
JsonSerdes.of(Class)The snippet calls JsonSerdes.flagCandidate() and aimStats()Common offers a generic factory; the detector uses it for its own typesCommon cannot name types that live in the detector module.
Event idsSpec rule 7: simulator output is a function of the seedName-based UUID of run, player and seqNo hidden randomness.
Population random streamShard k uses SplittableRandom(seed + k)The population uses seed ^ 0x9E3779B97F4A7C15Otherwise population draws would replay shard 0's numbers.
MatchRoom.PlayerTotalsRule: every state value is a recordA small mutable classA thread-confined accumulator, updated thousands of times a second.
Random bytes"16 random bytes"The first byte is forced to 0xFFSo the garbage can never accidentally be valid JSON.
Strict UUID check"playerId is a UUID"isUuid compares the parsed UUID back with the inputUUID.fromString accepts shortened forms like 1-1-1-1-1.
FailsafeNot mentionedListed explicitly in the parent pomBoot only manages its version, so verify silently skipped every *IT.
JacksonJackson 3 onlyJackson 3 in source; Jackson 2 still on the runtime classpathkafka-streams brings Jackson 2 along. The rule is about source imports. com.fasterxml.jackson.annotation stays, because Jackson 3 keeps that package.
Verifier coverage85% per moduleThe check is vacuous for the verifier todayWith no test run there is no coverage file, so JaCoCo skips the check. It became real in step 9: the verifier is at 100%.
Arena configAn ArenaProperties record with validation, never defined in fullTwo records of that name, one per app, validated in their constructors (a blank service password fails startup when Arena is enabled)The two apps share no module for it, and they need different keys (registration versus retries).
finalSeq in the manifestNot specifiedThe last seq generated, not the last acknowledgedA failed send then shows up as finalSeq > acked instead of hiding.
Rooms of a shard"A worker reshuffles its own players into new rooms"All rooms of one shard start and end togetherRooms that end at different moments would need a reshuffle rule; this one is trivial.
telemetry.sim.autostartNot in the specA @ConditionalOnProperty on SimulationEngine, default trueTests and tools can have a Spring context without starting a 5-minute run, and no SimulatorProperties constructor call changes.
Registering players before the clock startsSpec order: run id, start time, population, registrationRegister first (using a throwaway population for the usernames), then start the clockRegistration can take a minute and cheat start times are offsets from the start time.
Verifier exit code 4 and settle-poll-intervalExit codes 0 to 3; a fixed 5 s re-readCode 4 for bad arguments; the interval is a propertyBad arguments must not look like a result; tests poll in milliseconds.
idIsGroup = false on the bridge listenerThe spec's snippet only sets idThe listener id no longer doubles as the consumer groupOtherwise arena-bridge silently overrides the group id in the YAML.
Memory caps in ComposeNo limitsmem_limit: 1g on the detector and simulatorWithout them the stack stalled a 16 GB laptop until Docker stopped answering.
Chaos scriptAs in the specUTC Z on the --since time, MSYS_NO_PATHCONV=1Docker would read the time as local time; Git Bash would rewrite /opt/kafka/... paths.

11. Interview preparation

Click a question to reveal a sample answer. The answers are in first person so you can adapt them. Every answer follows the same four beats, and you should too:

  1. Decision: what I chose.
  2. Reason: why, in one or two sentences.
  3. Trade-off: what it costs or what it gives up.
  4. How I checked: the specific test or run that backs it up.
How to talk about numbers Separate three things and never blur them: built and tested (the code and its automated tests), measured (what a real run produced, on which machine, how many times) and not done. In this project the honest split is: built and tested = all four modules, 286 passing tests. Measured = one 5-minute run per processing guarantee on one laptop: throughput, detection, exactly-once and failover. Not done = a live run against a real Arena, and the failover target (14.4 s measured against under 10 s). Say the missed targets first; it earns you trust for the rest.

The two pitches

The 30-second pitch

I built Fairplay Telemetry, a real-time cheat-detection pipeline in Java. A simulator pretends to be a thousand players, five percent of them cheating, and sends about 25,000 events a second into Kafka. A Kafka Streams detector, running as three instances, watches five-second windows of their shots and movements and flags aimbots and speedhacks. A verifier then scores the flags against the simulator's answer sheet: on my laptop it caught all 50 cheaters with 0.11 percent false positives, and after I killed one of the three detectors mid-run it still proved that not one of 7.4 million events was lost or duplicated. The one target I missed was failover: 14 seconds against my goal of 10.

The 3-minute pitch

The problem. Online shooters have cheaters: aimbots that hit almost everything with inhuman reaction times, and speedhacks that move faster than the game allows. I wanted to build the part of the system that notices them within seconds, and to be able to prove how well it works. This is project two of three. Project one is a stats service called Arena, and project three is an AI "judge" that reads my suspect list and decides what to do. My detector only flags. It never bans.

The shape. A simulator makes a thousand fake players, a few of them secretly cheating, and writes their events to Kafka, keyed by player id so each player's events stay in order. A Kafka Streams detector reads them. It cuts each player's events into five-second windows that overlap, and checks simple rules, for example "at least 12 hits, an average reaction under 140 milliseconds, and a headshot rate of at least 62 percent". A flag is joined to the player's profile, de-duplicated with a ten-minute cooldown, and published to a topic.

The interesting engineering. First, exactly-once. Every event carries a per-player sequence number, so the detector can count how many arrived, how many arrived twice, and how many are missing. That turns "I used exactly-once" into something checkable. Second, a trap I found: Kafka Streams quietly added two internal topics that would have re-written every event through the broker. I found it by printing the topology, fixed it by parsing in a way that promises not to change the key, and wrote a test that fails if the topics come back. Third, I kept the detection logic free of Spring and injected the clock, so I can test "eleven minutes later" instantly.

Testing. There are 269 unit tests and 17 integration tests that start a real Kafka in Docker, all passing, with the 85 percent coverage gate met in all four modules. A chaos run is part of the evidence: I kill one of three detectors 90 seconds into a five-minute run.

Where it honestly stands. All twelve steps are built and measured on one laptop. Measured: about 24,900 events a second, 100 percent recall, 0.11 percent false positives, the median flag 3.9 seconds after an aimbot starts, and zero lost or duplicated events after the kill. Not met: failover took 14.4 seconds against a 10-second target, and the consumer lag peaked near 380,000 events after the kill instead of staying flat. And I tested the Arena integration only against a stub Arena, not a live one. I would rather tell you that than have you find it.

Design and architecture

1. Why Kafka Streams, rather than a hand-written consumer or something like Flink?

I chose Kafka Streams because the detector needs windows, state that survives a restart, joins and exactly-once, and Kafka Streams gives all of those as a library inside a normal Spring Boot app. There is no separate cluster to run. The trade-off is that I am tied to Kafka for both input and output, and I scale by running more copies of the app, not by handing work to a scheduler. I checked that the approach works with in-memory topology tests and two tests against a real broker. I did not benchmark it against Flink or a plain consumer, so I cannot claim it is the fastest option, only that it fit the problem and kept the deployment small.

2. Why is the detector's topology plain code with no Spring?

I wrote TelemetryTopology so that it only takes the settings, a Clock and a metrics interface, and builds the wiring diagram. Spring's only job is to hand those in, in a three-line class called StreamsWiring. The reason is testability: a test can build the whole topology in memory and run it in milliseconds, with the clock under its control. The cost is one extra interface, TopologyMetrics, so the topology does not depend on Micrometer. I checked it by the fact that all the topology tests, such as AimbotTopologyTest, run through TopologyFixture with no Spring context at all.

3. Why 5-second windows that advance every second, with a 2-second grace?

A flag needs enough evidence, 12 hits, and an aimbot lands roughly three to four hits a second, so it needs about three to four and a half seconds of cheating. A new window every second means that soon after the cheat starts there is a window made mostly of cheating shots. One long window would mix honest and cheating shots and react slower. The grace lets slightly late events still count. The trade-off is that every event lands in five windows, so there is five times the state updating, and one cheater trips several overlapping windows, which is why I needed a cooldown. I checked the thresholds at their exact boundaries in DetectionRulesTest and a full burst scenario in AimbotTopologyTest. The "three to four seconds" figure is arithmetic from the simulator settings, not a measurement.

4. Why is the Kafka key the player id, and what happens if a record's key does not match?

Kafka keeps records with the same key in the same partition, in order. Keying by player id gives me per-player ordering, which the sequence counting and the windows need, and it makes match-events and player-profiles co-partitioned, which the join needs. If a record's key is not its own playerId, that would break the ordering promise, so EventParser sends it to the dead-letter topic. The cost is that a producer bug becomes a dead letter rather than a quiet success. I checked it with DeadLetterTopologyTest.aKeyThatIsNotThePlayerIdGoesToTheDeadLetterTopic and AimbotTopologyTest.eventsKeyedByAnotherPlayerAreNotCountedEither. I also removed the check on purpose and three tests failed, so I know the tests really guard it.

5. Why re-emit every player's integrity stats every 5 seconds instead of only the ones that changed?

The reader of integrity-stats only needs the latest value per player, and the topic is compacted. After a failover the new instance has no memory of what changed recently, so "emit everything on a timer" is the simplest rule that is always right. The trade-off is repeated records when nothing changed. At 1,000 players that is small, but it would not scale to millions of players. I checked the behaviour with IntegrityTopologyTest.everyPlayerIsReEmittedOnEveryIntervalEvenWhenNothingChanged. I have not measured the actual output volume.

6. Why a ground-truth topic that the detector is not allowed to read, and a separate verifier?

If the detector could see who really cheats, its accuracy score would be meaningless. So the simulator writes the answer sheet to cheater-ground-truth and only the verifier reads it. The cost is one more program, which I built as a one-shot command-line app. I enforce the rule with TopologyShapeTest.neverMentionsTheGroundTruthTopic, which inspects the topology description and fails if that topic appears anywhere. The measured result from the verifier: 50 of 50 cheaters detected, one false positive among 950 honest and pro players.

The key technologies

7. What is exactly_once_v2, and how do you know it works?

I set Kafka Streams to exactly_once_v2, which makes "read input, update state, write output, record my position" one atomic Kafka transaction. The reason is that after a crash I do not want duplicates or lost events in the output. The trade-offs are some added commit latency and that readers of my topics must use read_committed; I have not measured the throughput cost. To check it I added the sequence counters, and ExactlyOnceIT sends 20 players times 500 events to a real broker and expects count 500, zero duplicates and zero gaps for each. It also re-sends one event on purpose and expects exactly one duplicate, so I know the check can fail. What I have not done is crash a detector mid-run. That proof is step 12.

8. Tell me about the repartition trap.

The spec's sample groups the output of process(...) into windows. Kafka Streams cannot know whether a custom processor changed the key, so it inserts a repartition topic before grouping. I printed topology.describe() and saw two extra internal topics, aimbot-windows-repartition and speedhack-windows-repartition. They would have pushed every shot and movement event through the broker a second time, around 24,000 a second by my arithmetic from the settings. My fix was to give the window branches their own parse using mapValues, which promises not to change the key, with the same EventParser so both paths accept exactly the same records. The cost is that each event is parsed twice, which is cheap but I have not measured it. TopologyShapeTest.addsNoRepartitionTopicInFrontOfTheWindows fails if the topics come back, and DetectorEndToEndIT checks the real broker's topic list.

9. Why read events as raw bytes and parse inside the topology, instead of a deserialization exception handler?

A deserialization handler lives in Kafka's own plumbing and can only skip or fail. I wanted to keep the bad record, the reason, its partition and offset, and its original bytes, in a dead-letter topic where an operator can see them. Parsing inside the topology makes a bad record just another branch. The cost is the double parse I mentioned. I checked it in DeadLetterTopologyTest, where garbage, wrong-key and invalid records all appear in the dead-letter output with the right reason, and good records after them still flow.

10. Why one standby replica and a 6-second session timeout, and what did you measure?

My target was to recover from a killed detector in under 10 seconds. Kafka's default session timeout of 45 seconds would make that impossible, so I used 6 seconds (the broker's minimum), and a standby replica keeps a warm copy of the state on another instance. I then killed one of three detectors with docker kill 90 seconds into a run and timed it: 14.4 seconds, and 16.7 seconds in the second run. So I missed the target. The logs show why: about 5 seconds until the group noticed the dead member, then three rebalances in a row as Kafka Streams moved tasks in steps. The session timeout is only part of the cost. What I would try next is tuning warm-up replicas and recovery lag, or Kafka 4.2's new Streams rebalance protocol. Nothing was lost during the failover, which the integrity check proved.

11. Why your own JsonSerde and Jackson 3?

Spring Boot 4 uses Jackson 3, and Spring Kafka's built-in serde is built on Jackson 2, so I wrote a small class, 67 lines, that is a serializer, a deserializer and a serde in one. It ignores unknown fields so a newer producer cannot break an older reader. The trade-off is one more class to maintain. Jackson 2 still arrives on the classpath through kafka-streams, so my rule is about what source files import, not what is on the classpath. I checked it with JsonSerdeTest, nine tests including unknown fields, null handling and failure wrapping.

12. Why check settings in a record's constructor instead of @Positive?

The validation library has no @Positive for Duration, and several rules compare two fields, for example "the window cannot advance by more than its size" or "players must be a multiple of match-size times shard-threads". So all rules live in the compact constructors, in one place, and the app refuses to start on a bad setting. The trade-off is hand-written checks instead of annotations. I checked it with DetectionPropertiesTest and SimulatorPropertiesTest, which try each broken rule and the exact boundary values.

Failure handling and reliability

13. What happens to a malformed event?

It never stops the detector. ParseProcessor sends null bytes, unreadable JSON, an invalid event, or a key that does not match the player id to match-events.DLT as a DeadLetter with the source topic, partition, offset, key, the original bytes as Base64, the error text capped at 500 characters, and a timestamp. The trade-off is that the dead-letter topic can grow, so it keeps data for 7 days. I checked it in DeadLetterTopologyTest and, with a real garbage record, in DetectorEndToEndIT, which decodes the Base64 payload and compares it with what was sent.

14. What happens if a detector instance dies?

There are two cases. If a stream thread dies, StreamsRuntimeConfig replaces it instead of letting the instance die, and logs the state changes. If a whole instance dies, the group notices after the 6-second session timeout, the partitions move to the other instances, and the standby replicas already hold the state. I tested that second case for real: scripts/chaos-kill.sh kills one of three replicas mid-run. The two survivors took over all 6 partitions in 14.4 seconds, the lag spiked to about 380,000 events and took roughly two minutes to drain, and the verifier then showed every one of 1,000 players with the right count, no duplicates and no gaps.

15. How do you stop one cheater from producing dozens of flags?

A hopping window re-fires on every new event once it crosses the threshold, and five windows overlap each event, so an aimbot would trigger constantly. CooldownProcessor keeps a store keyed by playerId|rule and drops a candidate if the last flag for that key was under 10 minutes ago. The key includes the rule, so an aimbot flag never hides a speedhack flag. The trade-off is that "now" is the real clock, as the spec says, not event time, so replaying old data would count real minutes. I checked it with the burst tests that jump the fake clock 30 seconds (silent) and 11 minutes (flags again), and with SpeedhackTopologyTest.anAimbotFlagDoesNotHideASpeedhackFlag.

16. Could a reader of suspicious-players ever see a flag that was rolled back?

Under exactly_once_v2, output is written inside a transaction. If it aborts, a reader using read_committed never sees it, but a reader using the default read_uncommitted could. So I made it a project rule that consumers of the detector's outputs read with read_committed, and my integration-test helpers do. The flag id is a random UUID, which is fine in the detector because the "everything is seeded" rule is about the simulator. I have not written a test that forces a transaction to abort, so this is a rule I follow, not something I have demonstrated.

Security and safety

17. How do you handle secrets?

The only secret is the password for Arena's service account, read from the environment variable TELEMETRY_SERVICE_PASSWORD. It defaults to blank in application.yml, .env is git-ignored and also excluded from the Docker build context by .dockerignore, and .env.example holds no real value. A blank password fails startup when Arena is enabled: the ArenaProperties constructor throws a message naming the variable, and ArenaPropertiesTest checks it. The player passwords the simulator registers are throwaway test values for fake accounts.

18. What are the known security gaps?

This is a local development project, and I would not expose it as is. The detector exposes the Actuator endpoints health, info, metrics and prometheus with no authentication, and there is no Spring Security in the module. The Kafka broker and Kafka UI use plain text with no login. A real deployment would need TLS and authentication on the broker, access rules on the topics, and protected management endpoints. None of that is built, and I would say so up front rather than let someone assume otherwise.

19. Is it safe to flag people automatically?

My design is that the detector only flags. It never bans. Each flag carries a score from 0.5 to 1.0 and the evidence numbers behind it, so a human, or the Judge in project three, can review it. The simulator includes 20 honest "pro" players on purpose, who sometimes look like a subtle aimbot, so the false-alarm case is exercised. In the measured run exactly one of the 20 pros was flagged (0.538 score, just over the line) and none of the 930 ordinary players: a false-positive rate of 0.11 percent overall, under my 1 percent target. The trade-off is that the one false positive is a real borderline player, which is exactly what the Judge must clear, and a single run is not a statistic.

Testing

20. How do you test logic that depends on time?

I never call the system clock in main code. Everything receives a java.time.Clock. In production that is the real clock. In tests it is MutableClock, and TopologyFixture.advanceTime moves both that clock and the Kafka Streams test driver's wall clock together. So "11 minutes later" takes no real time. The cost is discipline, since one stray System.currentTimeMillis() would break testability. The tests that rely on it are aSecondBurstInsideTheCooldownDoesNotFlagAgain and aBurstAfterTheCooldownFlagsAgain.

21. Why do your integration tests use random ids, and why do they not check exact evidence numbers?

All integration tests share one real broker per JVM. If two Streams apps used the same application id they would join the same consumer group and fight over partitions, so each test uses a random application id and a random state directory, and it filters its assertions by its own random player ids. They do not check exact evidence because production settings use a 64 MB cache and emit window results at commit time, so the first flag can come from a different window than in the in-memory test. The integration tests assert who is flagged and how many times. The exact numbers are pinned in the in-memory tests, where emission is immediate.

22. You have near-100% line coverage. Does that mean it is correct?

No. Coverage only says a line ran during some test. I use an 85 percent gate per module to catch untested code. A real example of the gap: every verifier unit test passed, but the verifier could not start in Docker, because VerifierRunner had two constructors and Spring cannot choose. The tests built the class by hand. I fixed it and added VerifierContextTest, which starts the whole application context. What makes me trust the tests is more specific than coverage: boundary tests at exactly 12 hits and 140 milliseconds, negative controls such as re-sending an event and expecting exactly one duplicate, and the fact that when I changed >= to > in aimbotTriggered on purpose, two test classes failed.

23. Tell me about the integration tests that were silently skipped.

Spring Boot's parent POM only manages the version of the failsafe plugin. It does not switch it on. So ./mvnw verify ran the unit tests and quietly skipped every class ending in IT. The fix, listing the plugin explicitly in the parent POM, went in with the commit that added the Kafka container smoke test. The lesson is that a green build can hide tests that did not run. I now check the log for the "Running ...IT" lines and the counts. On 18 November the full build ran all four integration tests.

24. How do you know a test can actually fail?

Two ways. First, a negative control inside the test: ExactlyOnceIT re-sends one event and expects exactly one duplicate, which proves the detector would notice one. Second, I deliberately break the code and watch the tests fail. Changing the min-hits comparison made two tests fail, and removing the key-mismatch check made three tests fail. I restored both files afterwards. It is a manual check, not an automated mutation-testing tool.

Java, language and code quality

25. Why are events and settings records, and why is ParseResult a sealed interface?

Records are immutable and have equals, toString and getters for free, so events, settings and stored state values cannot be changed behind your back. A sealed interface lists every allowed implementation, so ParseResult is exactly Parsed or Failed, and the compiler knows there is no third case. The trade-off is that a record is not a good fit for a hot mutable counter, so MatchRoom.PlayerTotals is a small mutable class, marked // DECISION in the code. The roundtrip tests in JsonSerdeTest cover the records.

26. Why constructor injection and a Clock parameter everywhere?

Constructor injection with private final fields means an object is complete the moment it exists, and a test can build one by hand without Spring. Passing a Clock instead of calling the system time is what makes the time-dependent tests possible. The cost is longer constructors. WiringUnitTest builds the Spring-layer classes directly with new, which is only possible because of this style.

27. What do you do when the specification is silent or wrong?

I choose the simplest option that keeps the frozen contracts (topic names, JSON fields, enum values, config keys), and I leave a comment starting // DECISION: that says what and why. There are 15 of them in the source, for example the extra dtMs > 0 validity rule and the 0xFF first byte in the garbage payload. When the spec was actually wrong, as with the repartition snippet, I changed the code, wrote it up in handoff.md, and added a test that guards the new behaviour. The trade-off is that the code and the spec now differ, which is why section 10 of this guide lists every difference.

Honesty questions

28. What is missing from this project?

Three things. First, I have not run the Arena integration against a real Arena: the bridge and the player registration are tested against a stub on a real broker, and I checked the request shapes against Arena's source, but a live end-to-end run needs Arena's Postgres and Redis. Second, the failover target: 14.4 seconds measured against under 10, and a lag spike afterwards because two instances have little headroom over 25,000 events a second. Third, everything was measured on one laptop with the load generator on the same machine, one run per configuration, so I would not quote the numbers as general. GitHub CI is also written but has not run yet.

29. Can you quote performance or accuracy numbers?

Yes, with the setup attached. On a Ryzen 5 3600 with 16 GB, Docker limited to 7.7 GB, three detectors capped at 1 GB each, 1,000 players for 5 minutes, seed 42: the simulator sustained about 24,900 events a second with zero failed sends. Detection: recall 100 percent (50 of 50), false-positive rate 0.11 percent (one pro player), median detection 3.9 seconds for aimbots and 0.9 seconds for speedhacks. Integrity after killing one detector: 7,466,329 events expected and counted, 0 duplicates, 0 gaps. Misses: failover 14.4 seconds, and lag well above the band I expected. A caveat I would give unprompted: one run per setting, so these are measurements, not statistics, and my "subtle" cheaters were still all caught, so a harder cheat mix might not look as good.

30. Tell me about a bug or a surprise.

The repartition trap is the best one, because the code worked and the tests passed, and the problem was invisible until I printed the topology and noticed topics I had not asked for. A smaller one: in Jackson 3, serializing new Object() no longer fails, so the obvious test for "serialization errors are wrapped" cannot fail. I had to use a class with a getter that throws. And the silently skipped integration tests. The common thread is that I now verify the structure and the test run itself, not just that the build is green.

31. What would you do differently?

I would build a thin version of the whole pipeline earlier, the simulator engine and a minimal verifier, so I had real end-to-end numbers sooner and could tune the thresholds against them, instead of finishing the detector in isolation. I would also look for a way to avoid parsing every event twice. And I would test my weakest assumption early: the subtle cheaters sit right next to the thresholds, so I would run a quick experiment on them before building anything around the 98 percent target.

31. You compared exactly-once with at-least-once. What did you find?

The same run, the same seed and the same kill, with only PROCESSING_GUARANTEE changed. Under exactly_once_v2 every player passed: 7,466,329 expected, 7,466,329 counted, 0 duplicates, 0 gaps. Under at_least_once: I ran it twice, once with the default settings and once with the state-store cache switched off, which is when a store write can run ahead of the offset commit. Both runs passed the integrity check: 7,465,795 and 7,467,529 events expected and counted, 0 duplicates and 0 gaps. So I could not reproduce the duplicates my plan predicted, and I cannot honestly claim exactly-once beat at-least-once here. What I can claim is that the exactly-once run is clean and that the integrity checker does detect a duplicate (ExactlyOnceIT re-sends one event and expects exactly one). The per-player sequence numbers are what make that visible: a total count could hide a duplicate and a gap cancelling out, but a sequence number cannot. I would add that one run is one run: with at-least-once, duplicates depend on when the instance dies relative to its commits.

32. How does the Arena bridge avoid losing or duplicating matches?

It does not try to be exactly-once itself. The listener is at-least-once: if the app dies after posting but before committing its offset, the match is posted again. Arena treats a match's externalId as idempotent, so the second post returns 200 and changes nothing, and the end result is exactly-once. Temporary failures (429, 5xx, connection errors) are retried with exponential backoff and then dead-lettered; permanent ones (400, 404, 409, 422) go straight to match-summaries.DLT. A 401 triggers one re-login. ArenaBridgeIT proves each path against a real broker: a 422 is one call, three 503s are exactly three calls, garbage JSON is zero calls.

33. Why test the Arena bridge against a stub instead of the real Arena?

A stub lets me script the failures a real Arena almost never produces on demand: two 503s then a 201, a 401 in the middle, a 422. It is a 100-line class on the JDK's own HTTP server with no extra dependency. The trade-off is that a stub only knows what I told it, so I also read Arena's source to copy the exact request and response shapes, including the problem JSON with code and detail. I have not run it against a live Arena, and I say so.

34. What did running the whole stack teach you that the tests did not?

Four things, all invisible to green builds. The verifier crashed on start inside Docker because a class had two constructors. The simulator moved to port 8082 because the shared .env sets SERVER_PORT for the detectors and an environment variable beats the YAML, so Prometheus could not scrape it. The first stack with no memory limits stalled my 16 GB laptop until Docker stopped answering, because each JVM sized its heap from the whole Docker VM. And docker compose down -v does not remove services behind a profile, so my second measured run nearly reused the previous run's finished simulator. Each is now fixed and written down, and the first one has a test.

35. Your lag did not stay flat. What happened?

Before the kill the per-partition maximum lag sat at 15,000 to 20,000, above the roughly 12,500 I expected. When I killed one of three detectors, the two survivors fell about 14 seconds behind (about 380,000 events) and had only roughly 5,000 events a second of headroom over the 25,000 a second coming in, so draining took about two minutes. I also know that Kafka's client lag metric is a windowed maximum that lags reality, so I checked the consumer group directly: its lag was 0 once the run ended. To fix the spike I would add stream threads or CPU per detector, and I would shorten the failover itself.

12. How to practise reading the code

Reading is not enough. For each exercise, write down your prediction first, then check. Exercises 2 and 3 change source files. Always undo them with git checkout -- <file> afterwards, and never commit the change.

Eight exercises

Exercise 1 (predict, then check): does a slow burst flag?

Read AimbotTopologyTest.shotsSpreadOverMoreThanAWindowDoNotAccumulate. It sends 15 hits, one every 500 ms, all headshots, 60 ms reaction. Predict: does it flag? Hint: how many shots can fit in one 5-second window?

Answer: No. At one shot per 500 ms a 5-second window holds at most 10 shots, and the rule needs 12 hits. Run it with ./mvnw -pl telemetry-detector -am test -Dtest=AimbotTopologyTest -Dsurefire.failIfNoSpecifiedTests=false. It passes.

Exercise 2 (break a rule): make the hit boundary exclusive

In DetectionRules.aimbotTriggered change s.hits() >= aim.minHits() to s.hits() > aim.minHits(). Predict which tests fail, then run ./mvnw -pl telemetry-detector -am test -Dtest='DetectionRulesTest,AimbotTopologyTest' -Dsurefire.failIfNoSpecifiedTests=false -Djacoco.skip=true.

Answer (I ran this on 18 November): two tests fail: DetectionRulesTest.aimbotNeedsTwelveHitsInclusive, and AimbotTopologyTest.anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence (the flag is now raised one shot later, so the evidence the test got was shots=13, hits=13, headshotHits=11, headshotRate=0.8462 instead of the expected 12, 12, 10, 0.8333; the window start and end it asserts on the lines before still matched). Undo with git checkout -- telemetry-detector/src/main/java/dev/fairplay/telemetry/detector/topology/DetectionRules.java.

Exercise 3 (break a rule): remove the key check

In EventParser.parse, make the condition if (!event.playerId().equals(key)) never true (for example, replace it with if (false)). Predict the failing tests, then run -Dtest='DeadLetterTopologyTest,AimbotTopologyTest'.

Answer (I ran this on 18 November): three tests fail: DeadLetterTopologyTest.aKeyThatIsNotThePlayerIdGoesToTheDeadLetterTopic, DeadLetterTopologyTest.validEventsAfterBadOnesAreStillProcessedAndNeverDeadLettered, and AimbotTopologyTest.eventsKeyedByAnotherPlayerAreNotCountedEither. The third shows that the same predicate protects both the dead-letter path and the window path. Undo with git checkout.

Exercise 4 (change a setting): lower min-hits in the yml

Change min-hits: 12 to 11 in telemetry-detector/src/main/resources/application.yml. Predict which test class notices, and which one does not, even though it is about the same rule. Hint: look at what TestDetection.defaults() is, and at DetectionPropertiesTest.applicationYmlDefaultsBindToTheSpecValues.

Answer (from reading the tests, I did not run this one): DetectionPropertiesTest.applicationYmlDefaultsBindToTheSpecValues fails, because it compares the yml with TestDetection.defaults(), which still says 12. AimbotTopologyTest keeps passing, because it builds its settings from TestDetection.defaults() and never reads the yml.

Exercise 5 (recompute by hand): two scores

(a) The worked example in section 9 with the reaction changed from 60 ms to 100 ms: what is the score? (b) A speedhack window with 60 moving samples, 33 of them over the limit: what is the score?

Answer (hand arithmetic, checked with a calculator): (a) rate = 10/12 = 0.8333, so 0.5 + 0.25 × (0.8333 − 0.62)/0.33 + 0.25 × (140 − 100)/110 = 0.5 + 0.1616 + 0.0909 = 0.7525, rounded half-up to 0.753. (b) ratio = 33/60 = 0.55, so 0.5 + 0.5 × (0.55 − 0.15)/0.85 = 0.5 + 0.2353 = 0.7353, rounded to 0.735. To verify (a) in code, change the 60s in anObviousAimbotBurst... to 100 and expect Evidence.aimbot(12, 12, 10, 0.8333, 100.0).

Exercise 6 (recompute by hand): the integrity counters

A player's events arrive with seq 1, 2, 3, 3, 7. Apply the three rules of IntegrityProcessor.apply starting from all zeros. What are count, lastSeq, duplicates and gaps?

Answer: seq 1, 2, 3 are each "last + 1": count 3, lastSeq 3. The second 3 is "at or below last": duplicates 1, count unchanged. Seq 7 is above last + 1: gaps += 7 − 3 − 1 = 3, count 4, lastSeq 7. Result: count 4, lastSeq 7, duplicates 1, gaps 3. Compare with IntegrityTopologyTest.jumpingToSequenceOneHundredFiveCountsFourGaps.

Exercise 7 (draw a sequence diagram): one garbage record

On paper, draw the path of the 16 random bytes from MalformedInjector through the detector, with these lanes: Kafka match-events, ParseProcessor, EventParser, the split, the dead-letter topic, and the window branch.

Answer: The bytes arrive keyed by a player id. ParseProcessor.process calls EventParser.parse, whose readValue throws, so it returns Outcome(null, message). The processor calls metrics.deadLettered(), builds a DeadLetter from context.recordMetadata() and forwards ParseResult.Failed. The split sends it to the parse-failed branch, mapValues unwraps the DeadLetter, and it is written to match-events.DLT. Separately, the window branch calls EventParser.parse again, gets a null event, and filter drops it. It never reaches the integrity processor or any window.

Exercise 8 (explain aloud): the repartition trap in 60 seconds

Close the guide. Out loud, explain: what the spec's snippet did, what Kafka Streams added, why, what you changed, what it costs, and which two tests guard it. Then do the 30-second pitch from section 11. Record yourself and check that every number you quote is one of the measured ones in section 5g, with its caveats.

Checklist: process output is treated as possibly key-changing; two extra internal topics; every event rewritten through the broker; fix is a re-parse with mapValues; cost is a double parse; guards are TopologyShapeTest.addsNoRepartitionTopicInFrontOfTheWindows and DetectorEndToEndIT.

A 5-step method for reading any unfamiliar file

  1. Read the class comment. This project writes one sentence about each class's job above it. Say it back in your own words.
  2. Find the inputs and outputs. Look at the constructor and the main method (process, parse, tick). What goes in, what comes out, what does the class remember?
  3. Find the state. Fields, and for Kafka Streams the state stores it names (seq-tracker, flag-cooldown). State is where surprises live.
  4. Follow one concrete value. Pick a number, such as seq 105 after seq 100, and trace it line by line, writing the values down.
  5. Read its test last, and as a promise. Each test name is a sentence about what the class guarantees. If a behaviour you found is in no test name, it is untested.

13. Glossary

TermMeaning
AimbotCheat software that aims for the player: very high hit rate, many headshots, extremely fast reaction.
At-least-onceA delivery guarantee that never loses data but may repeat it. Contrast with exactly-once.
BrokerA Kafka server.
Co-partitionedTwo topics with the same number of partitions and the same key, so matching keys sit in matching partitions and can be joined.
CompactionKafka keeping only the newest record per key.
Consumer groupA set of program instances that share the partitions of a topic between them.
CooldownA quiet period after a flag during which the same player and rule are not flagged again.
DLT / dead letterA topic that holds records that could not be processed, plus the reason.
DTOData Transfer Object: a plain object that just carries data.
Epoch millisecondsMilliseconds since 1 January 1970 UTC.
Event timeWhen something happened in the game, as opposed to when we processed it.
FailoverAnother instance taking over when one dies.
False positiveAn innocent player wrongly flagged.
Grace periodHow late (in event time) an event may arrive and still be counted in a window.
Hopping windowA fixed-length time window that starts at regular steps shorter than its length, so windows overlap.
IdempotentDoing it twice has the same effect as once.
Integration test (IT)A test that uses real external pieces, here a real Kafka in Docker.
JSONA text format for structured data.
KRaftKafka's built-in coordination mode that replaced ZooKeeper.
KStream / KTableKafka Streams' view of a topic as a stream of facts / as a latest-value table.
LagHow far a reader is behind the newest record of a topic.
OffsetA record's position number within a partition.
PartitionOne ordered lane of a topic.
Precision / recallOf the players flagged, how many were truly cheating / of all true cheaters, how many were flagged.
ProcessorCustom Kafka Streams code handling one record at a time.
Record (Java)An immutable data-holding class defined in one line.
Record (Kafka)One entry in a topic: key, value, timestamp.
RepartitionRe-writing a stream through an internal topic so its records are grouped by a (new) key.
SeedThe starting number of a random generator; same seed, same sequence.
SerdeA serializer and deserializer pair.
Sealed interfaceA Java interface that lists all its allowed implementations.
SpeedhackCheat that moves the character faster than the game allows.
Standby replicaA warm copy of a task's state on another instance, for fast failover.
State storeA local database that a stream processor uses to remember things, backed up in Kafka.
Stream timeThe latest event timestamp a stream task has seen.
TestcontainersA library that starts Docker containers (like Kafka) for tests.
TopologyThe complete processing graph of a Kafka Streams application.
UUIDA 128-bit identifier written like 3f2a9c10-....
Wall-clock timeThe real clock, as opposed to stream time.

More terms from the later sections

TermMeaning
ActuatorThe Spring Boot add-on that serves /actuator/health, /actuator/metrics and similar status pages.
Arrange / act / assertThe three parts of a test: set things up, do the thing, check the result.
AssertJA test library with readable checks such as assertThat(x).isEqualTo(y).
AwaitilityA test library that retries a check until it passes or a time limit runs out.
BeanAn object that Spring creates and manages for you.
Changelog topicThe Kafka topic where a state store's changes are backed up, so another instance can rebuild it.
Commit intervalHow often Kafka Streams saves its progress and flushes cached output (500 ms here).
Compact constructorA record constructor with no parameter list, used to check or adjust values.
Dependency injectionPassing an object what it needs (here, through its constructor) instead of letting it create it.
Failsafe / SurefireThe Maven plugins that run integration tests (*IT) and unit tests.
JaCoCoThe tool that measures which lines the tests executed.
JacksonThe Java library that turns objects into JSON and back (version 3 here).
MicrometerThe metrics library behind Spring Boot's counters; Prometheus can read it.
Negative controlA deliberate wrong input in a test, to prove the check can actually fail.
Prometheus / GrafanaA metrics store and a dashboard tool. Both run in the Compose stack; Prometheus scrapes the apps every 5 seconds and Grafana draws the "Fairplay Telemetry" dashboard from it.
PunctuatorA timer inside a Kafka Streams processor (context.schedule).
read_committedA consumer setting that hides records from transactions that were aborted.
RocksDBThe small on-disk database Kafka Streams uses for state stores.
Test-jarA jar of one module's test helpers, shared with other modules' tests.
TopologyTestDriverA Kafka Streams test tool that runs a topology in memory, with controllable time.
Transaction (Kafka)A group of reads, writes and offset commits that all succeed or all vanish.