Fairplay Telemetry, explained from scratch
A guided tour of Project 2 for someone who has never touched Kafka, Kafka Streams or Spring Boot. Every idea is explained the first time it appears, and every code sample is taken from the real files in this repository.
0. Where the project stands
The project was built in 12 steps (the plan comes from the written specification in the project root). All twelve are finished and committed to git. Everything marked built exists and is tested. The one thing that is described but not done is a live run against a real Arena (Project 1), see the table below.
| Step | What | Status |
|---|---|---|
| 1 | Maven skeleton, four modules, Docker Compose with a Kafka broker | done |
| 2 | telemetry-common: shared data shapes, topic names, JSON conversion, test helpers | done |
| 3 | Simulator core: the fake players, how they behave, how they generate events | done |
| 4 to 6 | The detector's brain: the Kafka Streams topology | done |
| 7 | The detector as a real Spring Boot app, tested against a real Kafka | done |
| 8 | Simulator app: the engine that actually sends events to Kafka | done |
| 9 | Verifier: scores the detector's accuracy and checks nothing was lost | done |
| 10 | Dockerfile, three detector copies, Prometheus, Grafana, CI, the chaos script | done |
| 11 | Arena integration (registering players and sending finished matches to Project 1) | done (tested against a stub Arena) |
| 12 | Measured benchmark runs and the README | done |
Feature by feature
| Feature | State | Evidence |
|---|---|---|
| Parse events, dead-letter bad ones | built and tested | DeadLetterTopologyTest (7 tests), DetectorEndToEndIT |
| Integrity tracking (count, duplicates, gaps) | built and tested | IntegrityTopologyTest (12), ExactlyOnceIT on a real broker, and the measured run |
| Aimbot and speedhack windows, scoring | built and tested | AimbotTopologyTest (10), SpeedhackTopologyTest (7), DetectionRulesTest (15) |
| Profile join and cooldown | built and tested | ProfileJoinTopologyTest (5), the cooldown tests inside the aimbot and speedhack classes |
| Spring wiring, health, metrics | built and tested | WiringUnitTest (7). Prometheus now really scrapes all three detectors (checked in the measured run) |
| Simulator: core, engine, publisher, manifest, ground-truth publishing | built and tested | 99 unit tests and SimulatorIT (2, real Kafka). A real run sustained about 24,900 events per second |
| Verifier | built and tested | 46 unit tests and TopicSnapshotReaderIT (5, real Kafka) |
| Arena bridge and player registration | built, stub-tested | ArenaBridgeIT (6, real Kafka plus a stub Arena) and the client and registrar tests. Not run against a real Arena. |
| Docker Compose, Prometheus, Grafana, CI | built and run | The full stack was started and measured. The CI workflow file exists but has not run on GitHub yet |
| Chaos script, benchmarks, README | done | scripts/chaos-kill.sh, scripts/measured-run.sh, and the numbers in README.md |
Real test numbers (run on 2026-11-19)
I ran ./mvnw verify for the whole project with Docker Desktop running. It finished with BUILD SUCCESS.
| Module | Unit tests | Integration tests (real Kafka in Docker) | Line coverage (JaCoCo) |
|---|---|---|---|
telemetry-common | 24 | 1 (KafkaTestContainerIT) | 100% (98 of 98 lines) |
telemetry-simulator | 99 | 2 (SimulatorIT) | 99.2% (589 of 594) |
telemetry-detector | 100 | 9 (DetectorEndToEndIT 2, ExactlyOnceIT 1, ArenaBridgeIT 6) | 100% (397 of 397) |
telemetry-verifier | 46 | 5 (TopicSnapshotReaderIT) | 100% (358 of 358) |
| Total | 269 | 17 | 286 tests, 0 failures |
The build's gate is 85% line coverage per module. *Application classes (the main methods) are excluded from the count. Coverage here means "this line ran during a test", not "this line is correct". A real example of the difference: every verifier unit test passed while the verifier could not start inside Spring (see section 5d).
1. The story: what is this project for?
Imagine an online shooting game with thousands of players. A small number of them cheat:
- An aimbot is software that aims for you. It hits almost every shot, lands far more headshots than a human could, and reacts in a few milliseconds instead of a few hundred.
- A speedhack makes your character run faster than the game allows.
The game's servers produce a constant stream of tiny facts: "player X fired a shot, it hit, it was a headshot, they reacted in 60 ms", "player X moved 0.4 units". Those facts are called events. This project is a system that watches the events as they happen and shouts "player X looks like a cheater" within a few seconds.
There is no real game here, so a program called the simulator pretends to be the game: 1,000 fake players, 5% of them secretly cheating, producing about 25,000 events per second. That is the test bench.
An everyday analogy
Think of a bank's fraud team watching card payments. Every card produces a stream of small payments. The team does not judge each payment alone. It looks at the last few minutes of each card and asks "does this pattern look like a thief?" It also numbers the receipts, so a missing or doubled receipt is noticed straight away.
Map it across: the cards are players, the payments are events, "the last few minutes" is a 5-second window, the fraud team is the detector, and the numbered receipts are the seq counters.
The design promises
The whole project is organised around five promises. Each has a test or a mechanism that keeps it, which is what makes it more than a slogan.
| Promise | How it is kept | What guards it |
|---|---|---|
| 1. The detector never peeks at the answer sheet. | The simulator writes who really cheats to cheater-ground-truth. Only the verifier may read it. If the detector could, its accuracy score would be meaningless. | TopologyShapeTest.neverMentionsTheGroundTruthTopic |
| 2. Exactly-once is proven, not just claimed. | Every event carries a per-player seq. The detector counts what arrived, what arrived twice and what is missing. | IntegrityTopologyTest, and ExactlyOnceIT which even re-sends one event on purpose to show a duplicate would be caught. Not yet proven across a crash at full speed. That is step 12. |
| 3. Bad data never stops the pipeline. | A record that cannot be read is parked in match-events.DLT with the reason, and processing continues. | DeadLetterTopologyTest.validEventsAfterBadOnesAreStillProcessedAndNeverDeadLettered |
| 4. The contracts are frozen. | Topic names, JSON field names, enum values, config keys and metric names are never renamed. The flag format is designed as the contract with the Judge. | JsonSerdeTest.omitsNullFieldsAndKeepsContractFieldNames. I did not check the Judge's own code against the format while writing this guide. |
| 5. The logic can be tested without Spring and without waiting. | The detector's wiring diagram is plain Kafka Streams code. Time comes from an injected Clock, so a test can jump 11 minutes instantly. | Every *TopologyTest uses TopologyFixture and MutableClock |
Where this fits
Project 1: Arena
A separate web service that stores players and finished matches. Telemetry sends it match results, but this is optional and switched off by default.Project 2: Telemetry (this one)
Generates events, detects cheaters in real time, and measures how well it did.Project 3: Judge
A future AI agent that reads Telemetry's suspect list and decides who to ban. Telemetry only flags; it never bans.The four programs in this repository
| Module | Plain-English job | Status |
|---|---|---|
telemetry-common | A shared library. Defines what an event looks like, the names of the Kafka topics, and how to turn objects into JSON text and back. | built |
telemetry-simulator | Pretends to be the game. Makes players, decides who cheats, produces events. | built |
telemetry-detector | The star. Reads events, spots cheaters, publishes flags, and keeps count of every event so loss can be proven. | built (Arena bridge not yet) |
telemetry-verifier | The examiner. After a run it compares the detector's flags with the simulator's secret answer sheet and prints a grade. | built |
2. Ideas you need first
You can skip anything you already know. Each idea is short and used later.
An event, and JSON
An event is one small record of something that happened. Ours look like this when written as JSON, a plain-text format where data is written as "name": value pairs inside curly braces:
{ "eventId": "0b9c...", "runId": "r-9f2c41", "playerId": "3f2a...", "matchId": "sim-r-9f2c41-000017",
"seq": 1532, "type": "SHOT", "ts": 1763372123456,
"hit": true, "headshot": false, "reactionMs": 231 }
Two fields deserve a note. ts is the time of the event as epoch milliseconds: the number of milliseconds since 1 January 1970, which is how computers store points in time. seq is a per-player counter: a player's first event has seq 1, the next 2, then 3, with no gaps. That counter is the secret behind the "nothing was lost" proof later.
There are three kinds of event: SHOT (fired a weapon), POSITION (moved a little), and KILL (killed someone).
Java records
A record is a short way to write a Java class that only holds values. This one line:
public record PlayerProfile(
String playerId,
String username,
String country,
SkillTier skillTier,
long registeredAt) {
}
gives you a class with those five fields, a constructor, getters (profile.username()), and correct equals/toString, all automatically. Records cannot be changed after creation, which makes them safe to pass around. The project rule is that every event, DTO (a data-carrying object) and stored state value is a record.
More Java you will meet
Beyond records (above), these Java features appear again and again. Each example is a real line from the project.
- Compact constructor. A record can run checks when it is created. The extra code goes in a constructor with no parameter list, and it can even change the incoming values:
common/model/DeadLetter.java
The configuration records (public DeadLetter { if (error != null && error.length() > MAX_ERROR_LENGTH) { error = error.substring(0, MAX_ERROR_LENGTH); } }SimulatorProperties,DetectionProperties) use the same trick to throw when a setting is nonsense, so the program refuses to start. - Sealed interface. An interface that lists every class allowed to implement it, so the compiler knows the full set of possibilities:
detector/topology/ParseResult.java
public sealed interface ParseResult permits ParseResult.Parsed, ParseResult.Failed { record Parsed(MatchEvent event) implements ParseResult { } record Failed(DeadLetter deadLetter) implements ParseResult { } } - Pattern matching with
instanceof.result instanceof ParseResult.Failedasks "is it this kind?". In the tests you also see the formnode instanceof TopologyDescription.Source source, which both asks and gives you a ready-typed variable. - Switch expression. A
switchthat produces a value and uses->.MatchEvent.isValid()uses one to apply a different rule per event type. - Lambda and method reference. A lambda is a tiny inline function:
(key, event) -> event.type() == EventType.SHOT. A method reference is a shorthand for one that just calls a method:AimStats::empty,FlagCandidate::withProfile. - try-with-resources.
try (KeyValueIterator<...> all = store.all()) { ... }closes the iterator automatically, even if the code inside fails. An unclosed state-store iterator keeps native resources alive. - Explicit generic arguments.
Materialized.<String, AimStats, WindowStore<Bytes, byte[]>>as("aimbot-windows"): the part in<...>tells the compiler the types when it cannot guess them. finaland constructor injection. Fields areprivate finaland set once in the constructor. Spring passes dependencies in through constructors, never through fields.
record (lower case keyword) is the short data class described above. Kafka Streams also has a class named Record<K, V> (in org.apache.kafka.streams.processor.api) which is one key-value entry flowing through a processor. When you see Record<String, MatchEvent> in a processor, it is the Kafka one.Kafka
Apache Kafka is a very fast, durable message pipe. Think of it as a set of append-only logbooks that many programs can write to and read from independently.
| Word | Meaning here |
|---|---|
| Topic | A named logbook, for example match-events. Producers add entries to the end; entries are kept for a while (24 hours for events) and are never edited. |
| Producer / consumer | A program that writes to a topic / a program that reads from it. They never talk to each other directly, only through the topic. |
| Record, key, value | One entry. The value is our JSON; the key is a label, here always the player's id. |
| Partition | A topic is split into lanes so many readers can work in parallel. Records with the same key always go to the same partition, and within a partition order is guaranteed. Our key is the player id, so each player's events stay in order. match-events has 6 partitions. |
| Offset | The position number of a record within a partition (0, 1, 2, ...). A reader remembers its offset so it can resume after a crash. |
| Broker | The Kafka server. We run exactly one, in Docker, in "KRaft" mode (Kafka 4 removed the old ZooKeeper helper). |
| Compacted topic | A topic that keeps only the latest record per key, like a phone book rather than a diary. Player profiles work this way. |
| Dead-letter topic (DLT) | A "returns desk". A record that cannot be processed is parked there with the reason, so one bad record cannot stop the pipeline. |
Kafka Streams
Kafka Streams is a Java library for processing topics as they fill up: read records, transform them, count things, join with other data, and write the results to new topics. The pieces you will meet:
- KStream: an endless stream of records. Each one is its own fact ("player X fired a shot").
- KTable: a stream viewed as "latest value per key", like a lookup table ("what is player X's username?").
- Window: a slice of time to group events into. We use hopping windows: 5 seconds long, a new one starting every second, so they overlap. Any event belongs to five windows. Why: so that wherever a cheat starts, some window soon contains only cheating shots.
- State store: a little database (RocksDB on disk, with a backup copy in a Kafka topic) that remembers things between events, for example "how many shots have I seen in this window".
- Processor: your own code that handles one record at a time and can read/write a state store.
- Topology: the whole wiring diagram of sources, steps and outputs. In this project it is one Java class,
TelemetryTopology. - Stream time vs wall-clock time: stream time advances with the timestamps found in events; wall-clock time is the real clock. Windows use stream time; a "do this every 5 seconds" timer uses the wall clock.
Exactly-once
When a program crashes halfway through a job, restarting it can lead to some events being processed twice (duplicates) or skipped (losses). Kafka Streams has a setting called exactly_once_v2 that makes "read input, update the state store, write the output, remember how far I got" happen as one atomic transaction: it either all happens or none of it does. This project does not just claim that; it checks it with the seq counters (see Step B). So far that check has run on 10,000 events with no crash; the kill-a-detector test is step 12.
Spring Boot and Maven
- Maven is the build tool. A
pom.xmlfile lists a project's dependencies (libraries) and how to build it../mvnwis a small script that downloads the right Maven version for you, so you do not need to install it. - This is a multi-module build: one parent
pom.xmland four child folders, each with its ownpom.xml.telemetry-commonis built first because the others depend on it. - Spring Boot is a framework that wires a Java application together. You mark classes with annotations like
@Componentor@Configuration, and Spring creates the objects and hands them to whoever needs them (dependency injection, here always through constructors). Settings live in anapplication.ymlfile and are loaded into typed objects by@ConfigurationProperties. - Everything uses Java 21, Spring Boot 4.1.2 and Kafka 4.2.2.
Testing ideas used here
- Unit test: checks one small piece of logic in isolation, runs in milliseconds.
- TopologyTestDriver: a Kafka Streams tool that runs the real topology in memory, with no Kafka server, and lets the test control time. This is how we test the detection logic precisely.
- Testcontainers: starts a real Kafka server in a throwaway Docker container for a test. Tests with names ending
IT(integration test) use it, so Docker Desktop must be running. - Injected clock: instead of asking the computer "what time is it?" directly, code receives a
java.time.Clock. In production it is the real clock; in tests it is aMutableClockthe test can move forward by 11 minutes instantly. The rule "never callSystem.currentTimeMillis()" exists for this reason. - Coverage (JaCoCo): the percentage of code lines the tests execute. The build fails if any module drops below 85%.
Language and framework things you will meet
| Thing | What it is | Example from this project |
|---|---|---|
record | An immutable data class in one line | PlayerProfile, MatchEvent, AimStats |
| Compact constructor | Checks or tidies values when a record is created | DeadLetter (caps the error text), DetectionProperties (refuses nonsense settings) |
sealed interface | A closed list of allowed implementations | ParseResult is either Parsed or Failed |
| Switch expression | A switch that returns a value | MatchEvent.isValid(), KafkaStreamsHealthIndicator.health() |
Method reference X::y | Shorthand for a lambda that calls one method | AimStats::empty in TelemetryTopology |
@SpringBootApplication | Marks the class where the program starts and Spring scans for beans | DetectorApplication |
@ConfigurationProperties("telemetry.detection") | Copies a block of application.yml into a typed object | DetectionProperties. @ConfigurationPropertiesScan finds it |
@Configuration and @Bean | A method whose return value Spring creates once and shares | ClockConfig.clock(), the NewTopic beans in TopicsConfig |
@Component | A class Spring creates itself | StreamsWiring, MicrometerTopologyMetrics, KafkaStreamsHealthIndicator |
@Import(TopicsConfig.class) | Pulls in a configuration class that lives outside the scanned package | DetectorApplication, SimulatorApplication |
@EnableKafkaStreams | Tells Spring Boot to create and run a Kafka Streams client | DetectorApplication |
${NAME:default} in YAML | Read an environment variable, or use the default after the colon | ${PROCESSING_GUARANTEE:exactly_once_v2} |
10m, 5s in YAML | Spring turns these into a Java Duration | cooldown: 10m |
Processor<KIn,VIn,KOut,VOut> | Kafka Streams interface for custom per-record code | IntegrityProcessor, CooldownProcessor, ParseProcessor |
@JsonInclude(NON_NULL), @JsonIgnore | Jackson annotations that shape the JSON | MatchEvent (null fields are left out; isValid() is not written) |
@SpringBootTest | Starts the real Spring application inside a test | ExactlyOnceIT |
@DynamicPropertySource | Sets a configuration value at test time | Points spring.kafka.bootstrap-servers at the Docker broker |
@DirtiesContext | Throw away the Spring context after the test class | Both detector ITs, so the next one starts clean |
JUnit 5 @Test | Marks a test method | Every *Test class |
AssertJ assertThat(x).isEqualTo(y) | Readable assertions | Every test |
Awaitility await().atMost(...) | Keeps retrying a check until it passes or time runs out | ExactlyOnceIT waits for stats to arrive |
Testcontainers KafkaContainer | Starts a throwaway Kafka in Docker for a test | KafkaTestContainer |
3. Map of the repository
Size today: 124 Java files, about 4,100 lines of production code and 5,600 lines of tests (counted with wc -l). There are 286 automated tests: 269 unit tests and 17 integration tests.
Recommended reading order
common/model/MatchEvent.javaandSuspiciousPlayer.java: the two shapes everything is about.detector/topology/DetectionRules.javaand its testDetectionRulesTest: the thresholds, in plain Java, with no Kafka.detector/topology/AimStats.javaandSpeedStats.java: what a window remembers.detector/topology/IntegrityProcessor.java: the smallest processor, and the clever one.detector/topology/TelemetryTopology.java: the wiring diagram, once you know every piece it connects.detector/src/test/.../TopologyFixture.javathenAimbotTopologyTest: how a test drives the whole thing.simulator/engine/EventFactory.java: where events come from.- Last, the Spring layer in
detector/*.javaand theapplication.ymlfiles.
4. The journey of one event
Before reading any code, here is the whole system as a picture. Purple boxes are Kafka topics; blue boxes are programs or steps.
The nine topics, all declared in code by TopicsConfig:
| Topic | Partitions | Kept | Written by | Holds |
|---|---|---|---|---|
match-events | 6 | 24 h | simulator | the raw gameplay events |
match-events.DLT | 6 | 7 d | detector | records that could not be parsed, with the reason |
player-profiles | 6 | compacted | simulator | latest username and skill tier per player |
suspicious-players | 6 | 7 d | detector | the flags (the contract with the Judge) |
integrity-stats | 6 | compacted | detector | per-player count, duplicates, gaps |
match-summaries | 3 | 7 d | simulator | finished matches, shaped for Arena |
match-summaries.DLT | 3 | 7 d | Arena bridge | matches Arena rejected |
cheater-ground-truth | 1 | compacted | simulator | who really cheats (verifier only) |
simulation-manifest | 1 | compacted | simulator | how many events were really sent per player |
The numbered steps, with the class that does each one
This is the path of a single SHOT event. The tag says whether the code for that step exists today.
- A game tick creates the event.
EventFactory.tick(...)decides whether the player fires, whether it hits, how fast they reacted, and takes the nextseqfromPlayerState.nextSeq(). built - The loop that calls it every 50 ms (
ShardWorker, four of them, started bySimulationEngine) and the publisher that sends the bytes to Kafka (EventPublisher, which also counts every acknowledged send in theManifestTracker). built - Sometimes a garbage record is sent too (
MalformedInjector.maybeNext, 0.1% of the time). built TheShardWorkersends each bad record throughEventPublisher.publishMalformed; it is not counted in the manifest. - Kafka stores the record. Its key is the player id, so it lands in one of the 6 partitions of
match-events, in order with that player's other events. built (topic declared byTopicsConfig) - The detector reads the raw bytes.
TelemetryTopology.addTodeclaresbuilder.stream(MATCH_EVENTS, ...ByteArray()). built - Parse.
ParseProcessorcallsEventParser.parse. A good event moves on. A bad one becomes aDeadLetterand is written tomatch-events.DLT. built - Count.
IntegrityProcessor.processcomparesseqwith the last one for this player and updates theseq-trackerstore. Every 5 secondsemitAllwrites every player's counts tointegrity-stats. built - Window. The same raw bytes are parsed a second time with
mapValues(to avoid a hidden repartition, see the trap). OnlySHOTs continue, grouped by player into 5-second windows that start every second.AimStats.addfolds each shot in. built - Rule check.
DetectionRules.aimbotTriggeredtests the window's stats. If true,FlagCandidate.aimbotpackages it with a score. built - Enrich. The candidate is merged with speedhack candidates, re-keyed through the named
flagsrepartition step, and left-joined to theplayer-profilestable withFlagCandidate.withProfile. built - De-duplicate and publish.
CooldownProcessordrops repeats within 10 minutes and writes oneSuspiciousPlayertosuspicious-players. built - Somebody reads the flag. The Judge (project 3) reads
suspicious-players. The verifier readssuspicious-players,integrity-stats, the manifest and the ground truth to grade the run (section 5d). built
5a. telemetry-common: the shared vocabulary built
Both the simulator and the detector must agree on exactly what an event looks like and what the topics are called. Putting that in one library means they cannot disagree. This module has no main method; it is just a dependency.
Topic names and creation
Topics is a list of constants, so no other file ever types "match-events" by hand:
public static final String MATCH_EVENTS = "match-events";
public static final String MATCH_EVENTS_DLT = "match-events.DLT";
public static final String PLAYER_PROFILES = "player-profiles";
public static final String SUSPICIOUS_PLAYERS = "suspicious-players";
...
public static final int PLAYER_PARTITIONS = 6;
TopicsConfig turns each row of the table above into a Spring bean of type NewTopic. When an application starts, Spring's KafkaAdmin notices these beans and creates any topic that does not exist yet, so nobody has to create topics by hand. TopicsProperties supplies the replication factor (how many broker copies; 1 for our single broker) from telemetry.topics.replication-factor.
TopicsConfig lives in the common package, so SimulatorApplication and DetectorApplication import it explicitly.Turning objects into JSON and back: JsonSerde
Kafka only stores bytes. A serializer turns a Java object into bytes; a deserializer does the reverse; a serde is the pair. JsonSerde<T> does this with the Jackson 3 library, for any record type:
public static final JsonMapper MAPPER = JsonMapper.builder()
.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
.build();
@Override
public byte[] serialize(String topic, T value) {
if (value == null) {
return null;
}
try {
return MAPPER.writeValueAsBytes(value);
} catch (RuntimeException e) {
throw new SerializationException("Cannot serialize " + type.getSimpleName(), e);
}
}
Notes: unknown JSON fields are ignored (so a newer producer does not break an older consumer); a null value stays null (Kafka uses null as "delete this key"); any failure is wrapped in Kafka's SerializationException. JsonSerdes offers shortcuts like JsonSerdes.matchEvent() and the generic JsonSerdes.of(AimStats.class) for types that live outside this module.
The data shapes (the model package)
| Record | What it represents |
|---|---|
MatchEvent | One gameplay event. Fields that do not apply to its type are null and left out of the JSON. |
PlayerProfile | playerId, username, country, SkillTier (BRONZE to PRO), registeredAt. |
SuspiciousPlayer | The flag: who, which Rule (AIMBOT or SPEEDHACK), a score 0.5 to 1.0, the time window, and Evidence. This is the contract with the Judge; its field names must never change. |
Evidence | The numbers behind a flag. Aimbot fills shots/hits/headshotHits/headshotRate/avgReactionMs; speedhack fills movingSamples/overspeedSamples/overspeedRatio/maxSpeed. |
IntegrityStats | Per player: count, lastSeq, duplicates, gaps, updatedAt. |
DeadLetter | sourceTopic, partition, offset, key, the original bytes as Base64 text, the error (capped at 500 characters), receivedAt. |
GroundTruth, CheatType, Segment | The answer sheet: is the player HONEST, PRO or a CHEATER, and which cheat from AIMBOT, AIMBOT_SUBTLE, SPEEDHACK, SPEEDHACK_SUBTLE. |
ManifestEntry, RunInfo | What the simulator really sent, for the verifier. |
MatchSummary, ParticipantSummary | A finished match, shaped like Arena's API body. |
MatchEvent.isValid(): what counts as a good event
common/model/MatchEvent.java
@JsonIgnore
public boolean isValid() {
if (type == null || seq < 1 || ts <= 0 || !isUuid(playerId)) {
return false;
}
if (isBlank(eventId) || isBlank(runId) || isBlank(matchId)) {
return false;
}
return switch (type) {
case SHOT -> hit != null && headshot != null && reactionMs != null;
case POSITION -> dx != null && dy != null && dtMs != null && dtMs > 0;
case KILL -> !isBlank(victimId) && headshot != null;
};
}
The detector treats anything failing this check exactly like unreadable garbage: it goes to the dead-letter topic. @JsonIgnore stops Jackson from writing a bogus "valid" field into the JSON. The extra dtMs > 0 check is our own addition, because a movement speed is distance divided by time and dividing by zero would produce infinite speed.
Test helpers shared with other modules
Packaged as a "test-jar" so the simulator and detector tests can reuse them:
MutableClock: aClockthe test canset()andadvance().TestEvents: one-line builders, e.g.TestEvents.shot(player, seq, ts, hit, headshot, reactionMs), with predictable ids.KafkaTestContainer: starts one real Kafka (apache/kafka-native:4.2.2) per test run.KafkaTestContainerITis a smoke test proving Docker and Testcontainers work.
Why it is written this way, pitfalls, and what the tests prove
| File | Why this way | Pitfall | Tests that prove it |
|---|---|---|---|
Topics, TopicsConfig | Names live in one place, so a typo cannot create a topic nobody reads. NewTopic beans mean the apps create their own topics (the broker has auto-creation switched off). | Each app must @Import(TopicsConfig.class), or no topic is created. | TopicsConfigTest.declaresTheNineTopicsWithPartitionsAndCleanupPolicies (also checks the 24 h and 7 d retention and the compacted ones) |
JsonSerde | Our own class, because Spring's built-in one uses Jackson 2 and this project uses Jackson 3. Unknown JSON fields are ignored, so a newer producer cannot break an older reader. | Jackson 3 will serialize new Object() without complaint (empty beans no longer fail), so the test for "serialization fails" needs a value with a getter that throws. | JsonSerdeTest (9 tests): round trips, unknownFieldsAreIgnored, unserializableValueThrowsSerializationException, null handling |
MatchEvent.isValid() | One definition of "a good event", shared by the dead-letter path and the window path. | UUID.fromString alone accepts odd shortened text like 1-1-1-1-1, so isUuid compares the parsed result with the input. | MatchEventTest (7 tests), e.g. playerIdMustBeAUuid, positionNeedsDxDyAndPositiveDt |
DeadLetter | The 500-character cap keeps a dead-letter record small even when a parser error message is long. | The cap is applied in the constructor, so even a record read back from JSON is capped. | ModelHelpersTest.deadLetterCapsTheErrorAt500Characters |
SuspiciousPlayer, Evidence | The flag format is the contract with the Judge, so the field names are fixed and unused evidence fields are left out of the JSON. | Never rename a field. The aimbot and speedhack evidence share one record, so half of it is always null. | JsonSerdeTest.omitsNullFieldsAndKeepsContractFieldNames |
MutableClock, TestEvents, KafkaTestContainer | Shared test helpers, packaged as a test-jar so the other modules can reuse them. | The Kafka image tag must match the Kafka version Boot resolves (4.2.2). Update both together. | KafkaTestContainerIT proves Docker and Testcontainers work at all. |
5b. telemetry-simulator: the fake game built
The simulator's job is to produce believable traffic, including known cheaters, and to remember the truth. Everything is driven by a seed, a starting number for the random-number generator: the same seed yields the same players and the same behavior every time, which makes experiments repeatable.
Settings: SimulatorProperties
Loaded from application.yml under telemetry.sim: seed: 42, players: 1000, duration: 5m, match-size: 10, match-duration: 120s, tick-hz: 20 (20 game ticks per second), shots-per-second: 4.5, kill-chance-per-hit: 0.25, shard-threads: 4, cheater-ratio: 0.05, pro-ratio: 0.02, malformed-ratio: 0.001, and the window in which cheating may start (10 to 60 seconds in).
It is a record whose constructor checks the rules and refuses to start if one is broken. Example rules: ratios must be between 0 and 1; cheat-start-max must be below duration; players must divide evenly into rooms across worker threads (1000 = 10 × 4 × 25).
Who is who: CheaterAssignment
Shuffles the numbers 0 to 999 with the seed, then hands out roles in order: the first 50 become cheaters (20 AIMBOT, 10 AIMBOT_SUBTLE, 12 SPEEDHACK, 8 SPEEDHACK_SUBTLE), the next 20 become PROs, the remaining 930 are HONEST.
How each player behaves: BehaviorProfile
Each player gets a profile drawn once from ranges, e.g. an honest player's hit chance is a random number between 0.20 and 0.45 and typical reaction time is about 240 ms. A cheater has two profiles: the normal one, and a cheat one they switch to at a random moment. Aimbots change only the aiming numbers; speedhackers change only movement.
| hit chance | headshot chance (when hit) | reaction (ms) | |
|---|---|---|---|
| Honest | 0.20 to 0.45 | 0.25 to 0.55 | about 240 |
| Pro | 0.40 to 0.55 | 0.55 to 0.68 | 155 to 185 |
| Aimbot | 0.60 to 0.90 | 0.75 to 0.95 | 40 to 80 |
| Subtle aimbot | 0.45 to 0.60 | 0.60 to 0.70 | 110 to 135 |
A speedhacker still draws a base speed of 5 to 7 units per second, but that speed is then multiplied by 1.5 to 2.0 (roughly 7.5 to 14), and a subtle one by only 1.12 to 1.20 (roughly 5.6 to 8.4, so some samples stay under the detector's 7.5 limit). An honest player's speed is 2 to 7. A player's skill tier (BRONZE...PRO) is derived from their base headshot chance, so a cheater's tier does not give them away.
Building the population: PopulationFactory and PlayerState
For player i: the id is UUID.nameUUIDFromBytes(seed + ":" + i) (a UUID computed from text, so it is always the same for the same seed), the username is sim_0001, sim_0002..., the country comes from the Datafaker fake-data library, and the profiles and the cheat start time come from one seeded random generator. The result is a list of PlayerState records:
public BehaviorProfile profileAt(long nowMs) {
return isCheatingAt(nowMs) ? cheat : base;
}
public boolean isCheatingAt(long nowMs) {
return cheatStartsAt != null && nowMs >= cheatStartsAt;
}
public long nextSeq() {
return seq.incrementAndGet();
}
That is the whole "cheater switches on" mechanism: before cheatStartsAt the player behaves honestly; from that millisecond on, the cheat profile applies. nextSeq() hands out 1, 2, 3, ... for each event of that player.
One tick of the game: EventFactory
Every tick (1/20 s = 50 ms), each player produces, in this order:
- Exactly one POSITION: with probability
moveChancethey move at a random speed (times the speedhack multiplier if cheating and started), in a random direction, with a little measurement noise; otherwise they stand still (dx = dy = 0). - Zero or more SHOTs: the count follows a Poisson distribution with mean 4.5/20 = 0.225 per tick (about 4.5 shots a second). Each shot is a hit with probability
hitChance; a hit is a headshot with probabilityheadshotChance; the reaction time is drawn from a normal (bell-curve) distribution around the player's mean, never below a floor. - At most one KILL after each hit: with probability 0.25 a random other player in the room dies.
boolean hit = rng.nextDouble() < aim.hitChance();
boolean headshot = hit && rng.nextDouble() < aim.headshotChance();
int reactionMs = (int) Math.max(aim.reactionFloor(),
Math.round(aim.reactionMean() + aim.reactionSd() * rng.nextGaussian()));
room.recordShot(player, hit, reactionMs);
long shotSeq = player.nextSeq();
sink.accept(new MatchEvent(eventId(player.playerId(), shotSeq), runId, player.playerId(),
room.externalId(), shotSeq, EventType.SHOT, now, hit, headshot, reactionMs, ...));
With 1,000 players that adds up to 20,000 POSITION + 4,500 SHOT + about 400 KILL events per second, the "25,000 per second" target. All events in one tick share one timestamp from the injected clock. Event ids are computed from run, player and seq (not random), so everything stays a function of the seed.
Matches: MatchRoom and MatchSummaryBuilder
Players are grouped into rooms of 10. A MatchRoom keeps each player's running totals (kills, deaths, headshot kills, shots fired, shots hit, total reaction time). When a match ends, MatchSummaryBuilder ranks players (most kills first, then fewest deaths, then player id) and produces a MatchSummary shaped exactly like the JSON Arena accepts. It guarantees Arena's rules: placements are exactly 1..10, headshots never exceed kills, hits never exceed shots, and the average reaction is at least 1 even for a player who never fired.
Bad data on purpose: MalformedInjector
A detector that only sees perfect data is untested. With probability 0.1% per event, the injector adds a broken record: (1) truncated JSON {"broken":, (2) valid JSON missing required fields, or (3) 16 random bytes (the first forced to 0xFF so it can never accidentally be valid JSON). The detector must route these to the dead-letter topic and carry on.
Why it is written this way, pitfalls, and what the tests prove
| File | Why this way | Pitfall | Tests that prove it |
|---|---|---|---|
SimulatorProperties | The constructor checks every rule, so a bad setting stops the program at startup, not ten minutes into a run. Extra rules were added beyond the spec: match-size at least 2, 1000 % tick-hz == 0, kill-chance-per-hit above 0, cheater-ratio + pro-ratio at most 1. | With the defaults, players must be a multiple of 40 (10 × 4). Changing SIM_PLAYERS to 1,010 stops the app. | SimulatorPropertiesTest (8), e.g. startupFailsWhenPlayersIsNotAMultipleOfMatchSizeTimesShards |
CheaterAssignment | Shuffle the indices with the seed, then deal roles in order. The same seed always gives the same answer sheet. With 1,000 players: 50 cheaters (50 × 20 / 100 = 10 subtle aimbots, 12 speedhackers, 8 subtle speedhackers, and the remaining 20 plain aimbots), 20 pros, 930 honest. | Percentages are rounded down, and any rounding remainder goes to plain AIMBOT. | CheaterAssignmentTest.thousandPlayersGiveTheSpecCounts, sameSeedGivesTheSameAssignment, roundingRemainderGoesToAimbot |
PopulationFactory, PlayerState | All randomness comes from a seeded generator, so a run is repeatable. Player ids are derived from the seed and index, so they repeat across runs. | That repetition is why docker compose down -v is needed before each measured run: old integrity counters would otherwise carry over. Also, PlayerState holds an AtomicLong, so comparing two players with equals compares the counter by identity. Tests compare a projection instead. | PopulationFactoryTest (9), e.g. sameSeedGivesTheSamePopulationIncludingProfilesAndStartTimes, profileSwitchesExactlyAtCheatStart |
BehaviorProfile | One table of ranges per kind of player, drawn once. A cheater's cheat profile changes only its own family (aim or movement), so an aimbot still moves like an honest player. | Subtle speedhackers multiply their speed by only 1.12 to 1.20. From a base of 5 to 7 units/s that is 5.6 to 8.4, and the detector's limit is 7.5, so some of their samples are legitimately below it. | BehaviorProfileTest (9), one test per profile kind plus skillTierBoundaries |
EventFactory | One tick of one player. The shot count is a Poisson draw (Knuth's algorithm), so shots arrive randomly rather than on a metronome. | All events of a tick share one timestamp. Event ids are name-based UUIDs of run, player and seq, not random, to keep the simulator a function of the seed. | EventFactoryTest (12), e.g. shotRateMatchesShotsPerSecond, aimbotProfileSwitchesAtCheatStart, sameSeedAndClockReproduceTheSameEvents |
MatchRoom, MatchSummaryBuilder | The room is touched by exactly one worker thread, so it needs no locks. The builder guarantees every rule Arena's API enforces. | PlayerTotals is a small mutable class, a deliberate exception to "everything is a record", because it is updated thousands of times per second and never leaves the thread. | MatchSummaryBuilderTest (11), especially aSimulatedMatchSatisfiesEveryArenaInvariant |
MalformedInjector | A detector fed only perfect data is untested. | The 16 random bytes could by chance form valid JSON, so the first byte is forced to 0xFF, which can never begin valid text. | MalformedInjectorTest.noPayloadParsesIntoAValidEvent, allThreeKindsAreProducedInRotation |
Running it for real: the engine built
| Class | What it does | Why this way |
|---|---|---|
EventPublisher | The only class that talks to Kafka. publish(event) turns the event into JSON bytes and sends it to match-events with the player id as the key and the event time as the record timestamp. When Kafka acknowledges, it tells the manifest; when a send fails, it counts a failure and logs it. It never retries by hand. | The producer is idempotent, so Kafka already retries safely. Metrics objects are created once in the constructor, because creating one per event at 25,000 a second would cost real time. |
ManifestTracker | Per player: how many sends were acknowledged, how many failed, and the highest acknowledged seq. At the end it becomes one ManifestEntry per player. | Every counter is a LongAdder/AtomicLong, because Kafka's I/O thread updates them while four worker threads keep sending. finalSeq is the last seq generated, not the last acknowledged, so a failed send shows up as finalSeq > acked. |
ShardWorker | One of 4 worker threads. It owns a fixed set of rooms (so no data is shared with other threads), has its own SplittableRandom(seed + k) and its own MalformedInjector, and every 50 ms plays one tick for each of its players. When a match is over it publishes the summaries and reshuffles its players into new rooms. | A task given to scheduleAtFixedRate that throws is silently never run again, which would freeze a quarter of the players with no visible error. So run() catches everything, logs it and lets the next tick happen. All rooms of one shard start and end together, which keeps the reshuffle trivial. |
SimulationEngine | Starts when the app is ready, on a thread named simulation-main so the app and its actuator stay responsive. It builds the population, publishes one profile and one ground-truth record per player, starts the workers, logs events per second every 10 s, and after duration stops the workers and flushes. Then it publishes the manifest and RunInfo (key __run__) and logs SIMULATION COMPLETE. The app keeps running afterwards so Prometheus can still scrape it. | Each shard gets its own single-thread executor, so a shard never moves between threads. telemetry.sim.autostart=false switches the automatic start off for tests. With Arena enabled it registers the players first (section 5e). |
Measured on a laptop: about 24,900 events per second for the whole 5 minutes, 7,466,329 events acknowledged, 0 failed sends and 7,503 malformed records (0.1%).
| Test class | What it proves |
|---|---|
ManifestTrackerTest (4) | Counts per player, highest acknowledged seq ignoring out-of-order acks, and entries using the generated seq |
EventPublisherTest (10) | The record's topic, key and timestamp; acknowledged, failed and thrown sends all counted correctly; malformed records counted but kept out of the manifest; every other topic and key |
ShardWorkerTest (6) | Rooms dealt round-robin; gapless seqs; a finished match is published and replaced; bad records follow the ratio; a failing tick does not escape |
SimulationEngineTest (4) | A failed setup send fails the run instead of hanging it; stopping interrupts a run; with Arena enabled every record uses Arena's ids |
SimulatorIT (2, real Kafka) | 40 players for 6 seconds: every player's events arrive as seq 1..N in order, the manifest agrees (acked == finalSeq, 0 failed), 40 profiles and 40 ground-truth records, every shard finished a match, and the dashboard metrics exist |
5c. telemetry-detector: the brain built
This is the centerpiece, and the part worth understanding deeply. It is split in two on purpose:
The topology package
Pure Kafka Streams logic. No Spring. It receives plain objects (settings, a clock, a metrics counter) and builds the wiring diagram. Because it needs no Spring, tests can run it in memory in milliseconds.The Spring layer
A thin outer shell that gives the topology its real clock, real settings and real Kafka connection, plus health checks and metrics.Settings: DetectionProperties
detector/src/main/resources/application.yml (the interesting part)
telemetry:
detection:
detector-version: "1.0.0"
cooldown: 10m
integrity-emit-interval: 5s
aimbot:
window-size: 5s
advance: 1s
grace: 2s
min-hits: 12
max-avg-reaction-ms: 140
min-headshot-rate: 0.62
speedhack:
window-size: 5s
advance: 1s
grace: 2s
min-moving-samples: 40
moving-speed-floor: 0.5
speed-limit: 7.5
min-overspeed-ratio: 0.15
The same file configures Kafka Streams itself: processing.guarantee: exactly_once_v2, two stream threads, one standby replica, a 500 ms commit interval, and session.timeout.ms: 6000. That last one matters for later: if a detector instance is killed, the cluster only notices after the session timeout, so the default of 45 seconds would make a "recover in under 10 seconds" goal impossible.
The topology, one screen
TelemetryTopology.addTo(builder) in four moves. Read this once, then the sub-sections explain each move.
ParseProcessorgood → continue
bad → match-events.DLT
IntegrityProcessor→ integrity-stats
Step A: parse, and send bad records to the dead-letter topic
The detector reads match-events as raw bytes, not as ready-made objects. Why? A normal deserializer that hits garbage throws an exception inside Kafka's own plumbing, where handling it is awkward. By parsing inside our own code, a bad record is just another branch of the diagram.
EventParser is the single definition of a good record:
static Outcome parse(String key, byte[] bytes) {
if (bytes == null) {
return new Outcome(null, "null payload");
}
try {
MatchEvent event = JsonSerde.MAPPER.readValue(bytes, MatchEvent.class);
if (event == null) {
return new Outcome(null, "payload is JSON null");
}
if (!event.isValid()) {
return new Outcome(null, "invalid event: a required field is missing or malformed");
}
if (!event.playerId().equals(key)) {
return new Outcome(null, "key mismatch: record key '" + key + "' is not playerId '"
+ event.playerId() + "'");
}
return new Outcome(event, null);
} catch (RuntimeException e) {
return new Outcome(null, e.getMessage() != null ? e.getMessage() : e.getClass().getSimpleName());
}
}
Four things make a record bad: no payload, unreadable JSON, a failed isValid(), or a record key that is not the event's own playerId (which would break the "same key, same partition" promise). The key check is important: if a wrong-key event sneaked through, a player's events could land in different partitions and arrive out of order.
ParseProcessor wraps this in Kafka Streams' Processor interface. For a bad record it builds a DeadLetter using context.recordMetadata() (which tells it the topic, partition and offset of the record it is looking at) and forwards a ParseResult.Failed; for a good one it forwards ParseResult.Parsed. ParseResult is a Java sealed interface with exactly those two record types, so the compiler knows there are no other possibilities. Then the topology splits the stream in two:
Map<String, KStream<String, ParseResult>> outcome = parsed.split(Named.as("parse-"))
.branch((key, result) -> result instanceof ParseResult.Failed, Branched.as("failed"))
.defaultBranch(Branched.as("ok"));
outcome.get("parse-failed")
.mapValues(result -> ((ParseResult.Failed) result).deadLetter())
.to(Topics.MATCH_EVENTS_DLT, Produced.with(Serdes.String(), JsonSerdes.deadLetter()));
Operators can open Kafka UI and see exactly what failed and why, with the original bytes preserved as Base64 text.
Step B: integrity tracking, the exactly-once check
This is the clever bit. Because every player's events carry seq numbers 1, 2, 3 with no gaps, the detector can audit itself. For each player it remembers the last seq seen and applies three rules (the real code in IntegrityProcessor.apply):
| Situation | Meaning | Update |
|---|---|---|
seq == lastSeq + 1 | the next one, in order | count + 1, lastSeq = seq |
seq <= lastSeq | seen before: a duplicate | duplicates + 1 |
seq > lastSeq + 1 | something is missing: a gap | gaps + (seq - lastSeq - 1), count + 1, lastSeq = seq |
If the system is truly exactly-once, then for every player at the end: count equals the number of events sent, duplicates is 0, gaps is 0. The verifier (step 9) will compare against what the simulator says it sent.
The counts live in a state store named seq-tracker. A timer (context.schedule(interval, WALL_CLOCK_TIME, ...)) fires every 5 seconds and re-emits every player's stats to integrity-stats, not just those that changed. This is deliberate: after a failover the new machine has no memory of what changed, and the reader only needs the latest value per player, so repeating is safe and simple.
Step C: the windows, where cheaters are caught
For aimbots, the topology takes only SHOT events, groups them by player, and slices time into 5-second windows that start every second. For each (player, window) it keeps an AimStats value:
public record AimStats(int shots, int hits, int headshotHits, long sumReactionMs) {
...
public AimStats add(MatchEvent shot) {
return new AimStats(
shots + 1,
hits + (Boolean.TRUE.equals(shot.hit()) ? 1 : 0),
headshotHits + (Boolean.TRUE.equals(shot.headshot()) ? 1 : 0),
sumReactionMs + (shot.reactionMs() == null ? 0 : shot.reactionMs()));
}
...
public double avgReactionMs() {
return shots == 0 ? 0.0 : (double) sumReactionMs / shots;
}
...
public double headshotRate() {
return hits == 0 ? 0.0 : (double) headshotHits / hits;
}
}
Each new shot produces a new AimStats (records are immutable) which Kafka Streams stores. Speedhack detection is the same shape using POSITION events and SpeedStats, where speed = sqrt(dx² + dy²) / (dtMs / 1000); a sample counts as moving if speed is at least 0.5, and as overspeed if it is moving and above 7.5. Idle samples count toward nothing, so a player who stands still for most of the window is not diluted into looking innocent.
The rules, in DetectionRules
| Rule | Fires when ALL are true (boundaries inclusive) | Score |
|---|---|---|
| AIMBOT | hits ≥ 12, average reaction ≤ 140 ms, headshot rate ≥ 0.62 | 0.5 + 0.25·c((rate-0.62)/(0.95-0.62)) + 0.25·c((140-avg)/(140-30)) |
| SPEEDHACK | moving samples ≥ 40, overspeed ratio ≥ 0.15 | 0.5 + 0.5·c((ratio-0.15)/(1-0.15)) |
c(x) means "clamp to between 0 and 1". So a window that only just crosses the line scores 0.5, and one far beyond scores 1.0. Scores are rounded to 3 decimals and rates to 4, using round-half-up. "Inclusive" matters: exactly 12 hits triggers, 11 does not; exactly 140 ms triggers, 141 does not. These edge cases are individually tested.
KStream<String, FlagCandidate> aimFlags = windowEvents
.filter((key, event) -> event.type() == EventType.SHOT)
.groupByKey(Grouped.with(Serdes.String(), JsonSerdes.matchEvent()))
.windowedBy(TimeWindows.ofSizeAndGrace(aim.windowSize(), aim.grace()).advanceBy(aim.advance()))
.aggregate(AimStats::empty, (key, event, stats) -> stats.add(event),
Materialized.<String, AimStats, WindowStore<Bytes, byte[]>>as(AIMBOT_WINDOWS)
.withKeySerde(Serdes.String())
.withValueSerde(JsonSerdes.of(AimStats.class)))
.toStream()
.filter((windowedKey, stats) -> stats != null && rules.aimbotTriggered(stats))
.map((windowedKey, stats) -> KeyValue.pair(windowedKey.key(),
FlagCandidate.aimbot(windowedKey.key(), windowedKey.window(), stats, rules)));
Read it top to bottom: keep only shots, group by player, cut into hopping windows (grace of 2 s means events arriving up to 2 s late are still counted), fold every shot into the running AimStats, turn the updates back into a stream, keep those that trip the rule, and wrap them as FlagCandidates.
Step D: join with the profile, then de-duplicate
- Merge the aimbot and speedhack candidates into one stream.
- Repartition it (write to and re-read from an internal topic that Kafka Streams names
<application-id>-flags-repartition, because the step is namedflags). Changing the key from a "windowed key" back to a plain player id counts as a key change, and Kafka Streams needs the data re-sorted into partitions before it can join with another topic. Flags are rare, so this costs almost nothing. - Left join with the
player-profilestable: attachusernameandskillTier. "Left" means: if there is no profile, still emit the flag, just with those two fields empty. - Cooldown:
CooldownProcessorremoves repeats.
Why repeats exist: a hopping window re-evaluates on every new event, and five windows overlap each event. An aimbot would be flagged dozens of times a minute. The cooldown keeps a state store flag-cooldown mapping playerId|rule to the time of the last published flag. A new candidate within 10 minutes of the last is dropped; otherwise it is stamped with a new flagId (random UUID) and detectedAt and published to suspicious-players.
String storeKey = candidate.playerId() + "|" + candidate.rule();
long now = clock.millis();
Long lastEmitted = store.get(storeKey);
if (lastEmitted != null && now - lastEmitted < cooldown.toMillis()) {
return; // seen recently: drop it
}
store.put(storeKey, now);
metrics.flagged(candidate.rule());
...
context.forward(record.withValue(flag));
Because the key includes the rule, an aimbot flag never hides a speedhack flag for the same player.
The Spring layer (step 7)
| Class | What it does |
|---|---|
DetectorApplication | The entry point. @EnableKafkaStreams tells Spring Boot to create and run a Kafka Streams client from the spring.kafka.streams.* settings. @ConfigurationPropertiesScan loads DetectionProperties. |
ClockConfig | Provides the real Clock (UTC system time) as a bean. |
StreamsWiring | Three lines: when Spring creates it, it receives the StreamsBuilder, the settings, the clock and the metrics, and calls new TelemetryTopology(...).addTo(builder). This is the only point where Spring meets the topology. |
MicrometerTopologyMetrics | Implements TopologyMetrics with counters: telemetry.detect.events.parsed, ...dead_lettered, and telemetry.detect.flags tagged by rule. These feed Prometheus/Grafana later. They can over-count after a crash-retry, so they are for dashboards only; exact numbers come from the verifier. |
KafkaStreamsHealthIndicator | Feeds the /actuator/health page: UP when the client is RUNNING, OUT_OF_SERVICE while CREATED or REBALANCING (or not yet created), DOWN otherwise. |
StreamsRuntimeConfig | If a stream thread crashes, replace it instead of killing the instance; and log every state change ("Kafka Streams state transition from REBALANCING to RUNNING"). The log lines are a second way to time a failover. |
The "repartition trap" (a deliberate departure from the written spec)
The spec's sample code parses with a process(...) step and then groups that result into windows. When we printed the finished topology to check it, it contained two extra internal topics: aimbot-windows-repartition and speedhack-windows-repartition. The reason: Kafka Streams cannot tell whether a custom processor changed the key, so after one it plays safe and re-writes every record through a new topic before grouping. That would push every shot and movement event (about 24,000 per second) through the broker a second time.
The fix, in TelemetryTopology: the window branches do not use the process output. They re-parse the raw stream with mapValues, which promises "I do not change keys", using the very same EventParser so both paths accept exactly the same records:
KStream<String, MatchEvent> windowEvents = raw
.mapValues((key, bytes) -> EventParser.parse(key, bytes).event())
.filter((key, event) -> event != null);
The price is parsing each event twice (cheap). Two tests guard it: TopologyShapeTest fails if a *-windows-repartition topic returns, and DetectorEndToEndIT checks the real broker's topic list.
Pitfalls, and what the tests prove
| Topic | The catch | Guarded by |
|---|---|---|
| The cooldown uses the wall clock | "Now" in CooldownProcessor is the injected Clock, not the event's own time. If you replay yesterday's events, the cooldown counts real minutes, not game minutes. This follows the spec on purpose. | AimbotTopologyTest.aSecondBurstInsideTheCooldownDoesNotFlagAgain (30 s later: silent) and aBurstAfterTheCooldownFlagsAgain (11 min later: flags) |
| Flags are emitted on every window update | A window re-fires on each new event once it crosses the threshold, and five windows overlap each event. Without the cooldown one aimbot would produce dozens of flags a minute. | The cooldown tests above; DetectorEndToEndIT asserts exactly one flag for the cheater on a real broker |
| One cooldown per player and rule | The store key is playerId|rule, so an aimbot flag never hides a speedhack flag. | SpeedhackTopologyTest.anAimbotFlagDoesNotHideASpeedhackFlag, AimbotTopologyTest.theCooldownIsPerPlayer |
flagId is a random UUID | That is fine in the detector (the "everything is seeded" rule is about the simulator). Under exactly_once_v2 a flag written inside a transaction that later aborts is never visible to a read_committed reader, so readers of suspicious-players must use that setting or they could see a flag that was rolled back. | Nothing tests the abort case directly. The rule is written down in CLAUDE.md. |
| Integrity treats "late" as "duplicate" | By the three rules in Step B, a seq at or below the last seen one counts as a duplicate. It relies on Kafka keeping one player's records in order inside a partition, which the key guarantees. | IntegrityTopologyTest.applyCoversEveryCase |
| Metrics can over-count | A retried transaction can count an event twice, so the Micrometer counters are for dashboards only. Exact numbers come from the verifier. | Stated in the class comment of MicrometerTopologyMetrics |
| Every event is parsed twice | The price of the repartition fix. It is cheap CPU work, but it is not zero. I have not measured it. | DeadLetterTopologyTest and AimbotTopologyTest.invalidEventsAreDeadLetteredAndNeverCountedInTheWindows prove both parse paths accept exactly the same records |
| The detector's own endpoints are open | application.yml exposes health,info,metrics,prometheus and the module has no security dependency, so anyone who can reach port 8082 can read them. Fine on a laptop, a gap anywhere else. The broker and Kafka UI are also unauthenticated and unencrypted. | Known gap, not tested |
| Integration tests do not check exact evidence numbers | The production cache (64 MB) emits window results at commit time, so the first flag can come from a different window than in the in-memory test. The ITs assert who is flagged and how often. | DetectorEndToEndIT filters everything by its own random player ids |
5d. telemetry-verifier: the examiner built
The verifier is a one-shot command-line program, not a server. You start it, it reads some topics, writes a Markdown report into reports/, prints it, and exits with a number that scripts and CI can check. It has two modes, and it exists so that claims like "caught 98% of cheaters" and "lost no events" come from a measurement, not from hope.
The classes
| Class | What it does | Worth knowing |
|---|---|---|
TopicSnapshotReader | Reads a whole topic as it stands now. A plain consumer with no group id, read_committed, no auto-commit. It writes down each partition's end offset once, then polls until it has reached them (or read-timeout passes). readLatest keeps the last value per key and a null value (a tombstone) removes the key; readAll keeps every record. | read_committed matters: the detector writes in Kafka transactions, and a record of an aborted transaction must never be counted. TopicSnapshotReaderIT proves it with a real aborted transaction. |
DetectionEvaluator | Takes the run's RunInfo, the ground truth and every flag, and produces a DetectionReport. A cheater is detected when at least one flag arrives at or after the moment their cheat started; the latency is the earliest such flag minus that moment. A honest or pro player with any flag is a false positive. | Flags before the run started, and flags for players not in this run's ground truth, are ignored. Percentiles use "nearest rank" (sort, take index ceil(p/100 × n) - 1). Any ratio with a zero denominator is 0.0, never NaN. |
IntegrityChecker | For each player in the manifest, compares it with that player's IntegrityStats. A player passes only if stats exist, the simulator reported no failed sends, and duplicates = 0, gaps = 0, lastSeq = finalSeq and count = finalSeq. | It lists at most 20 failing players (with every reason) but counts them all. |
ReportWriter | Renders a report as Markdown and writes reports/detection-<runId>.md or integrity-<runId>.md. The first lines are the run id, start and end times and the mode. | Plain string building, so a report is easy to diff and paste into the README. |
VerifierRunner | Parses --mode, --min-recall, --max-fpr and --settle, runs the mode, and remembers the exit code. It is both a CommandLineRunner and an ExitCodeGenerator, so VerifierApplication.main can end with System.exit(SpringApplication.exit(context)). | Exit codes: 0 fine, 1 integrity failed, 2 a threshold was missed, 3 the simulation has not finished (no __run__ record), 4 bad arguments. Code 4 is our own addition. |
VerifierProperties | reports-dir, read-timeout and settle-poll-interval, validated in the constructor like the other property records. | settle-poll-interval is not in the written spec; it lets tests poll in milliseconds instead of 5 seconds. |
Why --settle exists
The detector publishes integrity totals every 5 seconds. When a run has just ended, the last totals may not have arrived yet, so a check at that instant could fail for no real reason. With --settle=60s the verifier re-reads integrity-stats every 5 seconds until every player passes or 60 seconds have gone by, then reports the last attempt.
VerifierRunner has two constructors: a public one for Spring and a package-private one that lets tests inject a clock. Spring cannot choose between two constructors, so the first run in Docker died with "No default constructor found", although every unit test was green (they built the class by hand). The fix was @Autowired on the public one, and VerifierContextTest now starts the whole application context so that kind of wiring mistake fails in mvn verify.What the tests prove
| Test class | What it proves |
|---|---|
DetectionEvaluatorTest (13) | A hand-built scenario of 7 players: recall 0.75, false-positive rate, precision, a wrong first rule not counted as correct, an early flag reported but not counted as detection, a missed cheater listed by username, flags before the run ignored, stale ground truth of another run ignored, nearest-rank percentiles for n = 1, 2 and 10. |
IntegrityCheckerTest (10) | The pass case and each failure reason on its own (duplicates, gaps, wrong last seq, wrong count, failed sends, missing stats), several reasons at once, the 20-failure cap, and an empty manifest not counting as a pass. |
ReportWriterTest (7) | The Markdown of both reports, the file name and directory creation, and an unwritable directory being an error. |
VerifierRunnerTest (12) | Every exit code, thresholds met and missed, a settle loop that needs three reads, a settle that gives up, and an interrupt during settling. |
VerifierContextTest (1) | The whole application starts the way the jar starts it. |
TopicSnapshotReaderIT (5, real Kafka) | Latest value per key, tombstones, typed reads, an empty topic returning at once, and an aborted transaction staying invisible. |
5e. The Arena integration built
Arena is Project 1, the game platform. With Arena enabled, the fake players become real accounts in Arena, and every finished match is recorded there. It is off by default (telemetry.arena.enabled=false): Telemetry runs fully on its own without it. It has two halves, one in each app.
In the simulator: registering the players
| Class | What it does |
|---|---|
arena/ArenaProperties | The connection settings. Its constructor refuses to start the app when Arena is enabled and service-password is blank. The password comes from TELEMETRY_SERVICE_PASSWORD, never from code. |
arena/ArenaClient | Uses Spring's RestClient. It logs in as telemetry-sim lazily and caches the token (logging in again once on any 401). register(player) posts to /api/v1/auth/register; on 409 USERNAME_TAKEN it looks the player up with GET /api/v1/players?q=... and takes the exact, case-insensitive username match. |
arena/ArenaPlayerRegistrar | Registers all players on virtual threads (cheap threads that are fine to block), limited by a semaphore of 16, because Arena hashes passwords slowly (bcrypt). It saves {"sim_0001": "<arena uuid>", ...} to sim-players.json, and a later run skips players already in that file. If any player cannot be registered the run does not start; the ones that worked are still saved. |
SimulationEngine (changed) | With a registrar present it asks Arena for ids before the run clock starts, then builds the real population and swaps each player's made-up id for Arena's with PlayerState.withPlayerId. So profiles, ground truth, events and match summaries all carry Arena's ids. |
runStartedAt (10 to 60 seconds in). Registering 1,000 players can take a minute, so starting the clock first would let cheats begin before any event was sent. The engine therefore builds a throwaway population only to learn the usernames (they do not depend on the clock), registers, starts the clock, and builds the real one.In the detector: the bridge
| Arena answers | The client does | Then |
|---|---|---|
| 201 | counts telemetry.bridge.matches.posted{outcome=created} | done |
| 200 | counts {outcome=replayed}: the match was already stored, this was a retry | done |
| 401 | logs in again and retries once inside the same call | a second 401 is non-retryable |
| 429, any 5xx, connection refused, timeout | throws RetryableArenaException | retried with exponential backoff (1 s, 2 s, 4 s), then dead-lettered |
| 400, 404, 409, 422 | throws NonRetryableArenaException carrying Arena's code and detail | dead-lettered at once |
Delivery is at-least-once (a record can be sent twice if the app dies at the wrong moment), but Arena treats a match's externalId as idempotent (a repeat returns 200 and changes nothing), so the end result is exactly-once. That is the same trick, "make the receiver tolerate repeats", used for the detector's integrity proof, only here it is Arena's job.
422 PLAYER_NOT_ELIGIBLE for any match that includes them, and that summary lands in match-summaries.DLT. That is correct behavior, not a bug.@KafkaListener(id = "arena-bridge") makes the listener id the consumer group id too, which would silently override the group-id: telemetry-arena-bridge in the YAML. The listener sets idIsGroup = false so the YAML value applies.What the tests prove
| Test class | What it proves |
|---|---|
StubArenaServer (in common's test-jar) | Not a test: a fake Arena built on the JDK's own HttpServer, answering from scripted queues of responses and recording every request. No extra dependency. |
ArenaClientTest (6), ArenaPlayerRegistrarTest (5), ArenaPropertiesTest (3) | Registration body and id, the 409 lookup with an exact match, token caching, one re-login on a 401, partial saves on failure, rerun skipping mapped players, a blank password failing startup. |
ArenaMatchClientTest (12) | Every row of the status table above, a refused connection being retryable, a login rejected versus a login that fails with 503. |
ArenaBridgeIT (6, real Kafka plus the stub) | 201: one call, no dead letter. 503, 503, 201: three calls, no dead letter. 503 forever: exactly three calls, then the dead-letter topic with Spring's headers. 422: one call, then the dead-letter topic. Garbage JSON: zero calls, dead-lettered. 401 then 201: a fresh token is used and the match still arrives. |
Real Arena: the client's request and response shapes were checked against Arena's source code in ../Project 1/arena (register, login, /players?q=, /matches, the problem JSON with code and detail). A live end-to-end run against a running Arena (which needs its Postgres and Redis) is described in the README but was not part of the measured runs.
5f. Delivery: Docker, Prometheus, Grafana, CI, scripts built
| File | What it is |
|---|---|
Dockerfile | One file for all three apps: docker build --build-arg MODULE=telemetry-detector .. A build stage compiles the module with the Maven wrapper (a cache mount keeps downloaded dependencies between builds), and a slim Java 21 runtime stage runs the jar as a non-root user with -XX:MaxRAMPercentage=75. |
.dockerignore | Keeps target/, reports, data, .git and .env out of the build context (so the password file is never copied into an image). |
compose.yaml | The whole stack. Default: Kafka, Kafka UI, 3 detector replicas (deploy.replicas: 3), Prometheus and Grafana. Behind profiles: sim (the simulator) and tools (the verifier), so each run starts on purpose. The detectors, simulator and verifier are capped at 1 GB each so the stack fits in Docker Desktop's VM on a 16 GB laptop. |
ops/prometheus.yml | Scrapes the simulator by name and finds the 3 detector replicas by DNS: Compose resolves the service name telemetry-detector to one address per replica. |
ops/grafana/... | A provisioned Prometheus data source (uid prometheus) and the "Fairplay Telemetry" dashboard with eight panels: events produced/sec, events processed/sec, stream process rate by instance, consumer lag, flags per minute by rule, dead letters per minute, cheaters active and bridge outcomes. The Kafka client metric names in it were read from a live /actuator/prometheus (kafka_stream_thread_process_rate, kafka_consumer_fetch_manager_records_lag_max), not guessed. |
.github/workflows/ci.yml | Runs ./mvnw -B verify on Ubuntu (which has Docker, so the Testcontainers tests run unchanged) and uploads the JaCoCo reports. |
scripts/chaos-kill.sh | Kills one detector replica with docker kill, waits until both survivors log Kafka Streams' own "State transition from REBALANCING to RUNNING", prints the elapsed milliseconds, and shows the consumer group so you can check that all 6 partitions have an owner. |
scripts/measured-run.sh | One whole measured run: fresh stack, the simulation, a chaos kill about 90 s in, then both verifier modes. Used for both the exactly-once and the at-least-once runs. |
Things the real run taught us
- An environment variable beats the YAML. The shared
.envsetsSERVER_PORT=8082for the detectors; the simulator also read it, so it listened on 8082 while Prometheus scraped 8081. Compose now setsSERVER_PORT=8081for the simulator explicitly. - Memory matters on a laptop. The first stack, with no limits, starved the 16 GB machine until Docker stopped answering and had to be restarted. Each JVM sized its heap from the whole Docker VM. The 1 GB caps fixed it.
docker compose down -vignores profile services. The simulator of the previous run survived it, so the measured-run script now names the profiles (--profile sim --profile tools down -v).- Git Bash rewrites paths.
/opt/kafka/bin/...becameC:/Program Files/Git/opt/...inside the chaos script;MSYS_NO_PATHCONV=1stops that.
5g. The measured results measured
These come from real runs on one laptop (AMD Ryzen 5 3600, 16 GB RAM, Docker Desktop with a 12-CPU, 7.7 GB VM). 1,000 players, 5 minutes, seed 42, three detector replicas, one killed with docker kill about 90 seconds in. Each configuration was run once.
| Claim | Target | Measured (exactly-once run) | Met? |
|---|---|---|---|
| Events per second | 25,000 | about 24,900 produced, steady for the whole 5 minutes (24,556 to 25,204 in every 10-second window); 7,466,329 events, 0 failed sends | yes |
Partitions of match-events | 6 | 6 | yes |
| Median detection latency, obvious cheaters | under 5 s | aimbot 3.9 s, speedhack 0.9 s | yes |
| Recall | at least 98% | 100% (50 of 50) | yes |
| False-positive rate | under 1% | 0.11% (1 PRO player of 950 honest and pro) | yes |
| Lost or duplicated events after killing 1 of 3 detectors | 0 and 0 | 0 and 0: 7,466,329 expected, 7,466,329 counted, all 1,000 players pass | yes |
| Failover | under 10 s | 14.4 s | no |
| Consumer lag within about 12,500 and flat | flat | 15,000 to 20,000 before the kill, a peak near 380,000 after it, drained in about 2 minutes | no |
| Test coverage | at least 85% | 100%, 99.2%, 100%, 100% (common, simulator, detector, verifier) | yes |
Exactly-once against at-least-once
Same seed, same kill about 90 seconds in, only the processing guarantee changed. The third run also disabled the state-store cache, which is the condition under which the specification expects an at-least-once detector to show duplicates.
| Run | Guarantee | Events expected = counted | Duplicates | Gaps | Failover | Integrity exit code |
|---|---|---|---|---|---|---|
r-49fe8f | exactly_once_v2 | 7,466,329 | 0 | 0 | 14.4 s | 0 |
r-df576c | at_least_once | 7,465,795 | 0 | 0 | 16.7 s | 0 |
r-8bfb5e | at_least_once, cache disabled | 7,467,529 | 0 | 0 | 14.5 s | 0 |
at_least_once (checked in their logged configuration). I did not find the reason. A likely candidate is that a duplicate needs the instance to die in the short window between a state-store write reaching the changelog and the offset commit, and that window was never hit; I did not verify that. The honest conclusion is narrow: these runs prove the exactly-once run lost and duplicated nothing, and ExactlyOnceIT proves the checker does catch a duplicate when one is injected, but they do not show exactly-once beating at-least-once. One more thing came out of the third run: with the cache off, the detector's median flag time rose from about 3 seconds to about 30 seconds, because writing every update to the changelog made it fall behind.Detection report of the exactly-once run
| Cheat type | Cheaters | Detected | p50 | p95 | max |
|---|---|---|---|---|---|
| AIMBOT | 20 | 20 | 3.9 s | 7.6 s | 10.9 s |
| AIMBOT_SUBTLE | 10 | 10 | 6.6 s | 21.4 s | 21.4 s |
| SPEEDHACK | 12 | 12 | 0.9 s | 2.5 s | 2.5 s |
| SPEEDHACK_SUBTLE | 8 | 8 | 2.0 s | 5.7 s | 5.7 s |
| All | 50 | 50 | 3.0 s | 17.6 s | 21.4 s |
All 50 first flags used the correct rule and none came before the cheat started. The single false positive was sim_0815, a PRO player, flagged by the aimbot rule with score 0.538 (12 hits, headshot rate 0.667, reaction 139 ms). That is the realistic borderline case the Judge is meant to clear: 1 of 20 pros (5%) against 0 of 930 ordinary players.
The failover, from the survivors' logs
- +0 s:
docker killon one of three detectors. - about +5 s: the group coordinator notices (the 6-second session timeout) and the first rebalance starts.
- +5 to +14 s: two more rebalances follow back to back, because Kafka Streams moves tasks in steps, each step requesting another rebalance.
- +14.4 s: both survivors log
State transition from REBALANCING to RUNNING. All 6match-eventspartitions are owned by the two survivors. - afterwards: a backlog of about 14 seconds of input (about 380,000 events) drains over roughly 2 minutes, because two instances have only about 5,000 events a second of headroom over the 25,000 a second arriving.
6. What is left, and what was learned all 12 steps done
Everything in the original plan now exists. This section lists what is honestly not finished, and the lessons the real runs taught.
Not done
| Item | Why | What it would take |
|---|---|---|
| A live run against a real Arena | The bridge and the registrar were tested against StubArenaServer, and their request shapes were checked against Arena's source code. Arena needs Postgres and Redis, which were not started. | Start Project 1, set ARENA_ENABLED=true, ARENA_BASE_URL and TELEMETRY_SERVICE_PASSWORD, run the simulator, then check GET /api/v1/players/{id}/matches |
| Failover under 10 seconds | Measured 14.4 s (exactly-once run) and 16.7 s (at-least-once run). About 5 s is the session timeout; the rest is three rebalances in a row as Kafka Streams moves tasks in steps. | Tune max.warmup.replicas, acceptable.recovery.lag and the heartbeat, or try Kafka 4.2's new Streams rebalance protocol; re-measure with scripts/measured-run.sh |
| Lag flat after a failover | Two surviving instances have only about 5,000 events/s of headroom over the 25,000/s input, so a 14-second outage left a backlog of about 380,000 events that took about 2 minutes to drain. | More stream threads or more CPU per detector |
| CI on GitHub | The workflow file exists and runs the same ./mvnw verify that passes locally, but it has not run on GitHub yet. | Push and look at the Actions tab |
| Avro and Schema Registry | An optional stretch in the spec. | A schema-registry service and an .avsc for MatchEvent |
Targets, now measured
| Target (spec section 15) | Result |
|---|---|
| 25,000 events per second sustained | about 24,900 produced and about 25,000 processed, steady for the 5 minutes (see section 5g) |
| Consumer lag flat and within about 12,500 | missed: 15,000 to 20,000 before the kill, a peak of about 380,000 after it |
| Median detection latency under 5 seconds for obvious cheaters | met: aimbot 3.9 s, speedhack 0.9 s |
| Recall at least 98% and false-positive rate under 1% | met: 100% and 0.11% |
| Zero lost and zero duplicated events after killing 1 of 3 detectors | met: 7,466,329 expected and counted, 0 duplicates, 0 gaps |
| Failover under 10 seconds | missed: 14.4 s |
Try it (Docker Desktop must be running)
# from the project folder
./mvnw verify # build everything and run every test (about 4 to 5 minutes)
cp .env.example .env
docker compose up -d --build # broker, Kafka UI, 3 detectors, Prometheus, Grafana
docker compose --profile sim up -d telemetry-simulator # the 5-minute run
docker compose --profile tools run --rm telemetry-verifier --mode=detection
docker compose --profile tools run --rm telemetry-verifier --mode=integrity --settle=60s
bash scripts/chaos-kill.sh # about 90 s into a run
docker compose --profile sim --profile tools down -v # reset before the next run
Kafka UI is on localhost:8090, Grafana on localhost:3000 (admin/admin) and Prometheus on localhost:9090. On Windows PowerShell use .\mvnw.cmd instead of ./mvnw.
7. The tests: how we know it works
There are three layers, from fast and small to slow and realistic:
1. Plain unit tests
One class, no Kafka, milliseconds. Example:DetectionRulesTest.2. Topology tests
The whole detector wiring runs in memory insideTopologyTestDriver, with a fake clock. Seconds each, because of the on-disk stores. Example: AimbotTopologyTest.3. Integration tests (*IT)
The real Spring app against a real Kafka broker in Docker. Slowest, but they see things the in-memory driver cannot: real partitions, real internal topics, real transactions.How to run them
./mvnw verify # everything, including the *IT classes. Needs Docker Desktop running.
./mvnw -pl telemetry-detector -am test -Dtest=AimbotTopologyTest -Dsurefire.failIfNoSpecifiedTests=false
# one unit test class. -am builds telemetry-common first.
./mvnw verify -DskipITs # unit tests and the coverage gate only, no Docker needed
Unit tests run under Maven's surefire plugin and *IT classes under failsafe. The parent pom.xml lists failsafe explicitly: Spring Boot only manages its version, and before that line was added every *IT was silently skipped.
I ran the full verify on 2026-11-19: 269 unit and 17 integration tests passed, and the 85% coverage gate was met in all four modules (see section 0).
The helper classes
| Helper | Where | What it gives a test |
|---|---|---|
MutableClock | common (test-jar) | A Clock a test can set() and advance() |
TestEvents | common (test-jar) | One-line builders such as TestEvents.shot(player, seq, ts, hit, headshot, reactionMs), and player(1) for a fixed UUID |
KafkaTestContainer | common (test-jar) | One real apache/kafka-native:4.2.2 broker per test JVM, started once |
TopologyFixture | detector tests | The real topology inside a TopologyTestDriver, a MutableClock, and ready-made input and output topics (fixture.events, fixture.flags, fixture.deadLetters, fixture.integrity). advanceTime(...) moves both the fake clock and the driver's wall clock. |
TestDetection.defaults() | detector tests | The standard thresholds as a DetectionProperties |
KafkaIt | detector tests | Plain-client helpers for ITs: produce events and read a whole topic with read_committed |
TestSimProps | simulator tests | A standard SimulatorProperties, with the seed, player count and malformed ratio adjustable |
StubArenaServer | common (test-jar) | A fake Arena on the JDK HttpServer: scripted responses per route, every request recorded, a login that hands out token-1, token-2, ... |
Test class to what it proves
| Test class | Tests | What it proves |
|---|---|---|
| telemetry-common | ||
JsonSerdeTest | 9 | Every event type and contract record survives JSON and back; null fields are omitted; field names are the contract; unknown fields are ignored; failures become SerializationException |
MatchEventTest | 7 | isValid() per event type, boundary cases, strict UUID check |
ModelHelpersTest | 4 | Dead-letter error cap at 500 characters, IntegrityStats.empty, the evidence factories |
TopicsConfigTest | 4 | Nine topics with the right partitions, cleanup policy and retention; replication factor from settings |
KafkaTestContainerIT | 1 | Docker and Testcontainers can start a Kafka broker that answers |
| telemetry-simulator | ||
SimulatorPropertiesTest | 8 | The yml defaults bind and validate; environment overrides work; every broken rule stops startup |
CheaterAssignmentTest | 6 | 1,000 players give 50 cheaters in the right mix and 20 pros; same seed, same answer |
PopulationFactoryTest | 9 | Id and username formulas; population repeats for a seed; profiles switch exactly at the cheat start; seq counts up without gaps |
BehaviorProfileTest | 9 | Every random draw stays inside its table range; tier boundaries |
EventFactoryTest | 12 | Every generated event is valid; seq grows by exactly 1; one POSITION first per tick; shot rate matches the setting; aimbot and speedhack switch on at the right moment; same seed and clock give the same events |
MatchSummaryBuilderTest | 11 | Placements 1..N, ranking and tie-breaks, Arena's invariants, ISO-8601 times |
MalformedInjectorTest | 5 | Three kinds in rotation; none ever parses as a valid event; rate follows the ratio |
| telemetry-detector | ||
DetectionRulesTest | 15 | Inclusive boundaries (12 hits yes, 11 no; 140 ms yes, 141 no; 0.62 yes) and the score formulas, plus HALF_UP rounding |
StatsTest | 8 | AimStats and SpeedStats accumulate correctly, are immutable, and survive the JSON used by the window stores |
DeadLetterTopologyTest | 7 | Garbage, invalid and wrong-key records reach the DLT with reason, offset and Base64 payload; good records after bad ones still flow |
IntegrityTopologyTest | 12 | Count, duplicates and gaps; every player re-emitted every interval; bad records never reach the tracker |
AimbotTopologyTest | 10 | An obvious burst gives exactly one flag with exact evidence; honest play never flags; cooldown; slow, few-headshot or too-few-hit bursts do not flag; bad records are not counted in windows |
SpeedhackTopologyTest | 7 | The same for speedhacks, including idle samples not diluting the ratio |
ProfileJoinTopologyTest | 5 | Username and tier are attached; unknown player still flagged with nulls; latest profile wins |
TopologyShapeTest | 5 | What the topology reads and writes; no *-windows-repartition topic; never touches the ground-truth topic |
DetectionPropertiesTest | 7 | The yml defaults bind; broken settings fail startup |
WiringUnitTest | 7 | Metrics counters, every health state, the dead-thread handler, the clock bean |
TopologyMetricsTest | 1 | The no-op metrics object does nothing and does not crash |
DetectorEndToEndIT | 2 | On a real broker: one flag for the cheater (with profile fields), none for the honest player, one dead letter; topic partition counts; no window repartition topic |
ExactlyOnceIT | 1 | 20 players × 500 events end at count 500, 0 duplicates, 0 gaps; a re-sent event is counted as exactly one duplicate |
| Added in steps 8 to 11 | ||
ManifestTrackerTest, EventPublisherTest, ShardWorkerTest, SimulationEngineTest, PlayerStateTest | 25 | The simulator engine: see section 5b |
SimulatorIT | 2 | Gapless in-order seqs per player on a real broker, and the manifest agreeing with the topic |
ArenaClientTest, ArenaPlayerRegistrarTest, ArenaPropertiesTest | 14 | The simulator's Arena registration: see section 5e |
ArenaMatchClientTest, ArenaBridgeUnitTest | 16 | The bridge's HTTP client and listener parsing |
ArenaBridgeIT | 6 | Retries, dead letters and re-login on a real broker with a stub Arena |
DetectionEvaluatorTest, IntegrityCheckerTest, ReportWriterTest, VerifierRunnerTest, VerifierPropertiesTest, VerifierContextTest | 46 | The verifier: see section 5d |
TopicSnapshotReaderIT | 5 | Latest value per key, tombstones, read_committed against an aborted transaction |
One real test, taken apart
AimbotTopologyTest.anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence. It first needs a helper that sends a burst of shots:
private long burst(String player, long firstSeq, long startTs, int count, long spacingMs, int headshots,
int reactionMs) {
for (int i = 0; i < count; i++) {
boolean headshot = i >= count - headshots;
fixture.send(TestEvents.shot(player, firstSeq + i, startTs + i * spacingMs, true, headshot, reactionMs));
}
return firstSeq + count;
}
Then the test itself:
@Test
void anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence() {
// 15 shots over 2.8 s, all hits, 13 headshots (the last 13), 60 ms reaction.
burst(P, 1, T0, 15, 200, 13, 60); // ARRANGE + ACT: feed the topology
List<SuspiciousPlayer> flags = fixture.flags.readValuesToList(); // ACT: read what it wrote
assertThat(flags).hasSize(1); // ASSERT: exactly one flag
SuspiciousPlayer flag = flags.get(0);
assertThat(flag.rule()).isEqualTo(Rule.AIMBOT);
...
assertThat(flag.detectedAt()).isEqualTo(T0);
assertThat(flag.windowStart()).isEqualTo(T0 - 2000);
assertThat(flag.windowEnd()).isEqualTo(T0 + 3000);
assertThat(flag.evidence()).isEqualTo(Evidence.aimbot(12, 12, 10, 0.8333, 60.0));
// 0.5 + 0.25 x (0.8333 - 0.62) / 0.33 + 0.25 x (140 - 60) / 110
assertThat(flag.score()).isEqualTo(0.843);
...
}
| Phase | What happens |
|---|---|
| Arrange | The class creates a TopologyFixture (the real topology, in memory, clock frozen at START = 2026-11-17T10:00:00Z). T0 is that moment in milliseconds. No broker, no Spring. |
| Act | burst(...) pipes 15 shots into match-events, 200 ms apart, each with its own event time. Then the test reads the suspicious-players output topic. |
| Assert | One flag only (so the cooldown works). The window is the earliest one containing shot 12, [T0−2 s, T0+3 s). The evidence is 12 shots, 12 hits, 10 headshots, rate 0.8333, average reaction 60. The score matches the formula worked out in the comment. detectedAt is T0 because the fake clock never moved. |
The same numbers are walked through step by step in section 9.
Why external services are faked (and when they are not)
- Speed. A test that needs a Kafka broker takes seconds to start. The in-memory driver lets most logic be checked in a fraction of that.
- Control of time. "11 minutes later" must take zero real minutes. A fake
Clockand the driver's controllable stream time allow it. - Repeatability. No network, no leftover state from earlier runs.
- But not everything is faked. The in-memory driver cannot tell you whether real topics have the right partition counts, whether Kafka Streams quietly added an internal topic, or how transactions behave. That is exactly why
DetectorEndToEndITandExactlyOnceITuse a real broker. - How the ITs stay independent. They share one broker per JVM, so each uses a random
application-idand state directory (otherwise two Streams apps would join the same consumer group),commit.interval.ms=100to be fast, and filters assertions by its own random player ids.
8. Build and configuration
pom.xml(parent): sets Spring Boot 4.1.2 as the base, lists the four modules, adds the shared test libraries, and configures two plugins. JaCoCo measures coverage and fails the build under 85% per module. Failsafe runs the*ITtests duringverify(Boot only declares it, so without listing it the integration tests were silently skipped).- Module poms: each lists exactly what it needs.
telemetry-commonalso produces a test-jar of helpers. compose.yaml: starts a single Kafka broker (apache/kafka:4.2.2, KRaft mode, auto-topic-creation off) plus Kafka UI on localhost:8090 for browsing topics, three detector replicas, Prometheus and Grafana, and (behind thesimandtoolsprofiles) the simulator and the verifier. Every app container is capped at 1 GB.Dockerfile: one file for the three apps, selected with--build-arg MODULE=telemetry-detector(or-simulator,-verifier). Stage one builds the jar with the Maven wrapper; stage two is a slim Java 21 runtime running as a non-root user.ops/:prometheus.yml(scrapes the simulator and finds the three detector replicas by DNS) andgrafana/(a provisioned data source and the "Fairplay Telemetry" dashboard).scripts/:chaos-kill.shandmeasured-run.sh..github/workflows/ci.yml:./mvnw -B verifyon every push, plus the coverage reports as an artifact..env.example: copy to.envto change settings (seed, player count, Kafka version...). The real.envis git-ignored.- Kafka version note: the written spec assumed Kafka 4.1; Spring Boot 4.1.2 actually manages 4.2.2, so that is what runs everywhere (container image, test container, libraries).
The module poms and the configuration files, one by one
| File | What is in it |
|---|---|
pom.xml (parent) | Parent is spring-boot-starter-parent 4.1.2 (the same patch as Arena). java.version is 21. jacoco.skip is true by default (the parent has no code) and each child module switches it to false. jacoco.minimum is 0.85. Test libraries shared by every module: Spring Boot test starter, spring-boot-starter-kafka-test, Testcontainers (testcontainers-junit-jupiter, testcontainers-kafka) and Awaitility. The JaCoCo agent ignores *Application classes. |
telemetry-common/pom.xml | Kafka starter, kafka-streams, and Jackson 3 (tools.jackson.core:jackson-databind). A plugin packages the test helpers as a test-jar so other modules can use them. |
telemetry-simulator/pom.xml | Common, web and actuator starters, Kafka starter, validation, Prometheus registry, and Datafaker (fake country codes). |
telemetry-detector/pom.xml | Common, web, Kafka starter, kafka-streams, actuator, validation, Prometheus registry (runtime), and for tests kafka-streams-test-utils (that is where TopologyTestDriver lives) plus the common test-jar. There is no security dependency. |
telemetry-verifier/pom.xml | Common, Kafka starter, validation. No web starter: it is a command-line app, spring.main.web-application-type: none. |
detector/.../application.yml | Streams settings (exactly_once_v2, 2 threads, 1 standby, 500 ms commit, 64 MB cache, session.timeout.ms: 6000, heartbeat.interval.ms: 2000), server port 8082, Actuator exposure, the detection thresholds, the telemetry.arena.* keys, and consumer and producer blocks that are used only by the Arena bridge (its consumer group telemetry-arena-bridge and the producer for dead letters). |
simulator/.../application.yml | Producer settings (acks: all, idempotence on, lz4 compression, linger.ms: 10), port 8081, the telemetry.sim.* numbers, and the Arena keys. EventPublisher sends through this producer. |
compose.yaml | One Kafka broker (apache/kafka:${KAFKA_VERSION}, KRaft, KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false", health-checked), Kafka UI on port 8090, the three detector replicas, the simulator and verifier behind profiles, Prometheus and Grafana. |
.env.example | Defaults for KAFKA_VERSION, SIM_SEED, SIM_PLAYERS, PROCESSING_GUARANTEE, ARENA_ENABLED=false, and a placeholder TELEMETRY_SERVICE_PASSWORD=change-me. Copy to .env (git-ignored), never commit real secrets. |
.gitignore | Ignores target/, .env, reports/, data/ and sim-players.json. |
9. One cheater, from first shot to flag
These are the real numbers of the scenario in AimbotTopologyTest.anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence, so every figure below is checked by a passing test. Let T0 be a moment aligned to a whole second. A cheater fires 15 shots, one every 200 ms, all hits, the last 13 headshots, each with a 60 ms reaction.
| shot # | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| time after T0 | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 1.2 | 1.4 | 1.6 | 1.8 | 2.0 | 2.2 | 2.4 | 2.6 | 2.8 |
| result | hit | hit | HS | HS | HS | HS | HS | HS | HS | HS | HS | HS | HS | HS | HS |
- Each shot arrives on
match-events, keyed by the player id, and is parsed (valid, key matches). - Each shot is added to five windows. The windows are 5 s long and start on whole seconds, so shot 12 (at 2.2 s) falls in windows starting at T0−2 s, T0−1 s, T0, T0+1 s and T0+2 s.
- Until shot 12 arrives, no window has 12 hits, so nothing fires. When shot 12 arrives, the earliest window, [T0−2 s, T0+3 s), holds shots 1 to 12:
shots 12, hits 12, headshotHits 10, rate 10/12 = 0.8333, average reaction 60. - The rule: hits 12 ≥ 12 ✓, 60 ≤ 140 ✓, 0.8333 ≥ 0.62 ✓. It fires.
- Score:
0.5 + 0.25 × (0.8333−0.62)/0.33 + 0.25 × (140−60)/110= 0.5 + 0.1616 + 0.1818 = 0.843. - The other overlapping windows also fire as more shots arrive. They flow into the
flagsrepartition, then the profile join, then the cooldown, which lets the first through and drops the rest. - One
SuspiciousPlayerrecord lands onsuspicious-players: rule AIMBOT, score 0.843, window [T0−2 s, T0+3 s), evidence{shots 12, hits 12, headshotHits 10, headshotRate 0.8333, avgReactionMs 60.0}, and the username/tier if a profile exists.
10. Decisions and deviations from the written spec
Where the spec was silent or wrong, the code marks the choice with a comment starting // DECISION: (there are 9 of them in the source). The table also includes things recorded in handoff.md.
| What | What the plan said | What was done | Why |
|---|---|---|---|
| Window branches | Spec section 9's snippet groups the output of the process(...) parse step into windows | The windows re-parse the raw stream with mapValues and the same EventParser | process marks its output "key may have changed", so Kafka Streams would add aimbot-windows-repartition and speedhack-windows-repartition topics and push all ~24,000 events/s through the broker again. Found by printing topology.describe(). Guarded by TopologyShapeTest. |
| Kafka version | Spec assumed Kafka 4.1.x | 4.2.2 everywhere (images, Testcontainers, libraries) | Spring Boot 4.1.2 manages kafka-clients 4.2.2. Update the image tag and KAFKA_VERSION together if Boot changes. |
| Config validation | Spec section 6: "all counts, Hz, durations and thresholds are @Positive" | Checks in the compact constructors of SimulatorProperties, DetectionProperties, TopicsProperties | Hibernate Validator has no @Positive for Duration, some rules compare two fields, and common then needs no validation starter. |
| Extra simulator rules | Not listed | match-size ≥ 2, 1000 % tick-hz == 0, kill-chance-per-hit > 0, cheater-ratio + pro-ratio ≤ 1 | A kill needs a victim; dtMs must be a whole number; a zero chance means no kills ever; the roles must fit in the population. |
dtMs > 0 for POSITION | Not listed | Added to MatchEvent.isValid() | Speed is distance divided by time; zero would divide by zero. |
Segment and RunInfo types | Used by the spec but never defined | Plain enum and record added in common | The spec needs them. |
JsonSerdes.of(Class) | The snippet calls JsonSerdes.flagCandidate() and aimStats() | Common offers a generic factory; the detector uses it for its own types | Common cannot name types that live in the detector module. |
| Event ids | Spec rule 7: simulator output is a function of the seed | Name-based UUID of run, player and seq | No hidden randomness. |
| Population random stream | Shard k uses SplittableRandom(seed + k) | The population uses seed ^ 0x9E3779B97F4A7C15 | Otherwise population draws would replay shard 0's numbers. |
MatchRoom.PlayerTotals | Rule: every state value is a record | A small mutable class | A thread-confined accumulator, updated thousands of times a second. |
| Random bytes | "16 random bytes" | The first byte is forced to 0xFF | So the garbage can never accidentally be valid JSON. |
| Strict UUID check | "playerId is a UUID" | isUuid compares the parsed UUID back with the input | UUID.fromString accepts shortened forms like 1-1-1-1-1. |
| Failsafe | Not mentioned | Listed explicitly in the parent pom | Boot only manages its version, so verify silently skipped every *IT. |
| Jackson | Jackson 3 only | Jackson 3 in source; Jackson 2 still on the runtime classpath | kafka-streams brings Jackson 2 along. The rule is about source imports. com.fasterxml.jackson.annotation stays, because Jackson 3 keeps that package. |
| Verifier coverage | 85% per module | The check is vacuous for the verifier today | With no test run there is no coverage file, so JaCoCo skips the check. It became real in step 9: the verifier is at 100%. |
| Arena config | An ArenaProperties record with validation, never defined in full | Two records of that name, one per app, validated in their constructors (a blank service password fails startup when Arena is enabled) | The two apps share no module for it, and they need different keys (registration versus retries). |
finalSeq in the manifest | Not specified | The last seq generated, not the last acknowledged | A failed send then shows up as finalSeq > acked instead of hiding. |
| Rooms of a shard | "A worker reshuffles its own players into new rooms" | All rooms of one shard start and end together | Rooms that end at different moments would need a reshuffle rule; this one is trivial. |
telemetry.sim.autostart | Not in the spec | A @ConditionalOnProperty on SimulationEngine, default true | Tests and tools can have a Spring context without starting a 5-minute run, and no SimulatorProperties constructor call changes. |
| Registering players before the clock starts | Spec order: run id, start time, population, registration | Register first (using a throwaway population for the usernames), then start the clock | Registration can take a minute and cheat start times are offsets from the start time. |
Verifier exit code 4 and settle-poll-interval | Exit codes 0 to 3; a fixed 5 s re-read | Code 4 for bad arguments; the interval is a property | Bad arguments must not look like a result; tests poll in milliseconds. |
idIsGroup = false on the bridge listener | The spec's snippet only sets id | The listener id no longer doubles as the consumer group | Otherwise arena-bridge silently overrides the group id in the YAML. |
| Memory caps in Compose | No limits | mem_limit: 1g on the detector and simulator | Without them the stack stalled a 16 GB laptop until Docker stopped answering. |
| Chaos script | As in the spec | UTC Z on the --since time, MSYS_NO_PATHCONV=1 | Docker would read the time as local time; Git Bash would rewrite /opt/kafka/... paths. |
11. Interview preparation
Click a question to reveal a sample answer. The answers are in first person so you can adapt them. Every answer follows the same four beats, and you should too:
- Decision: what I chose.
- Reason: why, in one or two sentences.
- Trade-off: what it costs or what it gives up.
- How I checked: the specific test or run that backs it up.
The two pitches
The 30-second pitch
The 3-minute pitch
The problem. Online shooters have cheaters: aimbots that hit almost everything with inhuman reaction times, and speedhacks that move faster than the game allows. I wanted to build the part of the system that notices them within seconds, and to be able to prove how well it works. This is project two of three. Project one is a stats service called Arena, and project three is an AI "judge" that reads my suspect list and decides what to do. My detector only flags. It never bans.
The shape. A simulator makes a thousand fake players, a few of them secretly cheating, and writes their events to Kafka, keyed by player id so each player's events stay in order. A Kafka Streams detector reads them. It cuts each player's events into five-second windows that overlap, and checks simple rules, for example "at least 12 hits, an average reaction under 140 milliseconds, and a headshot rate of at least 62 percent". A flag is joined to the player's profile, de-duplicated with a ten-minute cooldown, and published to a topic.
The interesting engineering. First, exactly-once. Every event carries a per-player sequence number, so the detector can count how many arrived, how many arrived twice, and how many are missing. That turns "I used exactly-once" into something checkable. Second, a trap I found: Kafka Streams quietly added two internal topics that would have re-written every event through the broker. I found it by printing the topology, fixed it by parsing in a way that promises not to change the key, and wrote a test that fails if the topics come back. Third, I kept the detection logic free of Spring and injected the clock, so I can test "eleven minutes later" instantly.
Testing. There are 269 unit tests and 17 integration tests that start a real Kafka in Docker, all passing, with the 85 percent coverage gate met in all four modules. A chaos run is part of the evidence: I kill one of three detectors 90 seconds into a five-minute run.
Where it honestly stands. All twelve steps are built and measured on one laptop. Measured: about 24,900 events a second, 100 percent recall, 0.11 percent false positives, the median flag 3.9 seconds after an aimbot starts, and zero lost or duplicated events after the kill. Not met: failover took 14.4 seconds against a 10-second target, and the consumer lag peaked near 380,000 events after the kill instead of staying flat. And I tested the Arena integration only against a stub Arena, not a live one. I would rather tell you that than have you find it.
Design and architecture
1. Why Kafka Streams, rather than a hand-written consumer or something like Flink?
I chose Kafka Streams because the detector needs windows, state that survives a restart, joins and exactly-once, and Kafka Streams gives all of those as a library inside a normal Spring Boot app. There is no separate cluster to run. The trade-off is that I am tied to Kafka for both input and output, and I scale by running more copies of the app, not by handing work to a scheduler. I checked that the approach works with in-memory topology tests and two tests against a real broker. I did not benchmark it against Flink or a plain consumer, so I cannot claim it is the fastest option, only that it fit the problem and kept the deployment small.
2. Why is the detector's topology plain code with no Spring?
I wrote TelemetryTopology so that it only takes the settings, a Clock and a metrics interface, and builds the wiring diagram. Spring's only job is to hand those in, in a three-line class called StreamsWiring. The reason is testability: a test can build the whole topology in memory and run it in milliseconds, with the clock under its control. The cost is one extra interface, TopologyMetrics, so the topology does not depend on Micrometer. I checked it by the fact that all the topology tests, such as AimbotTopologyTest, run through TopologyFixture with no Spring context at all.
3. Why 5-second windows that advance every second, with a 2-second grace?
A flag needs enough evidence, 12 hits, and an aimbot lands roughly three to four hits a second, so it needs about three to four and a half seconds of cheating. A new window every second means that soon after the cheat starts there is a window made mostly of cheating shots. One long window would mix honest and cheating shots and react slower. The grace lets slightly late events still count. The trade-off is that every event lands in five windows, so there is five times the state updating, and one cheater trips several overlapping windows, which is why I needed a cooldown. I checked the thresholds at their exact boundaries in DetectionRulesTest and a full burst scenario in AimbotTopologyTest. The "three to four seconds" figure is arithmetic from the simulator settings, not a measurement.
4. Why is the Kafka key the player id, and what happens if a record's key does not match?
Kafka keeps records with the same key in the same partition, in order. Keying by player id gives me per-player ordering, which the sequence counting and the windows need, and it makes match-events and player-profiles co-partitioned, which the join needs. If a record's key is not its own playerId, that would break the ordering promise, so EventParser sends it to the dead-letter topic. The cost is that a producer bug becomes a dead letter rather than a quiet success. I checked it with DeadLetterTopologyTest.aKeyThatIsNotThePlayerIdGoesToTheDeadLetterTopic and AimbotTopologyTest.eventsKeyedByAnotherPlayerAreNotCountedEither. I also removed the check on purpose and three tests failed, so I know the tests really guard it.
5. Why re-emit every player's integrity stats every 5 seconds instead of only the ones that changed?
The reader of integrity-stats only needs the latest value per player, and the topic is compacted. After a failover the new instance has no memory of what changed recently, so "emit everything on a timer" is the simplest rule that is always right. The trade-off is repeated records when nothing changed. At 1,000 players that is small, but it would not scale to millions of players. I checked the behaviour with IntegrityTopologyTest.everyPlayerIsReEmittedOnEveryIntervalEvenWhenNothingChanged. I have not measured the actual output volume.
6. Why a ground-truth topic that the detector is not allowed to read, and a separate verifier?
If the detector could see who really cheats, its accuracy score would be meaningless. So the simulator writes the answer sheet to cheater-ground-truth and only the verifier reads it. The cost is one more program, which I built as a one-shot command-line app. I enforce the rule with TopologyShapeTest.neverMentionsTheGroundTruthTopic, which inspects the topology description and fails if that topic appears anywhere. The measured result from the verifier: 50 of 50 cheaters detected, one false positive among 950 honest and pro players.
The key technologies
7. What is exactly_once_v2, and how do you know it works?
I set Kafka Streams to exactly_once_v2, which makes "read input, update state, write output, record my position" one atomic Kafka transaction. The reason is that after a crash I do not want duplicates or lost events in the output. The trade-offs are some added commit latency and that readers of my topics must use read_committed; I have not measured the throughput cost. To check it I added the sequence counters, and ExactlyOnceIT sends 20 players times 500 events to a real broker and expects count 500, zero duplicates and zero gaps for each. It also re-sends one event on purpose and expects exactly one duplicate, so I know the check can fail. What I have not done is crash a detector mid-run. That proof is step 12.
8. Tell me about the repartition trap.
The spec's sample groups the output of process(...) into windows. Kafka Streams cannot know whether a custom processor changed the key, so it inserts a repartition topic before grouping. I printed topology.describe() and saw two extra internal topics, aimbot-windows-repartition and speedhack-windows-repartition. They would have pushed every shot and movement event through the broker a second time, around 24,000 a second by my arithmetic from the settings. My fix was to give the window branches their own parse using mapValues, which promises not to change the key, with the same EventParser so both paths accept exactly the same records. The cost is that each event is parsed twice, which is cheap but I have not measured it. TopologyShapeTest.addsNoRepartitionTopicInFrontOfTheWindows fails if the topics come back, and DetectorEndToEndIT checks the real broker's topic list.
9. Why read events as raw bytes and parse inside the topology, instead of a deserialization exception handler?
A deserialization handler lives in Kafka's own plumbing and can only skip or fail. I wanted to keep the bad record, the reason, its partition and offset, and its original bytes, in a dead-letter topic where an operator can see them. Parsing inside the topology makes a bad record just another branch. The cost is the double parse I mentioned. I checked it in DeadLetterTopologyTest, where garbage, wrong-key and invalid records all appear in the dead-letter output with the right reason, and good records after them still flow.
10. Why one standby replica and a 6-second session timeout, and what did you measure?
My target was to recover from a killed detector in under 10 seconds. Kafka's default session timeout of 45 seconds would make that impossible, so I used 6 seconds (the broker's minimum), and a standby replica keeps a warm copy of the state on another instance. I then killed one of three detectors with docker kill 90 seconds into a run and timed it: 14.4 seconds, and 16.7 seconds in the second run. So I missed the target. The logs show why: about 5 seconds until the group noticed the dead member, then three rebalances in a row as Kafka Streams moved tasks in steps. The session timeout is only part of the cost. What I would try next is tuning warm-up replicas and recovery lag, or Kafka 4.2's new Streams rebalance protocol. Nothing was lost during the failover, which the integrity check proved.
11. Why your own JsonSerde and Jackson 3?
Spring Boot 4 uses Jackson 3, and Spring Kafka's built-in serde is built on Jackson 2, so I wrote a small class, 67 lines, that is a serializer, a deserializer and a serde in one. It ignores unknown fields so a newer producer cannot break an older reader. The trade-off is one more class to maintain. Jackson 2 still arrives on the classpath through kafka-streams, so my rule is about what source files import, not what is on the classpath. I checked it with JsonSerdeTest, nine tests including unknown fields, null handling and failure wrapping.
12. Why check settings in a record's constructor instead of @Positive?
The validation library has no @Positive for Duration, and several rules compare two fields, for example "the window cannot advance by more than its size" or "players must be a multiple of match-size times shard-threads". So all rules live in the compact constructors, in one place, and the app refuses to start on a bad setting. The trade-off is hand-written checks instead of annotations. I checked it with DetectionPropertiesTest and SimulatorPropertiesTest, which try each broken rule and the exact boundary values.
Failure handling and reliability
13. What happens to a malformed event?
It never stops the detector. ParseProcessor sends null bytes, unreadable JSON, an invalid event, or a key that does not match the player id to match-events.DLT as a DeadLetter with the source topic, partition, offset, key, the original bytes as Base64, the error text capped at 500 characters, and a timestamp. The trade-off is that the dead-letter topic can grow, so it keeps data for 7 days. I checked it in DeadLetterTopologyTest and, with a real garbage record, in DetectorEndToEndIT, which decodes the Base64 payload and compares it with what was sent.
14. What happens if a detector instance dies?
There are two cases. If a stream thread dies, StreamsRuntimeConfig replaces it instead of letting the instance die, and logs the state changes. If a whole instance dies, the group notices after the 6-second session timeout, the partitions move to the other instances, and the standby replicas already hold the state. I tested that second case for real: scripts/chaos-kill.sh kills one of three replicas mid-run. The two survivors took over all 6 partitions in 14.4 seconds, the lag spiked to about 380,000 events and took roughly two minutes to drain, and the verifier then showed every one of 1,000 players with the right count, no duplicates and no gaps.
15. How do you stop one cheater from producing dozens of flags?
A hopping window re-fires on every new event once it crosses the threshold, and five windows overlap each event, so an aimbot would trigger constantly. CooldownProcessor keeps a store keyed by playerId|rule and drops a candidate if the last flag for that key was under 10 minutes ago. The key includes the rule, so an aimbot flag never hides a speedhack flag. The trade-off is that "now" is the real clock, as the spec says, not event time, so replaying old data would count real minutes. I checked it with the burst tests that jump the fake clock 30 seconds (silent) and 11 minutes (flags again), and with SpeedhackTopologyTest.anAimbotFlagDoesNotHideASpeedhackFlag.
16. Could a reader of suspicious-players ever see a flag that was rolled back?
Under exactly_once_v2, output is written inside a transaction. If it aborts, a reader using read_committed never sees it, but a reader using the default read_uncommitted could. So I made it a project rule that consumers of the detector's outputs read with read_committed, and my integration-test helpers do. The flag id is a random UUID, which is fine in the detector because the "everything is seeded" rule is about the simulator. I have not written a test that forces a transaction to abort, so this is a rule I follow, not something I have demonstrated.
Security and safety
17. How do you handle secrets?
The only secret is the password for Arena's service account, read from the environment variable TELEMETRY_SERVICE_PASSWORD. It defaults to blank in application.yml, .env is git-ignored and also excluded from the Docker build context by .dockerignore, and .env.example holds no real value. A blank password fails startup when Arena is enabled: the ArenaProperties constructor throws a message naming the variable, and ArenaPropertiesTest checks it. The player passwords the simulator registers are throwaway test values for fake accounts.
18. What are the known security gaps?
This is a local development project, and I would not expose it as is. The detector exposes the Actuator endpoints health, info, metrics and prometheus with no authentication, and there is no Spring Security in the module. The Kafka broker and Kafka UI use plain text with no login. A real deployment would need TLS and authentication on the broker, access rules on the topics, and protected management endpoints. None of that is built, and I would say so up front rather than let someone assume otherwise.
19. Is it safe to flag people automatically?
My design is that the detector only flags. It never bans. Each flag carries a score from 0.5 to 1.0 and the evidence numbers behind it, so a human, or the Judge in project three, can review it. The simulator includes 20 honest "pro" players on purpose, who sometimes look like a subtle aimbot, so the false-alarm case is exercised. In the measured run exactly one of the 20 pros was flagged (0.538 score, just over the line) and none of the 930 ordinary players: a false-positive rate of 0.11 percent overall, under my 1 percent target. The trade-off is that the one false positive is a real borderline player, which is exactly what the Judge must clear, and a single run is not a statistic.
Testing
20. How do you test logic that depends on time?
I never call the system clock in main code. Everything receives a java.time.Clock. In production that is the real clock. In tests it is MutableClock, and TopologyFixture.advanceTime moves both that clock and the Kafka Streams test driver's wall clock together. So "11 minutes later" takes no real time. The cost is discipline, since one stray System.currentTimeMillis() would break testability. The tests that rely on it are aSecondBurstInsideTheCooldownDoesNotFlagAgain and aBurstAfterTheCooldownFlagsAgain.
21. Why do your integration tests use random ids, and why do they not check exact evidence numbers?
All integration tests share one real broker per JVM. If two Streams apps used the same application id they would join the same consumer group and fight over partitions, so each test uses a random application id and a random state directory, and it filters its assertions by its own random player ids. They do not check exact evidence because production settings use a 64 MB cache and emit window results at commit time, so the first flag can come from a different window than in the in-memory test. The integration tests assert who is flagged and how many times. The exact numbers are pinned in the in-memory tests, where emission is immediate.
22. You have near-100% line coverage. Does that mean it is correct?
No. Coverage only says a line ran during some test. I use an 85 percent gate per module to catch untested code. A real example of the gap: every verifier unit test passed, but the verifier could not start in Docker, because VerifierRunner had two constructors and Spring cannot choose. The tests built the class by hand. I fixed it and added VerifierContextTest, which starts the whole application context. What makes me trust the tests is more specific than coverage: boundary tests at exactly 12 hits and 140 milliseconds, negative controls such as re-sending an event and expecting exactly one duplicate, and the fact that when I changed >= to > in aimbotTriggered on purpose, two test classes failed.
23. Tell me about the integration tests that were silently skipped.
Spring Boot's parent POM only manages the version of the failsafe plugin. It does not switch it on. So ./mvnw verify ran the unit tests and quietly skipped every class ending in IT. The fix, listing the plugin explicitly in the parent POM, went in with the commit that added the Kafka container smoke test. The lesson is that a green build can hide tests that did not run. I now check the log for the "Running ...IT" lines and the counts. On 18 November the full build ran all four integration tests.
24. How do you know a test can actually fail?
Two ways. First, a negative control inside the test: ExactlyOnceIT re-sends one event and expects exactly one duplicate, which proves the detector would notice one. Second, I deliberately break the code and watch the tests fail. Changing the min-hits comparison made two tests fail, and removing the key-mismatch check made three tests fail. I restored both files afterwards. It is a manual check, not an automated mutation-testing tool.
Java, language and code quality
25. Why are events and settings records, and why is ParseResult a sealed interface?
Records are immutable and have equals, toString and getters for free, so events, settings and stored state values cannot be changed behind your back. A sealed interface lists every allowed implementation, so ParseResult is exactly Parsed or Failed, and the compiler knows there is no third case. The trade-off is that a record is not a good fit for a hot mutable counter, so MatchRoom.PlayerTotals is a small mutable class, marked // DECISION in the code. The roundtrip tests in JsonSerdeTest cover the records.
26. Why constructor injection and a Clock parameter everywhere?
Constructor injection with private final fields means an object is complete the moment it exists, and a test can build one by hand without Spring. Passing a Clock instead of calling the system time is what makes the time-dependent tests possible. The cost is longer constructors. WiringUnitTest builds the Spring-layer classes directly with new, which is only possible because of this style.
27. What do you do when the specification is silent or wrong?
I choose the simplest option that keeps the frozen contracts (topic names, JSON fields, enum values, config keys), and I leave a comment starting // DECISION: that says what and why. There are 15 of them in the source, for example the extra dtMs > 0 validity rule and the 0xFF first byte in the garbage payload. When the spec was actually wrong, as with the repartition snippet, I changed the code, wrote it up in handoff.md, and added a test that guards the new behaviour. The trade-off is that the code and the spec now differ, which is why section 10 of this guide lists every difference.
Honesty questions
28. What is missing from this project?
Three things. First, I have not run the Arena integration against a real Arena: the bridge and the player registration are tested against a stub on a real broker, and I checked the request shapes against Arena's source, but a live end-to-end run needs Arena's Postgres and Redis. Second, the failover target: 14.4 seconds measured against under 10, and a lag spike afterwards because two instances have little headroom over 25,000 events a second. Third, everything was measured on one laptop with the load generator on the same machine, one run per configuration, so I would not quote the numbers as general. GitHub CI is also written but has not run yet.
29. Can you quote performance or accuracy numbers?
Yes, with the setup attached. On a Ryzen 5 3600 with 16 GB, Docker limited to 7.7 GB, three detectors capped at 1 GB each, 1,000 players for 5 minutes, seed 42: the simulator sustained about 24,900 events a second with zero failed sends. Detection: recall 100 percent (50 of 50), false-positive rate 0.11 percent (one pro player), median detection 3.9 seconds for aimbots and 0.9 seconds for speedhacks. Integrity after killing one detector: 7,466,329 events expected and counted, 0 duplicates, 0 gaps. Misses: failover 14.4 seconds, and lag well above the band I expected. A caveat I would give unprompted: one run per setting, so these are measurements, not statistics, and my "subtle" cheaters were still all caught, so a harder cheat mix might not look as good.
30. Tell me about a bug or a surprise.
The repartition trap is the best one, because the code worked and the tests passed, and the problem was invisible until I printed the topology and noticed topics I had not asked for. A smaller one: in Jackson 3, serializing new Object() no longer fails, so the obvious test for "serialization errors are wrapped" cannot fail. I had to use a class with a getter that throws. And the silently skipped integration tests. The common thread is that I now verify the structure and the test run itself, not just that the build is green.
31. What would you do differently?
I would build a thin version of the whole pipeline earlier, the simulator engine and a minimal verifier, so I had real end-to-end numbers sooner and could tune the thresholds against them, instead of finishing the detector in isolation. I would also look for a way to avoid parsing every event twice. And I would test my weakest assumption early: the subtle cheaters sit right next to the thresholds, so I would run a quick experiment on them before building anything around the 98 percent target.
31. You compared exactly-once with at-least-once. What did you find?
The same run, the same seed and the same kill, with only PROCESSING_GUARANTEE changed. Under exactly_once_v2 every player passed: 7,466,329 expected, 7,466,329 counted, 0 duplicates, 0 gaps. Under at_least_once: I ran it twice, once with the default settings and once with the state-store cache switched off, which is when a store write can run ahead of the offset commit. Both runs passed the integrity check: 7,465,795 and 7,467,529 events expected and counted, 0 duplicates and 0 gaps. So I could not reproduce the duplicates my plan predicted, and I cannot honestly claim exactly-once beat at-least-once here. What I can claim is that the exactly-once run is clean and that the integrity checker does detect a duplicate (ExactlyOnceIT re-sends one event and expects exactly one). The per-player sequence numbers are what make that visible: a total count could hide a duplicate and a gap cancelling out, but a sequence number cannot. I would add that one run is one run: with at-least-once, duplicates depend on when the instance dies relative to its commits.
32. How does the Arena bridge avoid losing or duplicating matches?
It does not try to be exactly-once itself. The listener is at-least-once: if the app dies after posting but before committing its offset, the match is posted again. Arena treats a match's externalId as idempotent, so the second post returns 200 and changes nothing, and the end result is exactly-once. Temporary failures (429, 5xx, connection errors) are retried with exponential backoff and then dead-lettered; permanent ones (400, 404, 409, 422) go straight to match-summaries.DLT. A 401 triggers one re-login. ArenaBridgeIT proves each path against a real broker: a 422 is one call, three 503s are exactly three calls, garbage JSON is zero calls.
33. Why test the Arena bridge against a stub instead of the real Arena?
A stub lets me script the failures a real Arena almost never produces on demand: two 503s then a 201, a 401 in the middle, a 422. It is a 100-line class on the JDK's own HTTP server with no extra dependency. The trade-off is that a stub only knows what I told it, so I also read Arena's source to copy the exact request and response shapes, including the problem JSON with code and detail. I have not run it against a live Arena, and I say so.
34. What did running the whole stack teach you that the tests did not?
Four things, all invisible to green builds. The verifier crashed on start inside Docker because a class had two constructors. The simulator moved to port 8082 because the shared .env sets SERVER_PORT for the detectors and an environment variable beats the YAML, so Prometheus could not scrape it. The first stack with no memory limits stalled my 16 GB laptop until Docker stopped answering, because each JVM sized its heap from the whole Docker VM. And docker compose down -v does not remove services behind a profile, so my second measured run nearly reused the previous run's finished simulator. Each is now fixed and written down, and the first one has a test.
35. Your lag did not stay flat. What happened?
Before the kill the per-partition maximum lag sat at 15,000 to 20,000, above the roughly 12,500 I expected. When I killed one of three detectors, the two survivors fell about 14 seconds behind (about 380,000 events) and had only roughly 5,000 events a second of headroom over the 25,000 a second coming in, so draining took about two minutes. I also know that Kafka's client lag metric is a windowed maximum that lags reality, so I checked the consumer group directly: its lag was 0 once the run ended. To fix the spike I would add stream threads or CPU per detector, and I would shorten the failover itself.
12. How to practise reading the code
Reading is not enough. For each exercise, write down your prediction first, then check. Exercises 2 and 3 change source files. Always undo them with git checkout -- <file> afterwards, and never commit the change.
Eight exercises
Exercise 1 (predict, then check): does a slow burst flag?
Read AimbotTopologyTest.shotsSpreadOverMoreThanAWindowDoNotAccumulate. It sends 15 hits, one every 500 ms, all headshots, 60 ms reaction. Predict: does it flag? Hint: how many shots can fit in one 5-second window?
Answer: No. At one shot per 500 ms a 5-second window holds at most 10 shots, and the rule needs 12 hits. Run it with ./mvnw -pl telemetry-detector -am test -Dtest=AimbotTopologyTest -Dsurefire.failIfNoSpecifiedTests=false. It passes.
Exercise 2 (break a rule): make the hit boundary exclusive
In DetectionRules.aimbotTriggered change s.hits() >= aim.minHits() to s.hits() > aim.minHits(). Predict which tests fail, then run ./mvnw -pl telemetry-detector -am test -Dtest='DetectionRulesTest,AimbotTopologyTest' -Dsurefire.failIfNoSpecifiedTests=false -Djacoco.skip=true.
Answer (I ran this on 18 November): two tests fail: DetectionRulesTest.aimbotNeedsTwelveHitsInclusive, and AimbotTopologyTest.anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence (the flag is now raised one shot later, so the evidence the test got was shots=13, hits=13, headshotHits=11, headshotRate=0.8462 instead of the expected 12, 12, 10, 0.8333; the window start and end it asserts on the lines before still matched). Undo with git checkout -- telemetry-detector/src/main/java/dev/fairplay/telemetry/detector/topology/DetectionRules.java.
Exercise 3 (break a rule): remove the key check
In EventParser.parse, make the condition if (!event.playerId().equals(key)) never true (for example, replace it with if (false)). Predict the failing tests, then run -Dtest='DeadLetterTopologyTest,AimbotTopologyTest'.
Answer (I ran this on 18 November): three tests fail: DeadLetterTopologyTest.aKeyThatIsNotThePlayerIdGoesToTheDeadLetterTopic, DeadLetterTopologyTest.validEventsAfterBadOnesAreStillProcessedAndNeverDeadLettered, and AimbotTopologyTest.eventsKeyedByAnotherPlayerAreNotCountedEither. The third shows that the same predicate protects both the dead-letter path and the window path. Undo with git checkout.
Exercise 4 (change a setting): lower min-hits in the yml
Change min-hits: 12 to 11 in telemetry-detector/src/main/resources/application.yml. Predict which test class notices, and which one does not, even though it is about the same rule. Hint: look at what TestDetection.defaults() is, and at DetectionPropertiesTest.applicationYmlDefaultsBindToTheSpecValues.
Answer (from reading the tests, I did not run this one): DetectionPropertiesTest.applicationYmlDefaultsBindToTheSpecValues fails, because it compares the yml with TestDetection.defaults(), which still says 12. AimbotTopologyTest keeps passing, because it builds its settings from TestDetection.defaults() and never reads the yml.
Exercise 5 (recompute by hand): two scores
(a) The worked example in section 9 with the reaction changed from 60 ms to 100 ms: what is the score? (b) A speedhack window with 60 moving samples, 33 of them over the limit: what is the score?
Answer (hand arithmetic, checked with a calculator): (a) rate = 10/12 = 0.8333, so 0.5 + 0.25 × (0.8333 − 0.62)/0.33 + 0.25 × (140 − 100)/110 = 0.5 + 0.1616 + 0.0909 = 0.7525, rounded half-up to 0.753. (b) ratio = 33/60 = 0.55, so 0.5 + 0.5 × (0.55 − 0.15)/0.85 = 0.5 + 0.2353 = 0.7353, rounded to 0.735. To verify (a) in code, change the 60s in anObviousAimbotBurst... to 100 and expect Evidence.aimbot(12, 12, 10, 0.8333, 100.0).
Exercise 6 (recompute by hand): the integrity counters
A player's events arrive with seq 1, 2, 3, 3, 7. Apply the three rules of IntegrityProcessor.apply starting from all zeros. What are count, lastSeq, duplicates and gaps?
Answer: seq 1, 2, 3 are each "last + 1": count 3, lastSeq 3. The second 3 is "at or below last": duplicates 1, count unchanged. Seq 7 is above last + 1: gaps += 7 − 3 − 1 = 3, count 4, lastSeq 7. Result: count 4, lastSeq 7, duplicates 1, gaps 3. Compare with IntegrityTopologyTest.jumpingToSequenceOneHundredFiveCountsFourGaps.
Exercise 7 (draw a sequence diagram): one garbage record
On paper, draw the path of the 16 random bytes from MalformedInjector through the detector, with these lanes: Kafka match-events, ParseProcessor, EventParser, the split, the dead-letter topic, and the window branch.
Answer: The bytes arrive keyed by a player id. ParseProcessor.process calls EventParser.parse, whose readValue throws, so it returns Outcome(null, message). The processor calls metrics.deadLettered(), builds a DeadLetter from context.recordMetadata() and forwards ParseResult.Failed. The split sends it to the parse-failed branch, mapValues unwraps the DeadLetter, and it is written to match-events.DLT. Separately, the window branch calls EventParser.parse again, gets a null event, and filter drops it. It never reaches the integrity processor or any window.
Exercise 8 (explain aloud): the repartition trap in 60 seconds
Close the guide. Out loud, explain: what the spec's snippet did, what Kafka Streams added, why, what you changed, what it costs, and which two tests guard it. Then do the 30-second pitch from section 11. Record yourself and check that every number you quote is one of the measured ones in section 5g, with its caveats.
Checklist: process output is treated as possibly key-changing; two extra internal topics; every event rewritten through the broker; fix is a re-parse with mapValues; cost is a double parse; guards are TopologyShapeTest.addsNoRepartitionTopicInFrontOfTheWindows and DetectorEndToEndIT.
A 5-step method for reading any unfamiliar file
- Read the class comment. This project writes one sentence about each class's job above it. Say it back in your own words.
- Find the inputs and outputs. Look at the constructor and the main method (
process,parse,tick). What goes in, what comes out, what does the class remember? - Find the state. Fields, and for Kafka Streams the state stores it names (
seq-tracker,flag-cooldown). State is where surprises live. - Follow one concrete value. Pick a number, such as seq 105 after seq 100, and trace it line by line, writing the values down.
- Read its test last, and as a promise. Each test name is a sentence about what the class guarantees. If a behaviour you found is in no test name, it is untested.
13. Glossary
| Term | Meaning |
|---|---|
| Aimbot | Cheat software that aims for the player: very high hit rate, many headshots, extremely fast reaction. |
| At-least-once | A delivery guarantee that never loses data but may repeat it. Contrast with exactly-once. |
| Broker | A Kafka server. |
| Co-partitioned | Two topics with the same number of partitions and the same key, so matching keys sit in matching partitions and can be joined. |
| Compaction | Kafka keeping only the newest record per key. |
| Consumer group | A set of program instances that share the partitions of a topic between them. |
| Cooldown | A quiet period after a flag during which the same player and rule are not flagged again. |
| DLT / dead letter | A topic that holds records that could not be processed, plus the reason. |
| DTO | Data Transfer Object: a plain object that just carries data. |
| Epoch milliseconds | Milliseconds since 1 January 1970 UTC. |
| Event time | When something happened in the game, as opposed to when we processed it. |
| Failover | Another instance taking over when one dies. |
| False positive | An innocent player wrongly flagged. |
| Grace period | How late (in event time) an event may arrive and still be counted in a window. |
| Hopping window | A fixed-length time window that starts at regular steps shorter than its length, so windows overlap. |
| Idempotent | Doing it twice has the same effect as once. |
| Integration test (IT) | A test that uses real external pieces, here a real Kafka in Docker. |
| JSON | A text format for structured data. |
| KRaft | Kafka's built-in coordination mode that replaced ZooKeeper. |
| KStream / KTable | Kafka Streams' view of a topic as a stream of facts / as a latest-value table. |
| Lag | How far a reader is behind the newest record of a topic. |
| Offset | A record's position number within a partition. |
| Partition | One ordered lane of a topic. |
| Precision / recall | Of the players flagged, how many were truly cheating / of all true cheaters, how many were flagged. |
| Processor | Custom Kafka Streams code handling one record at a time. |
| Record (Java) | An immutable data-holding class defined in one line. |
| Record (Kafka) | One entry in a topic: key, value, timestamp. |
| Repartition | Re-writing a stream through an internal topic so its records are grouped by a (new) key. |
| Seed | The starting number of a random generator; same seed, same sequence. |
| Serde | A serializer and deserializer pair. |
| Sealed interface | A Java interface that lists all its allowed implementations. |
| Speedhack | Cheat that moves the character faster than the game allows. |
| Standby replica | A warm copy of a task's state on another instance, for fast failover. |
| State store | A local database that a stream processor uses to remember things, backed up in Kafka. |
| Stream time | The latest event timestamp a stream task has seen. |
| Testcontainers | A library that starts Docker containers (like Kafka) for tests. |
| Topology | The complete processing graph of a Kafka Streams application. |
| UUID | A 128-bit identifier written like 3f2a9c10-.... |
| Wall-clock time | The real clock, as opposed to stream time. |
More terms from the later sections
| Term | Meaning |
|---|---|
| Actuator | The Spring Boot add-on that serves /actuator/health, /actuator/metrics and similar status pages. |
| Arrange / act / assert | The three parts of a test: set things up, do the thing, check the result. |
| AssertJ | A test library with readable checks such as assertThat(x).isEqualTo(y). |
| Awaitility | A test library that retries a check until it passes or a time limit runs out. |
| Bean | An object that Spring creates and manages for you. |
| Changelog topic | The Kafka topic where a state store's changes are backed up, so another instance can rebuild it. |
| Commit interval | How often Kafka Streams saves its progress and flushes cached output (500 ms here). |
| Compact constructor | A record constructor with no parameter list, used to check or adjust values. |
| Dependency injection | Passing an object what it needs (here, through its constructor) instead of letting it create it. |
| Failsafe / Surefire | The Maven plugins that run integration tests (*IT) and unit tests. |
| JaCoCo | The tool that measures which lines the tests executed. |
| Jackson | The Java library that turns objects into JSON and back (version 3 here). |
| Micrometer | The metrics library behind Spring Boot's counters; Prometheus can read it. |
| Negative control | A deliberate wrong input in a test, to prove the check can actually fail. |
| Prometheus / Grafana | A metrics store and a dashboard tool. Both run in the Compose stack; Prometheus scrapes the apps every 5 seconds and Grafana draws the "Fairplay Telemetry" dashboard from it. |
| Punctuator | A timer inside a Kafka Streams processor (context.schedule). |
read_committed | A consumer setting that hides records from transactions that were aborted. |
| RocksDB | The small on-disk database Kafka Streams uses for state stores. |
| Test-jar | A jar of one module's test helpers, shared with other modules' tests. |
| TopologyTestDriver | A Kafka Streams test tool that runs a topology in memory, with controllable time. |
| Transaction (Kafka) | A group of reads, writes and offset commits that all succeed or all vanish. |