Testing
Your projects have more than 700 automated tests, coverage gates of 85% per module, real databases and brokers in containers, a fake LLM, fault injection and a negative control. Interviewers rarely see junior candidates who test like this. This module explains the tools (JUnit 5, AssertJ, Mockito, MockMvc, Testcontainers, WireMock, TopologyTestDriver, Awaitility, JaCoCo) and, more importantly, the strategy behind which test goes where.
By the end you can answer
- "How do you decide between a unit test and an integration test?"
- "Why Testcontainers instead of H2 or embedded Kafka?"
- "Mock, stub, fake: what's the difference? When shouldn't you mock?"
- "How do you test time-dependent or asynchronous code without flakiness?"
- "How do you test code that calls an LLM?"
- "What does coverage tell you, and what doesn't it?"
- "Surefire vs Failsafe?"
1. Your test suites in numbers
| Project | Tests (last full run) | Line coverage | Notes |
|---|---|---|---|
| Arena | 71 | 89.6% (gate 85%) | A small test-clock fix landed after that run; it passes alone, but the full suite hasn't been re-run since. Re-run before quoting "all green". |
| Telemetry | 269 unit + 17 integration | 100% common, 99.2% simulator, 100% detector, 100% verifier (gate 85% per module) | All passing on 2026-11-19 |
| Judge | judge-core 194; service 77 + 12 IT; eval 105 + 1 IT | 97.2% / 96.9% / 96.2% (gate 85% per module) | All passing, with no API key needed |
2. Strategy: the test pyramid, applied
▲ few, slow, realistic
/ \ end-to-end: chaos run, measured benchmarks, the live smoke test (manual)
/───\
/ \ integration: Spring context + real Postgres/Redis/Kafka/WireMock in containers
/───────\
/ \ component: TopologyTestDriver (whole topology, in memory), scripted LLM gateway
/───────────\
/ \ unit: pure logic (state machine, rules, policy, calculators), milliseconds each
▼ many, fast, precise
The guiding principle in all three projects is to put the logic that matters in pure classes, so it can be tested exhaustively and fast, and use a smaller number of integration tests to prove the wiring and real-dependency behaviour.
| Layer | Arena | Telemetry | Judge |
|---|---|---|---|
| Unit | AccountStateMachineTest (all 35 pairs), RatingCalculatorTest, StatDistributionServiceTest, TokenServiceTest | Detection rules, window stats, the validator, property validation, seeded simulator parts, verifier math | Policy table, escalation rule, validator, cost calculator, tool guard, memoizing executor |
| Component | – | TopologyTestDriver tests via TopologyFixture | AgentLoopTest etc. with ScriptedGateway |
| Integration | MockMvc + Testcontainers (Postgres 17, Redis 7): auth, security, matches, leaderboard, moderation, appeals, audit trigger | *IT on apache/kafka-native: simulator end to end, detector end to end, ExactlyOnceIT, verifier reader, Arena bridge vs a stub Arena | JudgeFlowIT, ProcessedFlagsIT, ResilienceIT on Kafka + a WireMock container |
3. JUnit 5 essentials
@Testmarks a test; JUnit creates a new instance of the test class per test method, so fields don't leak between tests.@BeforeEach/@AfterEachrun around every test (Arena'sresetState()truncates tables; Telemetry's fixture is closed in@AfterEach).@BeforeAll/@AfterAllrun once per class.- Parameterised tests run one test body over many inputs:
arena/.../moderation/AccountStateMachineTest.java
It generates every combination, so every illegal transition is proven illegal too, not just the legal ones. "Exhaustive" is a powerful word in an interview, and here it's literally true.static Stream<TransitionCase> allPairs() { // 5 statuses × 7 actions = 35 cases return Stream.of(AccountStatus.values()) .flatMap(status -> Stream.of(ModerationActionType.values()) .map(action -> new TransitionCase(status, action, VALID_TRANSITIONS.get(new TransitionCaseKey(status, action))))); } @ParameterizedTest(name = "{0}") @MethodSource("allPairs") void testsAll35ActionStatusPairs(TransitionCase tc) { if (tc.expectedTo() != null) { assertThat(stateMachine.next(tc.from(), tc.action())).isEqualTo(tc.expectedTo()); } else { assertThatThrownBy(() -> stateMachine.next(tc.from(), tc.action())) .isInstanceOf(ApiException.class) .satisfies(ex -> assertThat(((ApiException) ex).getErrorCode()) .isEqualTo(ErrorCode.INVALID_STATE_TRANSITION)); } } - Tags:
@Tag("live")onSdkSmokeTest; the Judge's POM excludes thelivegroup by default (test.excludedGroups), so the one test that calls the real API only runs on purpose.
4. AssertJ: assertions that explain themselves
assertThat(flags).hasSize(1);
assertThat(flag.rule()).isEqualTo(Rule.AIMBOT);
assertThat(result.stats().stopReasons()).containsExactly("tool_use", "end_turn");
assertThat(tools.calls()).singleElement().satisfies(call -> {
assertThat(call.toolName()).isEqualTo("get_stat_percentiles");
});
assertThatThrownBy(() -> jdbc.update("UPDATE moderation_actions SET reason = 'tampered' WHERE id = ?", id))
.isInstanceOf(DataAccessException.class)
.hasMessageContaining("moderation_actions is append-only");
Fluent assertions read like sentences and produce precise failure messages ("expected size 1 but was 2: [...]"). assertThatThrownBy tests that something fails correctly, which matters as much as testing that it succeeds.
5. Test doubles: mocks, stubs and fakes
| Double | What it is | Your examples |
|---|---|---|
| Stub | Returns canned answers | WireMock JSON mappings (final-ban.json, tool-use-turn.json); StubArenaServer (a fake Arena on the JDK HttpServer) |
| Fake | A working lightweight implementation | ScriptedGateway (plays a script of model responses), FakeToolExecutor, MutableClock, RecordedToolExecutor |
| Mock | Records interactions so you can verify calls | Mockito mocks in some unit tests; @MockitoBean to replace a bean in a Spring context |
| Spy | Wraps a real object and records calls | Rarely needed |
// Mockito in one glance
ArenaClient arena = mock(ArenaClient.class);
when(arena.player(id)).thenReturn(Optional.of(playerJson)); // stub behaviour
service.handle(flag);
verify(arena).applyAction(eq(id), any()); // verify interaction
- Don't mock what you don't own (the Anthropic SDK, a Kafka consumer). Your mock encodes your assumptions about the library, which may be wrong. Put an interface you own in front of it (
MessagesGateway) and fake that, then test the real adapter against a realistic server (WireMock). - Real story from the Judge: Spring Kafka's
DeadLetterPublishingRecovererasks the consumer for the target topic's partition count. A Mockito-mocked consumer returns an empty list, so the dead-lettered record lost its partition, a "bug" that existed only in the mock world. That's the trap in one sentence. - Prefer fakes with real behaviour for time (
MutableClock) and for your own boundaries; prefer real dependencies in containers for databases and brokers.
6. Testing Spring applications
@SpringBootTeststarts the real application context. With@AutoConfigureMockMvcyou get aMockMvcthat sends requests through the whole Spring MVC and Security stack without opening a network port.- Context caching: Spring reuses one context across test classes with the same configuration, which keeps suites fast. That's a reason Arena puts the configuration in one
AbstractIntegrationTestbase class.@DirtiesContextforces a fresh context (Telemetry's detector ITs useAFTER_CLASSbecause each starts its own Streams app). @ServiceConnectionon a Testcontainers field tells Boot to wire the datasource or Redis URL from the container automatically. No hand-written property plumbing.- Boot 4:
@MockitoBean(not@MockBean), andAutoConfigureMockMvcmoved toorg.springframework.boot.webmvc.test.autoconfigure.
mvc.perform(post("/api/v1/matches") ... ) // as the SERVICE account, valid body
.andExpect(status().isCreated())
.andExpect(jsonPath("$.externalId").value("sim-2026-11-17-0001"))
.andExpect(jsonPath("$.participants[0].ratingBefore").value(1000))
.andExpect(jsonPath("$.participants[0].ratingAfter").value(1030))
.andExpect(jsonPath("$.participants[1].ratingAfter").value(970));
mvc.perform(post("/api/v1/matches") ... ) // the same externalId again
.andExpect(status().isOk()) // idempotent replay
.andExpect(jsonPath("$.participants[0].ratingAfter").value(1030)); // not applied twice
@ServiceConnection
static final PostgreSQLContainer postgres = new PostgreSQLContainer("postgres:17-alpine");
@ServiceConnection(name = "redis")
static final GenericContainer<?> redis = new GenericContainer<>("redis:7-alpine").withExposedPorts(6379);
static { postgres.start(); redis.start(); } // once per JVM: the "singleton container" pattern
@BeforeEach
void resetState() {
jdbc.execute("TRUNCATE appeals, moderation_actions, match_participants, matches, refresh_tokens, players CASCADE");
redisTemplate.execute((RedisCallback<Void>) c -> { c.serverCommands().flushAll(); return null; });
}
Containers start once for all test classes (a static initialiser), which is much faster than per class. Isolation comes from wiping data before each test, so tests can run in any order.
7. Testcontainers: real dependencies, disposable
Testcontainers starts throwaway Docker containers from test code. Your suites use PostgreSQL 17, Redis 7, Kafka (apache/kafka-native, which starts fast) and WireMock 3.13.2.
Why not H2, an embedded Redis or @EmbeddedKafka?
Because "almost the same" hides exactly the bugs you care about. Arena depends on PostgreSQL-specific behaviour that H2 doesn't reproduce faithfully: the PL/pgSQL trigger, percentile_cont, FOR UPDATE locking, JSONB, COLLATE "C", JOIN LATERAL, transaction-abort semantics. Testing against the production engine and version (the same Flyway migrations too) means a green build means something. Telemetry's rules explicitly forbid @EmbeddedKafka in favour of a real broker image. The cost is that Docker must be running and the suite takes about 2 minutes.
Isolation on a shared broker
Telemetry's detector ITs share one Kafka broker per JVM, so each test class uses a random application.id, a random state.dir, a short commit.interval.ms (100 ms) and its own random player ids, and filters assertions by those ids. Otherwise two Streams apps would join the same consumer group and steal each other's partitions.
8. WireMock: faking HTTP services, including the LLM
- WireMock is an HTTP server you program with request matchers and responses. The Judge runs it as a container (no WireMock Java dependency) that impersonates both the Claude API and Arena.
- The Anthropic client's base URL comes from Spring's
Environment(ANTHROPIC_BASE_URL), so a test can point it at the container's random port. The SDK'sfromEnv()couldn't do that, which is whyAnthropicConfigdeviates from the spec (a documented decision). - Canned model responses live in
src/test/resources/wiremock/(tool-use-turn.json,final-ban.json,final-clear.json). Stateful scenarios ("fail twice, then succeed") driveResilienceIT's fault table (module 12). - Honest limit: stubs model the API as you believe it behaves. Only the live smoke test can confirm the real request shape, and it hasn't been run.
9. TopologyTestDriver (recap from module 10)
Runs a Kafka Streams topology synchronously in memory: pipe in records with exact timestamps, advance wall-clock time by hand, read output topics, with no broker or threads. Combined with MutableClock, window boundaries, grace periods, joins, cooldowns and punctuators are tested to the millisecond. Shape tests on topology.describe() guard architecture rules (no window repartition topics; the ground-truth topic never used).
10. Asynchronous code: Awaitility, never Thread.sleep
// judge-service JudgeFlowIT (shape)
Awaitility.await().atMost(WAIT).pollInterval(200, TimeUnit.MILLISECONDS)
.untilAsserted(() -> assertThat(readVerdict(flagId)).isPresent());
With Kafka, the result appears "soon". Thread.sleep(5000) is both slow (always waits the full time) and flaky (sometimes 5 s isn't enough on a busy CI machine). Awaitility polls a condition until it holds or a timeout expires: fast when things are fast, patient when they're slow, with a clear failure message.
11. Making tests deterministic
- Time: inject a
Clock; tests use fixed or mutable clocks. (Arena'sTokenServiceTestbug: the JWT decoder validated expiry against the system clock instead of the test's fixed clock, so the test failed once the fixed date was in the past. Fixed by giving the decoder the test clock.) - Randomness: seed it. The simulator with seed 42 produces the same population and cheaters every run.
- Data: unique ids per test; reset state before each test.
- Concurrency: await conditions; don't sleep.
- External services: containers with pinned image versions, never shared environments.
12. Writing good tests
- Arrange, act, assert (or given, when, then), with one behaviour per test.
- Names that state the behaviour:
anObviousAimbotBurstProducesExactlyOneFlagWithCorrectEvidence,anHonestStreamForAMinuteNeverFlags,databaseRejectsUpdateAndDeleteOnModerationActions. A failing test's name should tell you what broke. - Test behaviour through public contracts, not private internals, so refactoring doesn't break tests.
- Table-driven tests for rule tables: the state machine (35 pairs),
DecisionPolicy(each row). - Test the failure paths: the audit trigger rejects tampering; a 400 goes to the DLT; an outage holds flags; a duplicate
externalIdreturns 200. - Negative controls: a checker that never fails proves nothing.
ExactlyOnceITdeliberately re-sends one event and asserts exactly one duplicate is detected, proving the integrity check can catch what it claims to catch. This idea impresses interviewers. - Test the wiring at least once:
VerifierContextTeststarts the whole verifier context. Unit tests passed at 100% coverage while the app couldn't start (two constructors).
13. Coverage with JaCoCo
- The JaCoCo Maven plugin attaches an agent (
prepare-agent) that instruments bytecode while tests run, writes a report (target/site/jacoco) and enforces rules incheckduringverify. - All three builds fail below 85% line coverage (per module in the multi-module builds). Excluded:
*Applicationclasses and generated*MapperImpl. - Line coverage: was each line executed? Branch coverage: was each side of each
iftaken? Branch is stricter. - What coverage cannot tell you: whether the assertions are meaningful, whether the wiring works, whether behaviour is correct. Telemetry's README says it best: "Coverage only says a line ran." A test with no assertions can reach 100%. Use coverage to find untested code, not to prove quality. Mutation testing (for example PIT) is the next level: it changes your code and checks that some test fails.
14. Surefire vs Failsafe, and CI
| Surefire | Failsafe | |
|---|---|---|
| Runs | Unit tests: *Test, *Tests | Integration tests: *IT |
| Phase | test | integration-test, then checks results in verify |
| On failure | Stops the build immediately | Lets post-integration-test cleanup run, then fails at verify |
| Your use | Telemetry and Judge: ./mvnw test needs no Docker | Telemetry and Judge *IT classes; need Docker |
Arena names its integration tests *IntegrationTest (ending in "Test"), so Surefire runs them and Arena's ./mvnw test needs Docker. Splitting into *IT gives a fast inner loop: unit tests in seconds, the full suite before pushing.
CI: each project's GitHub Actions workflow runs ./mvnw -B verify on ubuntu-latest with Temurin 21 and Maven caching. GitHub's runners have Docker, so Testcontainers works in CI, and Arena uploads the JaCoCo report as an artifact even when the build fails (if: always()).
15. Interview questions
How do you decide what to unit test vs integration test? Basic
Put the important rules in pure classes (the state machine, rating calculator, detection rules, decision policy) and unit test them exhaustively: they're fast and precise. Use integration tests for what only exists when components meet real infrastructure: SQL and triggers, locking, security filters, serialisation, Kafka transactions, wiring. Then a few end-to-end checks (the chaos run, benchmarks) for system properties.
Why Testcontainers instead of H2 or embedded Kafka? Intermediate
Fidelity. Arena relies on PostgreSQL features (a PL/pgSQL trigger, percentile_cont, FOR UPDATE, JSONB, COLLATE "C", JOIN LATERAL, transaction-abort behaviour) that an in-memory database emulates poorly or not at all. Running the same engine version and the same Flyway migrations as production means a passing test actually proves something. Kafka transactions and read_committed behaviour likewise need a real broker. The cost is Docker and a couple of minutes; I keep containers as JVM-wide singletons to limit that.
How do you keep integration tests isolated when they share containers? Intermediate
In Arena, a @BeforeEach truncates every table and flushes Redis, so each test starts empty. For Kafka, where you can't easily wipe a broker, each test class uses random application ids, state directories and player ids, and filters its assertions by them, so concurrent Streams apps don't share consumer groups or data.
How do you test code that calls an LLM? Deep
Put an interface I own (MessagesGateway) between my logic and the SDK. Unit tests use a ScriptedGateway that plays back a scripted sequence of responses (a tool call, then a verdict; an invalid stop reason; too many tools), so the loop, routing, validation and policy are fully deterministic. The real resilient gateway is tested against the real SDK and a local stub server. Integration tests run a WireMock container impersonating the Claude API and Arena, including fault scenarios. One live-tagged smoke test, excluded by default, checks the real API. Quality of the model's judgement is measured separately by the evaluation harness, not by unit tests.
What's the difference between a mock and a fake? When do you avoid mocks? Intermediate
A mock records interactions so you can verify calls; a fake is a working lightweight implementation, like my MutableClock or the scripted gateway. I avoid mocking types I don't own, because the mock encodes my assumptions about the library. I hit exactly that: a mocked Kafka consumer returned no partition info, so Spring's dead-letter recoverer lost the record's partition, a behaviour that only existed with the mock. I prefer fakes at my own boundaries and real dependencies in containers.
How do you test time-based logic like cooldowns or token expiry? Basic
Inject a java.time.Clock everywhere and use a fixed or mutable clock in tests: advance it 10 minutes to test the cooldown, or set it to exactly the expiry instant for tokens. Kafka Streams tests also control record timestamps and advance wall-clock time on TopologyTestDriver. Never Thread.sleep to make time pass.
Your coverage is 100%. Does that mean the code is correct? Intermediate
No. Coverage says which lines ran, not whether behaviour was checked. Telemetry's verifier had 100% line coverage while the application couldn't even start, because Spring couldn't choose between two constructors and no unit test started the context. I added a context test. Coverage finds untested code; assertions, negative controls and mutation testing tell you whether the tests are any good.
What is a negative control and where did you use one? Deep
A test that deliberately introduces the fault a checker is supposed to detect, to prove the checker can fail. My exactly-once verification counts duplicates per player; if it reported zero duplicates because of a bug, every run would look perfect. ExactlyOnceIT re-sends one event on purpose and asserts that exactly one duplicate is reported. Only then does "0 duplicates" in the real run mean something.
How do you test asynchronous Kafka flows without flakiness? Intermediate
Awaitility: poll a condition with a timeout instead of sleeping, so tests are fast when the system is fast and tolerant when CI is slow. Also read_committed consumers in tests, short commit intervals, unique ids, and assertions that don't depend on timing details, such as which window emits first under caching.
Surefire vs Failsafe? Basic
Surefire runs unit tests in the test phase and stops at the first failing module. Failsafe runs *IT integration tests in integration-test, lets cleanup run even on failure, and fails the build in verify. Splitting them means ./mvnw test is fast and needs no Docker, while ./mvnw verify runs everything, including the JaCoCo gate.
Why does your role-check test send a valid request body? Curveball
Because Spring validates @Valid @RequestBody during argument resolution, before @PreAuthorize's method interceptor runs. An invalid body returns 400 before the role check, so a test expecting 403 must send a valid body. Tests have to understand the framework's pipeline, not just the business rule.
16. Quick quiz
1. Which test double is MutableClock?
A working lightweight implementation.
2. Why avoid Thread.sleep in async tests?
Use Awaitility's polling with a timeout.
3. Failsafe runs which classes by default?
Surefire takes *Test and *Tests.
4. A suite has 100% line coverage. You know for sure that:
Nothing more.
5. Why does Arena run PostgreSQL in a container for tests?
Fidelity to production behaviour.
6. ExactlyOnceIT re-sends one event on purpose. That's a:
It proves the checker can detect the fault.
17. Cheat sheet
- Strategy: logic in pure classes, tested exhaustively; integration tests for SQL, security, Kafka and wiring; a few end-to-end checks.
- JUnit 5: a new instance per test;
@BeforeEach;@ParameterizedTest+@MethodSource(35 pairs);@Tag("live")excluded by default. - AssertJ:
assertThat,containsExactly,singleElement().satisfies,assertThatThrownBy. - Doubles: stub (WireMock, StubArenaServer), fake (ScriptedGateway, MutableClock), mock (Mockito,
@MockitoBean). Don't mock what you don't own. - Spring:
@SpringBootTest+ MockMvc +jsonPath; context caching;@ServiceConnection;@DirtiesContextfor Streams ITs. - Testcontainers: Postgres 17, Redis 7, kafka-native, WireMock 3.13.2; singleton containers; TRUNCATE + FLUSHALL; random ids, app ids and state dirs.
- Determinism: injected Clock, seeded randomness, Awaitility, unique data.
- Good tests: behaviour names, one behaviour each, failure paths, table-driven, negative controls, at least one context test.
- JaCoCo: 85% line gate per module; line vs branch; "coverage only says a line ran".
- Surefire (
*Test, test phase) vs Failsafe (*IT, verify); CI runs./mvnw -B verify.