Maven, Docker, CI and observability
Code is only useful if it builds reproducibly, runs anywhere, is checked on every push and can be watched in production. This module covers the delivery side of your projects: Maven (lifecycle, dependencies, multi-module builds, plugins), Docker and Compose, GitHub Actions, and observability with Micrometer, Prometheus and Grafana. It also covers how to load-test with k6 and read percentiles honestly.
By the end you can answer
- "What happens when you run
./mvnw verify?" - "What are dependency scopes? What does the Spring Boot parent POM give you?"
- "Why multi-module? What does
-pl ... -amdo?" - "Walk me through your Dockerfile. Why multi-stage? Why a non-root user?"
- "How do services find each other in Docker Compose? What are healthchecks and profiles for?"
- "Counter vs gauge vs timer? Why percentiles instead of averages?"
- "How did you run your benchmark, and what are its limitations?"
1. Maven: the build tool
Maven builds Java projects by convention: sources in src/main/java, resources in src/main/resources, tests in src/test/java, output in target/. The pom.xml (Project Object Model) declares the project's coordinates (groupId:artifactId:version, such as dev.fairplay:arena:0.0.1-SNAPSHOT; SNAPSHOT means a work in progress), its dependencies and its plugins.
The lifecycle
validate → compile → test → package → integration-test → verify → install → deploy
│ │ │ │ │
javac + Surefire fat jar Failsafe (*IT) JaCoCo check (≥ 85%),
annotation (unit) (Boot against Failsafe results
processors plugin) containers
Running a phase runs every phase before it. So ./mvnw verify compiles, runs unit tests, packages, runs integration tests and enforces coverage: the full gate CI uses. ./mvnw test stops after unit tests. Plugins attach their goals to phases (JaCoCo's check to verify, for example).
Dependencies and scopes
| Scope | On the classpath for | Your examples |
|---|---|---|
compile (default) | Compiling, testing and running | Spring starters, MapStruct, the Anthropic SDK |
runtime | Running and testing, not compiling | The PostgreSQL JDBC driver, micrometer-registry-prometheus: code only talks to interfaces (JDBC, Micrometer), so it can't accidentally depend on the implementation |
test | Tests only | Testcontainers, Awaitility, the *-test starters |
optional | Not passed on to projects depending on yours | Lombok (a compile-time tool) |
Transitive dependencies: you get your dependencies' dependencies automatically, which is convenient and is where version conflicts come from. ./mvnw dependency:tree shows the full tree; Telemetry used it to discover that kafka-clients resolved to 4.2.2, not the 4.1.x the spec assumed, and pinned the Docker image to match.
The Spring Boot parent POM
All three projects inherit from spring-boot-starter-parent:4.1.2. It gives dependency management (a curated, tested set of compatible versions for hundreds of libraries, which is why starters are listed without <version>), sensible plugin configuration, and the java.version property. All three projects use the same Boot patch version so behaviour matches across the platform.
Multi-module builds (Telemetry and Judge)
telemetry (parent, packaging=pom) judge (parent, packaging=pom) ├── telemetry-common ◄─┬─┬─┐ ├── judge-core (no Spring, no Kafka) ◄─┬─┐ ├── telemetry-simulator ──┘ │ │ ├── judge-service (Spring Boot app) ──────┘ │ ├── telemetry-detector ────┘ │ └── judge-eval (CLI app) ─────────────────┘ └── telemetry-verifier ──────┘
- The reactor builds modules in dependency order (common or core first).
- Module boundaries enforce architecture:
judge-corehas no Spring or Kafka dependency, so it is impossible (it won't compile) for the agent loop or policy to accidentally depend on them. That's stronger than any code-review rule. - Shared test helpers via a test-jar:
telemetry-commonpublishes its test classes (MutableClock,TestEvents,KafkaTestContainer,StubArenaServer) as a separate artifact;judge-coredoes the same (ScriptedGateway,TestSettings,FakeToolExecutor). ./mvnw -pl telemetry-detector -am verify:-plpicks the module,-am("also make") builds what it depends on. It's needed because the build neverinstalls modules into~/.m2. When running one test class with-am,-Dsurefire.failIfNoSpecifiedTests=falsestops upstream modules from failing with "no tests matched".- Per-module coverage: the parent sets
jacoco.skip=true(it has no code) and each child enforces 85%, so a well-tested module can't hide a poorly tested one.
Plugins in your builds
| Plugin | Does |
|---|---|
spring-boot-maven-plugin | Repackages the jar into an executable "fat jar": your classes plus every dependency jar plus a launcher, runnable with java -jar |
maven-compiler-plugin | Runs javac with the annotation processors in order (lombok, then lombok-mapstruct-binding, then mapstruct-processor) and passes -Amapstruct.defaultComponentModel=spring |
maven-surefire-plugin / maven-failsafe-plugin | Unit tests / integration tests (module 13). The Judge configures excludedGroups to skip live tests |
jacoco-maven-plugin 0.8.13 | prepare-agent, report, check (BUNDLE, LINE, COVEREDRATIO ≥ 0.85); excludes *Application and generated mappers |
The Maven Wrapper
./mvnw (or mvnw.cmd on Windows) downloads and runs the exact Maven version pinned in .mvn/wrapper/maven-wrapper.properties. Nobody needs Maven installed, and every machine and the CI runner build with the same version. Your machine has no global mvn at all, by design.
2. Docker
A container is an isolated process (or group of processes) on the host's kernel, with its own filesystem, network and resource limits (Linux namespaces and cgroups). It's much lighter than a virtual machine, which runs a whole guest operating system. An image is the read-only template, built in layers; a container is a running instance with a thin writable layer on top.
Arena's Dockerfile, line by line
# ---- stage 1: build ----
FROM eclipse-temurin:21-jdk AS build # full JDK to compile
WORKDIR /app
COPY .mvn .mvn
COPY mvnw pom.xml ./ # 1. only the build definition first...
RUN ./mvnw -q -B dependency:go-offline # 2. ...download dependencies: a cached layer
COPY src src # 3. then the source (changes often)
RUN ./mvnw -q -B package -DskipTests # tests already ran in CI
# ---- stage 2: run ----
FROM eclipse-temurin:21-jre # runtime only: smaller, no compiler
WORKDIR /app
RUN useradd --system --uid 1001 arena # a non-root user
COPY --from=build /app/target/arena-*.jar app.jar # only the jar crosses over
USER arena
EXPOSE 8080
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75", "-jar", "app.jar"]
- Layer caching: Docker reuses a cached layer if its instruction and inputs haven't changed. Copying
pom.xmlbeforesrcmeans a code change doesn't re-download every dependency. Telemetry and the Judge get the same effect with a BuildKit cache mount:RUN --mount=type=cache,target=/root/.m2 ./mvnw .... - Multi-stage: the build stage has the JDK, Maven and source; the final image has only a JRE and one jar. That makes it smaller, faster to pull, and gives a smaller attack surface (no compiler or source code inside).
- Non-root: if the app is compromised, the attacker isn't root inside the container.
- Exec-form ENTRYPOINT (the JSON array): Java runs as PID 1 and receives
SIGTERMdirectly, so Spring can shut down gracefully (finish requests, close Kafka consumers). - Memory:
-XX:MaxRAMPercentage=75sizes the heap from the container's memory limit. Combined withmem_limit: 1gin Compose, that prevented three detector JVMs from starving the laptop (module 02). - One Dockerfile for many modules (Telemetry, Judge):
ARG MODULEplus./mvnw -pl ${MODULE} -am packagebuilds any module into its own image. .dockerignorekeeps.env(secrets) andtarget/out of the build context, so secrets can never end up in an image layer. Secrets are injected at runtime withenv_file.
3. Docker Compose
Compose describes a multi-container application in one YAML file and starts it with docker compose up.
services:
postgres:
image: postgres:17-alpine
volumes: ["pgdata:/var/lib/postgresql/data"] # named volume: data survives container restarts
healthcheck:
test: ["CMD-SHELL", "pg_isready -U arena -d arena"]
interval: 5s
retries: 10
redis:
image: redis:7-alpine # no volume: rebuildable from Postgres
healthcheck: { test: ["CMD", "redis-cli", "ping"], interval: 5s, retries: 10 }
arena:
build: .
env_file: .env # secrets at runtime
environment:
DB_URL: jdbc:postgresql://postgres:5432/arena # "postgres" = the service name
REDIS_HOST: redis
ports: ["8080:8080"] # host:container
depends_on:
postgres: { condition: service_healthy } # wait until actually ready
redis: { condition: service_healthy }
prometheus:
image: prom/prometheus:latest
profiles: ["observability"] # only with --profile observability
- Service discovery by DNS: Compose puts services on a shared network where each service name resolves to its container(s). Arena reaches PostgreSQL at
postgres:5432, neverlocalhost(inside a container,localhostis the container itself). - Healthchecks +
depends_on: service_healthy: "the container started" is not "the database accepts connections". Without this, Arena would race PostgreSQL at startup and crash. - Profiles: optional services. Arena's
observability; Telemetry'ssim(start a run deliberately) andtools(the one-shot verifier,docker compose run --rm). Gotcha:docker compose down -vdoesn't touch profiled services unless you name the profiles. - Replicas: Telemetry's detector has
deploy.replicas: 3, three identical containers in one Kafka Streams group, with no host ports (they would clash). - YAML anchors: Telemetry defines shared settings once (
x-app: &app) and merges them into each app (<<: *app). - Cross-project networking: the Judge's Compose joins Telemetry's network as an external network (
telemetry_default) to reach Kafka atkafka:19092, and reaches Arena on the host throughhost.docker.internal(extra_hosts: host-gateway). - Windows gotcha: Git Bash rewrites paths like
/opt/kafka/bin/...passed todocker compose exec; the scripts setMSYS_NO_PATHCONV=1.
4. Continuous integration with GitHub Actions
name: ci
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with: { distribution: temurin, java-version: "21", cache: maven }
- run: ./mvnw -B verify # -B = batch mode: no interactive or colour output
- uses: actions/upload-artifact@v4
if: always() # keep the coverage report even when the build fails
with: { name: jacoco-report, path: target/site/jacoco }
- Every push and pull request runs the full gate: compile, unit and integration tests (GitHub's Ubuntu runners include Docker, so Testcontainers works unchanged) and the coverage threshold.
- The Judge's CI has no API key on purpose: no test needs one.
- Arena also has a JetBrains Qodana static-analysis workflow configured.
- What's not there (and what you'd add for CD): build and push images to a registry, scan them, deploy to an environment, run smoke tests.
5. Observability: metrics, logs and traces
Monitoring tells you that something is wrong; observability lets you ask why from the outside. The three pillars:
| Pillar | What | Your projects |
|---|---|---|
| Metrics | Numbers over time (rates, latencies, sizes): cheap, aggregated, good for alerts | Micrometer → Prometheus → Grafana |
| Logs | Discrete events with context | Structured JSON (ECS) to the console; no secrets or prompt text |
| Traces | One request's path across services, with timings | Not implemented; next would be OpenTelemetry / Micrometer Tracing, propagating trace ids over HTTP and Kafka headers |
Micrometer: a facade for metrics
Micrometer is to metrics what SLF4J is to logging: you code against its API (MeterRegistry), and a registry implementation exports to a backend. Here that's micrometer-registry-prometheus, at runtime scope. Spring Boot auto-instruments HTTP requests, JVM memory and GC, connection pools, Kafka clients and more; you add business metrics.
| Meter | Measures | Your examples |
|---|---|---|
| Counter | A monotonically increasing count (use rates of it) | arena.matches.recorded; arena.moderation.actions tagged action and source; arena.leaderboard.sync.failures; telemetry.bridge.dead_lettered; events sent by type |
| Gauge | A current value that goes up and down, sampled when scraped | arena.leaderboard.size (Redis ZCARD); telemetry.sim.players.cheating |
| Timer | Count, total time and distribution of durations | arena.leaderboard.rank.lookup tagged source=redis|db |
| Distribution summary | The distribution of non-time values (sizes, scores) | (possible for flag scores or token counts) |
Timer redisLookupTimer = Timer.builder("arena.leaderboard.rank.lookup").tag("source", "redis").register(registry);
Timer.Sample sample = Timer.start(meterRegistry);
try {
return lookupFromRedis(playerId);
} finally {
sample.stop(redisLookupTimer); // always recorded, even on exceptions
}
Each unique combination of tag values creates a separate time series. Tags like action (7 values) or source (4) are fine. Tagging by playerId would create a million series and take down Prometheus. Per-entity detail belongs in logs, not metrics.
Histograms for percentiles: Arena enables percentiles-histogram for http.server.requests and the rank-lookup timer. Micrometer then exports bucket counters (..._bucket{le="0.005"} and so on), so Prometheus can compute percentiles across all instances with histogram_quantile. Percentiles computed inside one app can't be averaged across instances; buckets can be summed. The cost: the result is only as precise as the bucket boundaries, which is why the README reports server-side values as "≤ 2.8 ms", an upper bound.
Prometheus: pull-based scraping
Prometheus periodically pulls (scrapes) /actuator/prometheus from each target and stores time series. Micrometer's names become Prometheus names (dots to underscores, with unit and type suffixes): arena.leaderboard.rank.lookup becomes arena_leaderboard_rank_lookup_seconds_*.
- job_name: detector
metrics_path: /actuator/prometheus
scrape_interval: 5s
dns_sd_configs: # 3 replicas, no fixed addresses:
- names: ["telemetry-detector"] # the service name resolves to one A record per replica
type: A
port: 8082
A taste of PromQL, the query language:
rate(arena_matches_recorded_total[1m]) # matches per second
histogram_quantile(0.95,
sum by (le) (rate(arena_leaderboard_rank_lookup_seconds_bucket{source="redis"}[5m]))) # p95 latency
Grafana
Grafana queries Prometheus and draws dashboards. Telemetry provisions its datasource and an 8-panel dashboard from files in ops/grafana/ (events produced and processed per second, stream process rate by instance, maximum consumer lag, flags per minute by rule, dead letters per minute, cheaters active, bridge outcomes), so the dashboard exists the moment the stack starts. Dashboards as code are reviewable and reproducible. The Judge ships its own dashboard JSON: verdicts per minute by action, average cost per verdict, p95 verdict latency, cache-hit share of input tokens, escalation rate, human-review share and circuit-breaker state.
6. Reading latency: percentiles, not averages
- p50 (median): half the requests were faster. p95: 95% were faster, so 1 in 20 was slower. p99: 1 in 100 was slower.
- Averages hide the tail: 99 requests at 10 ms and one at 5 s average to about 60 ms, which describes nobody's experience. Users notice the tail, and in a page that makes many calls, someone almost always hits the slow one.
- Report percentiles with the load (concurrency, request rate), the data size, the machine and the warm-up. A latency number without its context means little.
- Client-side and server-side numbers measure different things (module 06): the client includes queueing for connections and threads, network and serialisation.
7. Load testing with k6
export const options = {
scenarios: {
warmup: { executor: 'constant-vus', vus: 50, duration: '15s', tags: { phase: 'warmup' } },
measure: { executor: 'constant-vus', vus: 200, duration: '60s', startTime: '15s', tags: { phase: 'measure' } },
},
thresholds: { 'http_req_duration{phase:measure}': ['p(95)<2000'] }, // a sanity limit
summaryTrendStats: ['avg', 'p(50)', 'p(95)', 'p(99)'],
};
export default function () {
const id = ids[Math.floor(Math.random() * ids.length)]; // 10,000 sampled players
const res = http.get(`${BASE}/api/v1/leaderboard/players/${id}?source=${SOURCE}`);
check(res, { 'status is 200': (r) => r.status === 200 });
}
- A virtual user (VU) runs the default function in a loop. With
constant-vus, 200 VUs send requests back to back, so throughput depends on latency (a "closed" model). That's why the slow database path managed only 12.6 req/s. - A warm-up phase lets the JIT compile hot code and pools fill before measuring; results are filtered by the
phase:measuretag. - Method: seed 1,000,000 players, restart so Redis is rebuilt, export 10,000 random ids, run each source three times, report the third run (the first two summaries weren't captured, which the README says).
Everything ran on one machine (a Ryzen 5 3600 under Docker Desktop): Arena, PostgreSQL, Redis and the k6 load generator competed for the same CPUs, which inflates latency and caps throughput. PostgreSQL ran with default settings. Only the third of three runs was fully captured. These don't change the conclusion (O(log N) versus O(N) at a million rows is decisive), but naming them shows you know how benchmarks go wrong.
8. Interview questions
What does ./mvnw verify do in your projects? Basic
It runs every lifecycle phase up to verify: compile with Lombok and MapStruct annotation processing, run unit tests with Surefire, package the Boot fat jar, run *IT integration tests with Failsafe against Testcontainers (Postgres, Redis, Kafka, WireMock), then check results and enforce the JaCoCo 85% line-coverage gate, per module in the multi-module builds. It's the exact command CI runs.
Why is the PostgreSQL driver runtime scope? Intermediate
My code only uses JDBC and JPA interfaces; the driver is needed only to run. Runtime scope keeps it off the compile classpath, so nobody can accidentally import driver-specific classes, which keeps the code portable. Same for the Prometheus registry behind the Micrometer API.
Why split Telemetry and the Judge into Maven modules? Intermediate
Separate deployables (simulator, detector, verifier; service and eval CLI) that share models and code; shared test helpers via a test-jar; and enforced boundaries. judge-core has no Spring or Kafka dependency, so the agent loop and policy physically can't use them, and the evaluation reuses exactly the production decision path. Coverage is enforced per module.
Walk me through your Dockerfile. Basic
Two stages. The build stage uses the Temurin 21 JDK, copies the Maven wrapper and POM first and downloads dependencies, a layer that stays cached until the POM changes, then copies the source and packages the jar. The runtime stage starts from the much smaller JRE image, creates a non-root user, copies only the jar, and runs Java in exec form with MaxRAMPercentage=75 so the heap respects the container's memory limit. Secrets are excluded by .dockerignore and supplied at runtime.
How do containers find each other in Compose? Basic
Compose creates a network where each service name is a DNS name. Arena connects to postgres:5432 and redis:6379; the detectors to kafka:19092. Inside a container, localhost is the container itself. Prometheus even discovers the three detector replicas through DNS A records for the service name.
What do healthchecks and depends_on: service_healthy solve? Intermediate
A started container isn't necessarily ready. PostgreSQL takes seconds to accept connections, and Kafka longer. Healthchecks (pg_isready, redis-cli ping, a Kafka API-versions call) define "ready", and condition: service_healthy makes dependants wait for it, so apps don't crash-loop at startup. Apps should still retry connections, since dependencies can restart later.
Counter, gauge or timer: which would you use for what? Basic
A counter for things that only increase (matches recorded, moderation actions by action and source, dead-letters, sync failures), queried as rates. A gauge for a current level that goes up and down (leaderboard size, cheaters currently active). A timer for durations (rank lookups tagged by source), which gives a count, a sum and, with histograms enabled, buckets for percentiles.
Why report p95/p99 instead of the average? Intermediate
Averages hide the tail, and users experience the tail, especially when one page makes many calls. p95 says what 1 in 20 requests endures. I report p50, p95 and p99 with the load, data size, machine and warm-up, and note that server-side histogram percentiles are bucket upper bounds.
Why can't you average percentiles across instances, and how does Prometheus handle it? Deep
The average of each instance's p95 isn't the p95 of all requests: percentiles don't compose. Instead each instance exports histogram bucket counters, which can be summed across instances; Prometheus then runs histogram_quantile over the summed buckets. The precision depends on the bucket boundaries, so results are approximate upper bounds.
What's wrong with tagging a metric by player id? Intermediate
Cardinality. Every distinct tag value creates a new time series; a million players means a million series per metric, which exhausts Prometheus memory and makes queries slow. Use low-cardinality tags (action, source, rule) for metrics, and put per-entity detail in logs or traces.
How trustworthy is your benchmark? Curveball
Trustworthy for its conclusion, imperfect in its precision, and I can say why. Strengths: a realistic 1M-row dataset with realistic ties, a warm-up, a tagged measurement phase, three runs, client and server measurements cross-checked, and the method and machine documented. Weaknesses: the load generator shared the machine with the services, PostgreSQL ran with defaults, and only the third run's full summary was captured. The O(log N) versus O(N) gap is so large (about 280× at p95) that none of that changes the decision.
What would you add to go from CI to CD? Intermediate
Build the image once per commit and push it to a registry tagged with the commit SHA; scan it for vulnerabilities; deploy that exact image to staging with externalised config and secrets from a secret manager; run smoke tests against health and readiness endpoints; promote to production with a rolling or blue-green rollout gated on metrics; and record the deployment. Flyway runs migrations on startup, so schema changes would follow the expand-and-contract pattern.
9. Quick quiz
1. ./mvnw package runs which phases?
A phase runs everything before it.
2. Why copy pom.xml before src in the Dockerfile?
Layer caching.
3. Inside the Arena container, PostgreSQL is reached at:
The Compose service name is a DNS name.
4. Which meter fits "current leaderboard size"?
It goes up and down.
5. -pl judge-service -am means:
-am = also make dependencies.
6. Prometheus collects metrics by:
A pull model; /actuator/prometheus every 5 s in Telemetry.
10. Cheat sheet
- Maven lifecycle: validate → compile → test (Surefire) → package (fat jar) → integration-test (Failsafe) → verify (JaCoCo ≥ 85%) → install → deploy.
- Scopes: compile, runtime (JDBC driver, Prometheus registry), test, optional (Lombok).
dependency:tree. - Boot parent = managed versions + plugin defaults. Wrapper = a pinned Maven, no install.
- Multi-module: enforced boundaries (judge-core has no Spring or Kafka), test-jars,
-pl X -am, per-module coverage. - Processors: lombok → lombok-mapstruct-binding → mapstruct.
- Dockerfile: multi-stage (JDK build → JRE run), cached dependency layer or a BuildKit cache mount, non-root, exec-form entrypoint,
MaxRAMPercentage=75,.dockerignore,ARG MODULE. - Compose: DNS by service name, named volumes, healthchecks +
service_healthy, profiles, replicas, anchors, an external network,host.docker.internal,mem_limit. - CI: GitHub Actions → checkout, Temurin 21, Maven cache,
./mvnw -B verify, upload JaCoCoif: always(); no secrets needed. - Observability: metrics (yes), structured logs (yes), traces (next: OpenTelemetry).
- Micrometer: Counter, Gauge, Timer; low-cardinality tags; histograms for cross-instance percentiles.
- Prometheus: pull and scrape; DNS service discovery for replicas;
rate(),histogram_quantile(). Grafana: provisioned dashboards. - Percentiles > averages; context matters; client ≠ server timing.
- k6: constant-VU scenarios, warm-up then a tagged measure phase, thresholds; single-machine caveat.