Fairplay Interview Prep
Module 14

Maven, Docker, CI and observability

Code is only useful if it builds reproducibly, runs anywhere, is checked on every push and can be watched in production. This module covers the delivery side of your projects: Maven (lifecycle, dependencies, multi-module builds, plugins), Docker and Compose, GitHub Actions, and observability with Micrometer, Prometheus and Grafana. It also covers how to load-test with k6 and read percentiles honestly.

P1P2P3· about 2 hours

By the end you can answer

  • "What happens when you run ./mvnw verify?"
  • "What are dependency scopes? What does the Spring Boot parent POM give you?"
  • "Why multi-module? What does -pl ... -am do?"
  • "Walk me through your Dockerfile. Why multi-stage? Why a non-root user?"
  • "How do services find each other in Docker Compose? What are healthchecks and profiles for?"
  • "Counter vs gauge vs timer? Why percentiles instead of averages?"
  • "How did you run your benchmark, and what are its limitations?"

1. Maven: the build tool

Maven builds Java projects by convention: sources in src/main/java, resources in src/main/resources, tests in src/test/java, output in target/. The pom.xml (Project Object Model) declares the project's coordinates (groupId:artifactId:version, such as dev.fairplay:arena:0.0.1-SNAPSHOT; SNAPSHOT means a work in progress), its dependencies and its plugins.

The lifecycle

 validate → compile → test → package → integration-test → verify → install → deploy
              │        │        │              │              │
         javac +   Surefire  fat jar      Failsafe (*IT)   JaCoCo check (≥ 85%),
         annotation (unit)   (Boot         against          Failsafe results
         processors           plugin)      containers

Running a phase runs every phase before it. So ./mvnw verify compiles, runs unit tests, packages, runs integration tests and enforces coverage: the full gate CI uses. ./mvnw test stops after unit tests. Plugins attach their goals to phases (JaCoCo's check to verify, for example).

Dependencies and scopes

ScopeOn the classpath forYour examples
compile (default)Compiling, testing and runningSpring starters, MapStruct, the Anthropic SDK
runtimeRunning and testing, not compilingThe PostgreSQL JDBC driver, micrometer-registry-prometheus: code only talks to interfaces (JDBC, Micrometer), so it can't accidentally depend on the implementation
testTests onlyTestcontainers, Awaitility, the *-test starters
optionalNot passed on to projects depending on yoursLombok (a compile-time tool)

Transitive dependencies: you get your dependencies' dependencies automatically, which is convenient and is where version conflicts come from. ./mvnw dependency:tree shows the full tree; Telemetry used it to discover that kafka-clients resolved to 4.2.2, not the 4.1.x the spec assumed, and pinned the Docker image to match.

The Spring Boot parent POM

All three projects inherit from spring-boot-starter-parent:4.1.2. It gives dependency management (a curated, tested set of compatible versions for hundreds of libraries, which is why starters are listed without <version>), sensible plugin configuration, and the java.version property. All three projects use the same Boot patch version so behaviour matches across the platform.

Multi-module builds (Telemetry and Judge)

 telemetry (parent, packaging=pom)           judge (parent, packaging=pom)
 ├── telemetry-common    ◄─┬─┬─┐             ├── judge-core     (no Spring, no Kafka)  ◄─┬─┐
 ├── telemetry-simulator ──┘ │ │             ├── judge-service  (Spring Boot app) ──────┘ │
 ├── telemetry-detector  ────┘ │             └── judge-eval     (CLI app) ─────────────────┘
 └── telemetry-verifier  ──────┘

Plugins in your builds

PluginDoes
spring-boot-maven-pluginRepackages the jar into an executable "fat jar": your classes plus every dependency jar plus a launcher, runnable with java -jar
maven-compiler-pluginRuns javac with the annotation processors in order (lombok, then lombok-mapstruct-binding, then mapstruct-processor) and passes -Amapstruct.defaultComponentModel=spring
maven-surefire-plugin / maven-failsafe-pluginUnit tests / integration tests (module 13). The Judge configures excludedGroups to skip live tests
jacoco-maven-plugin 0.8.13prepare-agent, report, check (BUNDLE, LINE, COVEREDRATIO ≥ 0.85); excludes *Application and generated mappers

The Maven Wrapper

./mvnw (or mvnw.cmd on Windows) downloads and runs the exact Maven version pinned in .mvn/wrapper/maven-wrapper.properties. Nobody needs Maven installed, and every machine and the CI runner build with the same version. Your machine has no global mvn at all, by design.

2. Docker

A container is an isolated process (or group of processes) on the host's kernel, with its own filesystem, network and resource limits (Linux namespaces and cgroups). It's much lighter than a virtual machine, which runs a whole guest operating system. An image is the read-only template, built in layers; a container is a running instance with a thin writable layer on top.

Arena's Dockerfile, line by line

arena/Dockerfile
# ---- stage 1: build ----
FROM eclipse-temurin:21-jdk AS build             # full JDK to compile
WORKDIR /app
COPY .mvn .mvn
COPY mvnw pom.xml ./                             # 1. only the build definition first...
RUN ./mvnw -q -B dependency:go-offline           # 2. ...download dependencies: a cached layer
COPY src src                                     # 3. then the source (changes often)
RUN ./mvnw -q -B package -DskipTests             # tests already ran in CI

# ---- stage 2: run ----
FROM eclipse-temurin:21-jre                      # runtime only: smaller, no compiler
WORKDIR /app
RUN useradd --system --uid 1001 arena            # a non-root user
COPY --from=build /app/target/arena-*.jar app.jar  # only the jar crosses over
USER arena
EXPOSE 8080
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75", "-jar", "app.jar"]

3. Docker Compose

Compose describes a multi-container application in one YAML file and starts it with docker compose up.

arena/compose.yaml (excerpt)
services:
  postgres:
    image: postgres:17-alpine
    volumes: ["pgdata:/var/lib/postgresql/data"]      # named volume: data survives container restarts
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U arena -d arena"]
      interval: 5s
      retries: 10
  redis:
    image: redis:7-alpine                             # no volume: rebuildable from Postgres
    healthcheck: { test: ["CMD", "redis-cli", "ping"], interval: 5s, retries: 10 }
  arena:
    build: .
    env_file: .env                                    # secrets at runtime
    environment:
      DB_URL: jdbc:postgresql://postgres:5432/arena   # "postgres" = the service name
      REDIS_HOST: redis
    ports: ["8080:8080"]                              # host:container
    depends_on:
      postgres: { condition: service_healthy }        # wait until actually ready
      redis:    { condition: service_healthy }
  prometheus:
    image: prom/prometheus:latest
    profiles: ["observability"]                       # only with --profile observability

4. Continuous integration with GitHub Actions

.github/workflows/ci.yml (Arena; the other two are nearly identical)
name: ci
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: "21", cache: maven }
      - run: ./mvnw -B verify                       # -B = batch mode: no interactive or colour output
      - uses: actions/upload-artifact@v4
        if: always()                                # keep the coverage report even when the build fails
        with: { name: jacoco-report, path: target/site/jacoco }

5. Observability: metrics, logs and traces

Monitoring tells you that something is wrong; observability lets you ask why from the outside. The three pillars:

PillarWhatYour projects
MetricsNumbers over time (rates, latencies, sizes): cheap, aggregated, good for alertsMicrometer → Prometheus → Grafana
LogsDiscrete events with contextStructured JSON (ECS) to the console; no secrets or prompt text
TracesOne request's path across services, with timingsNot implemented; next would be OpenTelemetry / Micrometer Tracing, propagating trace ids over HTTP and Kafka headers

Micrometer: a facade for metrics

Micrometer is to metrics what SLF4J is to logging: you code against its API (MeterRegistry), and a registry implementation exports to a backend. Here that's micrometer-registry-prometheus, at runtime scope. Spring Boot auto-instruments HTTP requests, JVM memory and GC, connection pools, Kafka clients and more; you add business metrics.

MeterMeasuresYour examples
CounterA monotonically increasing count (use rates of it)arena.matches.recorded; arena.moderation.actions tagged action and source; arena.leaderboard.sync.failures; telemetry.bridge.dead_lettered; events sent by type
GaugeA current value that goes up and down, sampled when scrapedarena.leaderboard.size (Redis ZCARD); telemetry.sim.players.cheating
TimerCount, total time and distribution of durationsarena.leaderboard.rank.lookup tagged source=redis|db
Distribution summaryThe distribution of non-time values (sizes, scores)(possible for flag scores or token counts)
arena/.../leaderboard/LeaderboardService.java
Timer redisLookupTimer = Timer.builder("arena.leaderboard.rank.lookup").tag("source", "redis").register(registry);
Timer.Sample sample = Timer.start(meterRegistry);
try {
    return lookupFromRedis(playerId);
} finally {
    sample.stop(redisLookupTimer);                 // always recorded, even on exceptions
}
Tag cardinality: the classic metrics mistake

Each unique combination of tag values creates a separate time series. Tags like action (7 values) or source (4) are fine. Tagging by playerId would create a million series and take down Prometheus. Per-entity detail belongs in logs, not metrics.

Histograms for percentiles: Arena enables percentiles-histogram for http.server.requests and the rank-lookup timer. Micrometer then exports bucket counters (..._bucket{le="0.005"} and so on), so Prometheus can compute percentiles across all instances with histogram_quantile. Percentiles computed inside one app can't be averaged across instances; buckets can be summed. The cost: the result is only as precise as the bucket boundaries, which is why the README reports server-side values as "≤ 2.8 ms", an upper bound.

Prometheus: pull-based scraping

Prometheus periodically pulls (scrapes) /actuator/prometheus from each target and stores time series. Micrometer's names become Prometheus names (dots to underscores, with unit and type suffixes): arena.leaderboard.rank.lookup becomes arena_leaderboard_rank_lookup_seconds_*.

telemetry ops/prometheus.yml
- job_name: detector
  metrics_path: /actuator/prometheus
  scrape_interval: 5s
  dns_sd_configs:                    # 3 replicas, no fixed addresses:
    - names: ["telemetry-detector"]  # the service name resolves to one A record per replica
      type: A
      port: 8082

A taste of PromQL, the query language:

rate(arena_matches_recorded_total[1m])                                  # matches per second
histogram_quantile(0.95,
  sum by (le) (rate(arena_leaderboard_rank_lookup_seconds_bucket{source="redis"}[5m])))   # p95 latency

Grafana

Grafana queries Prometheus and draws dashboards. Telemetry provisions its datasource and an 8-panel dashboard from files in ops/grafana/ (events produced and processed per second, stream process rate by instance, maximum consumer lag, flags per minute by rule, dead letters per minute, cheaters active, bridge outcomes), so the dashboard exists the moment the stack starts. Dashboards as code are reviewable and reproducible. The Judge ships its own dashboard JSON: verdicts per minute by action, average cost per verdict, p95 verdict latency, cache-hit share of input tokens, escalation rate, human-review share and circuit-breaker state.

6. Reading latency: percentiles, not averages

7. Load testing with k6

arena/benchmark/rank-lookup.js
export const options = {
  scenarios: {
    warmup:  { executor: 'constant-vus', vus: 50,  duration: '15s', tags: { phase: 'warmup' } },
    measure: { executor: 'constant-vus', vus: 200, duration: '60s', startTime: '15s', tags: { phase: 'measure' } },
  },
  thresholds: { 'http_req_duration{phase:measure}': ['p(95)<2000'] },     // a sanity limit
  summaryTrendStats: ['avg', 'p(50)', 'p(95)', 'p(99)'],
};
export default function () {
  const id = ids[Math.floor(Math.random() * ids.length)];                  // 10,000 sampled players
  const res = http.get(`${BASE}/api/v1/leaderboard/players/${id}?source=${SOURCE}`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}
Limitations to state yourself

Everything ran on one machine (a Ryzen 5 3600 under Docker Desktop): Arena, PostgreSQL, Redis and the k6 load generator competed for the same CPUs, which inflates latency and caps throughput. PostgreSQL ran with default settings. Only the third of three runs was fully captured. These don't change the conclusion (O(log N) versus O(N) at a million rows is decisive), but naming them shows you know how benchmarks go wrong.

8. Interview questions

What does ./mvnw verify do in your projects? Basic

It runs every lifecycle phase up to verify: compile with Lombok and MapStruct annotation processing, run unit tests with Surefire, package the Boot fat jar, run *IT integration tests with Failsafe against Testcontainers (Postgres, Redis, Kafka, WireMock), then check results and enforce the JaCoCo 85% line-coverage gate, per module in the multi-module builds. It's the exact command CI runs.

Why is the PostgreSQL driver runtime scope? Intermediate

My code only uses JDBC and JPA interfaces; the driver is needed only to run. Runtime scope keeps it off the compile classpath, so nobody can accidentally import driver-specific classes, which keeps the code portable. Same for the Prometheus registry behind the Micrometer API.

Why split Telemetry and the Judge into Maven modules? Intermediate

Separate deployables (simulator, detector, verifier; service and eval CLI) that share models and code; shared test helpers via a test-jar; and enforced boundaries. judge-core has no Spring or Kafka dependency, so the agent loop and policy physically can't use them, and the evaluation reuses exactly the production decision path. Coverage is enforced per module.

Walk me through your Dockerfile. Basic

Two stages. The build stage uses the Temurin 21 JDK, copies the Maven wrapper and POM first and downloads dependencies, a layer that stays cached until the POM changes, then copies the source and packages the jar. The runtime stage starts from the much smaller JRE image, creates a non-root user, copies only the jar, and runs Java in exec form with MaxRAMPercentage=75 so the heap respects the container's memory limit. Secrets are excluded by .dockerignore and supplied at runtime.

How do containers find each other in Compose? Basic

Compose creates a network where each service name is a DNS name. Arena connects to postgres:5432 and redis:6379; the detectors to kafka:19092. Inside a container, localhost is the container itself. Prometheus even discovers the three detector replicas through DNS A records for the service name.

What do healthchecks and depends_on: service_healthy solve? Intermediate

A started container isn't necessarily ready. PostgreSQL takes seconds to accept connections, and Kafka longer. Healthchecks (pg_isready, redis-cli ping, a Kafka API-versions call) define "ready", and condition: service_healthy makes dependants wait for it, so apps don't crash-loop at startup. Apps should still retry connections, since dependencies can restart later.

Counter, gauge or timer: which would you use for what? Basic

A counter for things that only increase (matches recorded, moderation actions by action and source, dead-letters, sync failures), queried as rates. A gauge for a current level that goes up and down (leaderboard size, cheaters currently active). A timer for durations (rank lookups tagged by source), which gives a count, a sum and, with histograms enabled, buckets for percentiles.

Why report p95/p99 instead of the average? Intermediate

Averages hide the tail, and users experience the tail, especially when one page makes many calls. p95 says what 1 in 20 requests endures. I report p50, p95 and p99 with the load, data size, machine and warm-up, and note that server-side histogram percentiles are bucket upper bounds.

Why can't you average percentiles across instances, and how does Prometheus handle it? Deep

The average of each instance's p95 isn't the p95 of all requests: percentiles don't compose. Instead each instance exports histogram bucket counters, which can be summed across instances; Prometheus then runs histogram_quantile over the summed buckets. The precision depends on the bucket boundaries, so results are approximate upper bounds.

What's wrong with tagging a metric by player id? Intermediate

Cardinality. Every distinct tag value creates a new time series; a million players means a million series per metric, which exhausts Prometheus memory and makes queries slow. Use low-cardinality tags (action, source, rule) for metrics, and put per-entity detail in logs or traces.

How trustworthy is your benchmark? Curveball

Trustworthy for its conclusion, imperfect in its precision, and I can say why. Strengths: a realistic 1M-row dataset with realistic ties, a warm-up, a tagged measurement phase, three runs, client and server measurements cross-checked, and the method and machine documented. Weaknesses: the load generator shared the machine with the services, PostgreSQL ran with defaults, and only the third run's full summary was captured. The O(log N) versus O(N) gap is so large (about 280× at p95) that none of that changes the decision.

What would you add to go from CI to CD? Intermediate

Build the image once per commit and push it to a registry tagged with the commit SHA; scan it for vulnerabilities; deploy that exact image to staging with externalised config and secrets from a secret manager; run smoke tests against health and readiness endpoints; promote to production with a rolling or blue-green rollout gated on metrics; and record the deployment. Flyway runs migrations on startup, so schema changes would follow the expand-and-contract pattern.

9. Quick quiz

1. ./mvnw package runs which phases?

A phase runs everything before it.

2. Why copy pom.xml before src in the Dockerfile?

Layer caching.

3. Inside the Arena container, PostgreSQL is reached at:

The Compose service name is a DNS name.

4. Which meter fits "current leaderboard size"?

It goes up and down.

5. -pl judge-service -am means:

-am = also make dependencies.

6. Prometheus collects metrics by:

A pull model; /actuator/prometheus every 5 s in Telemetry.

10. Cheat sheet

  • Maven lifecycle: validate → compile → test (Surefire) → package (fat jar) → integration-test (Failsafe) → verify (JaCoCo ≥ 85%) → install → deploy.
  • Scopes: compile, runtime (JDBC driver, Prometheus registry), test, optional (Lombok). dependency:tree.
  • Boot parent = managed versions + plugin defaults. Wrapper = a pinned Maven, no install.
  • Multi-module: enforced boundaries (judge-core has no Spring or Kafka), test-jars, -pl X -am, per-module coverage.
  • Processors: lombok → lombok-mapstruct-binding → mapstruct.
  • Dockerfile: multi-stage (JDK build → JRE run), cached dependency layer or a BuildKit cache mount, non-root, exec-form entrypoint, MaxRAMPercentage=75, .dockerignore, ARG MODULE.
  • Compose: DNS by service name, named volumes, healthchecks + service_healthy, profiles, replicas, anchors, an external network, host.docker.internal, mem_limit.
  • CI: GitHub Actions → checkout, Temurin 21, Maven cache, ./mvnw -B verify, upload JaCoCo if: always(); no secrets needed.
  • Observability: metrics (yes), structured logs (yes), traces (next: OpenTelemetry).
  • Micrometer: Counter, Gauge, Timer; low-cardinality tags; histograms for cross-instance percentiles.
  • Prometheus: pull and scrape; DNS service discovery for replicas; rate(), histogram_quantile(). Grafana: provisioned dashboards.
  • Percentiles > averages; context matters; client ≠ server timing.
  • k6: constant-VU scenarios, warm-up then a tagged measure phase, thresholds; single-machine caveat.
Saved in this browser.