Fairplay Interview Prep
Module 18

Scaling Fairplay to huge numbers

"How would this work with a million players?" is the most common follow-up to any portfolio project. This module answers it with arithmetic, not hand-waving. It starts from what was measured on one laptop, multiplies to a target, finds what breaks first in each project, and lays out a staged plan (10×, 100×, 1,000×) with the specific changes at each stage. All projections are back-of-envelope estimates, labelled as such, with the formulas shown so you can redo them live.

P1P2P3· about 2.5 hours

By the end you can answer

  • "How would you scale this to a million concurrent players?"
  • "What breaks first?" (for each of the three services)
  • "How many partitions or brokers would you need? Show me the maths."
  • "How do you keep LLM cost and rate limits under control at scale?"
  • "How would you go multi-region?"

1. How to answer any scaling question

  1. Fix the target in numbers: users, requests per second, events per second, data per day, cost per day.
  2. Start from measured baselines, not guesses. You have real ones.
  3. Multiply and find the first bottleneck: the resource that hits its limit first (CPU, a single database primary, a hot key, a rate limit, a budget, human reviewers).
  4. Fix that bottleneck (scale out, partition, cache, reduce the work, change the data structure), then repeat.
  5. Name the trade-offs: consistency, latency, cost, operational complexity.
The most powerful scaling move is usually "do less work"

Before adding machines, ask whether the work is necessary: send fewer events, smaller events, fewer flags to the LLM, fewer rows to scan. Several of the biggest wins below come from reducing work, not distributing it.

2. What was measured (the starting point)

ProjectMeasured on one laptop (Ryzen 5 3600, Docker Desktop)What it implies (estimate)
ArenaRedis rank lookups: 5,764 req/s at p95 63.5 ms (200 virtual users, load generator on the same machine); server-side mean 2.9 ms. Rebuild of 1M players in 22.6 sOne Arena instance with Redis serves thousands of reads per second even while sharing its CPU with the load generator
Telemetry~24,900 events/s from 1,000 players; 3 detector instances × 2 stream threads; after a kill, 2 instances had ~5,000 events/s of headroom2 instances ≈ 30,000 events/s maximum, so roughly 7,000–8,000 events/s per stream thread on that machine
Telemetry (load shape)EventFactory: per player per second, 20 POSITION + ~4.5 SHOT + ~0.3 KILL~25 events per player per second; JSON events ≈ 250–300 bytes each
Telemetry (flags)51 flags in a 5-minute run for 1,000 players with 5% cheaters (31 aimbot + 20 speedhack)~10 flags per minute per 1,000 players at that cheater rate
JudgeNo throughput or cost measured (no API key yet). Configured limiter: 120 model calls per minuteThroughput is bounded by the provider's limits and by budget, not by the code

3. The target: one million concurrent players

Let's size a big but realistic game: 10 million registered players, 1 million playing at the same time, in 10-player matches of 2 minutes. The formulas:

 events/s      = concurrent players × events per player per second   = 1,000,000 × 25       = 25,000,000 /s
 bytes/s (raw) = events/s × bytes per event                         = 25,000,000 × ~275 B  ≈ 6.9 GB/s
 matches/s     = (concurrent players ÷ players per match) ÷ match seconds = (1,000,000 ÷ 10) ÷ 120 ≈ 833 /s
 flags/s       = players × flag rate                               = 1,000 × 10/min       ≈ 10,000/min ≈ 170 /s  (5% cheaters)
                                                                                                    ≈ 35 /s    (1% cheaters)
 token refreshes/s = concurrent players ÷ access-token lifetime    = 1,000,000 ÷ 900 s    ≈ 1,100 /s

Every number above is an estimate built from your own measurements and code. Now let's see what each one does to each service.

4. Telemetry at scale: the event firehose

4.1 What breaks first

At 25 million JSON events per second and ~7,500 events per second per stream thread, you'd need about 3,300 stream threads (25,000,000 ÷ 7,500), and ingesting ~6.9 GB/s, roughly 20 GB/s across the cluster with replication factor 3. It's possible, but very expensive. The first move is to send less:

LeverEffect (estimate)Trade-off
Aggregate movement at the edge. 20 POSITION events/s are 80% of the volume. The game server (authoritative over movement anyway) can send one summary per second: samples, moving samples, overspeed samples, max speed25 → ~6 events per player per second (−75%)Detection works on 1 s summaries instead of raw samples; the speedhack window aggregates summaries
Binary encoding (Avro or Protobuf with a schema registry, already on the roadmap): field names aren't repeated in every record~275 B → ~60–90 B per eventSchema management; less human-readable topics
Parse once. Today each event is parsed twice (to avoid the repartition trap). With a binary format and a key-preserving processor, parse onceNoticeably cheaper per eventMove to the new KStream#process overloads that KIP-1128 (a draft under discussion as of November 2026) proposes in place of processValues (banned by the spec), once a release ships them
Compression (already lz4) with bigger batchesFewer bytes on the wireA few milliseconds of latency

With edge aggregation and binary encoding: ~6 million events/s × ~80 B ≈ ~0.5 GB/s before replication. That's a normal, mid-sized Kafka cluster rather than a heroic one.

4.2 Partitions, instances and brokers

 stream threads  ≈ events/s ÷ per-thread throughput  ≈ 6,000,000 ÷ ~10,000 (cheaper parsing) ≈ 600
 partitions      ≥ stream threads (one task per partition) → choose 1,024 up front (room to grow)
 instances       ≈ 600 threads ÷ 4 threads per instance   ≈ 150 detector instances (+ headroom for N−1)
 brokers         ≈ (bytes/s × replication factor) ÷ practical per-broker write throughput
                 ≈ (0.5 GB/s × 3) ÷ ~0.1–0.3 GB/s          ≈ 5–15 brokers (measure! disks, network, compression vary)

4.3 State and the integrity proof

4.4 Multi-region

Players play on regional game servers, and detection only needs one player's events, so run a full Telemetry pipeline per region (regional Kafka, regional detectors) close to the game servers. Replicate only the output (flags, small) to a central cluster for the Judge and global reporting, using MirrorMaker 2 or a managed equivalent. Raw events never cross oceans.

5. The Judge at scale: rate limits, cost and humans

5.1 What breaks first: not the code

 flags/s           ≈ 170 /s (5% cheaters)        ≈ 35 /s (1% cheaters)
 model calls/flag  ≈ 3  (a tool round or two, plus a verdict; occasionally an escalation) — an assumption
 model calls/min   ≈ 170 × 3 × 60 ≈ 30,600 /min  vs the configured limiter: 120 /min  →  ~250× too low
 cost/day          = flags/day × cost per verdict
                   ≈ 14.7 million flags/day × c   (e.g. if c were one cent, that's ~$147,000 per day — illustrative only)
 human reviews/day = flags/day × review share     (even 2% of 14.7M ≈ 294,000 reviews/day — no team can do that)

Three limits arrive long before any CPU limit: the provider's rate limits, the budget, and human review capacity. So the scaling strategy is mostly about judging fewer flags, more cheaply.

5.2 Judge fewer flags

5.3 Judge more cheaply

5.4 Throughput mechanics

5.5 The human bottleneck

Any "send to human review" percentage multiplied by millions of flags is beyond any moderation team. At scale you need review prioritisation (highest impact first: top-tier players, high scores, repeat offenders), batching (similar cases reviewed together), sampling (audit a random fraction of auto-decisions to measure quality), and a feedback loop that turns human decisions into labelled evaluation data, which improves the prompts and the triage model, which lowers the review rate. That loop is the long-term scaling plan for an AI moderation system.

6. Arena at scale: writes, reads, growth

6.1 Write load

 matches/s              ≈ 833
 per match              : 1 match insert + 10 participant inserts + 10 player row updates (locked) + 10 ZADDs
 row writes/s           ≈ 833 + 8,330 + 8,330 ≈ 17,500 /s (plus index and WAL writes)
 participant rows/day   ≈ 8,330 × 86,400 ≈ 720 million rows/day
 refresh-token rows/day ≈ 1,100 /s × 86,400 ≈ 95 million rows/day  (and nothing deletes them today)

6.2 Read load and caching

6.3 Background jobs that don't scale as written

JobTodayAt scale
Percentiles (StatDistributionService)Every instance runs three percentile_cont sorts over all eligible players every 60 sRun once (a scheduled job with a distributed lock such as ShedLock), store the 101 cut points in Redis or a table; or maintain approximate distributions incrementally (t-digest)
Leaderboard rebuildEvery instance checks at startup and may rebuild (1M in 22.6 s; 10M ≈ 4 minutes by extrapolation)Run as a single coordinated job with leader election; never on every replica's startup
Redis syncIn-memory after-commit eventsA transactional outbox to Kafka, keyed by player for ordering, consumed idempotently
Refresh-token cleanupNonePartition by creation time and drop old partitions

6.4 Auth at scale

7. Cross-cutting concerns at scale

Orchestration

Kubernetes; autoscale detectors and the Judge on consumer lag (for example with KEDA), Arena on CPU and latency; pod disruption budgets so rolling deploys don't trigger mass rebalances.

Contracts

A schema registry with compatibility checks (backward or full) for every topic; contract tests for the HTTP APIs between services.

Observability

Distributed tracing across HTTP and Kafka headers; SLOs (detection latency, verdict latency, API p99); keep metric tags low-cardinality; sample traces.

Testing at scale

Distributed load generation (multiple k6 runners, never on the system under test); chaos experiments as routine (kill brokers, not just consumers); shadow traffic for new detector rules and prompt versions.

Data governance

Per-region data residency; retention policies; player data out of logs (already a rule); right-to-erasure versus an append-only audit (pseudonymise identities rather than delete history).

Cost

Cost per player per day as a first-class metric: broker storage, compute, LLM tokens and human review time.

8. The staged roadmap: 10×, 100×, 1,000×

StageTelemetryJudgeArena
10×
10k players, ~250k events/s
3-broker cluster (RF 3); partitions up front (for example 48); ~10–15 detector instances with headroom; static membershipMore partitions; higher provider limits; expire ProcessedFlags2–3 instances behind a load balancer; PgBouncer; Redis replica; refresh-token cleanup; the refresh race fixed
100×
100k players, ~2.5M events/s raw
Edge aggregation of movement; Avro + schema registry; parse once; integrity proof only on changed players or a canaryClassical triage before the LLM; one case per player per window; a distributed rate limiter; batch API for re-reviews; review prioritisationTransactional outbox → Kafka; time-partitioned tables; percentiles and rebuild as single coordinated jobs; read replicas
1,000×
1M concurrent, 10M registered
Regional pipelines; flags replicated centrally; tiered storage; hundreds of partitions per regionAn ML scoring model in the stream; budget-aware throttling; human-in-the-loop data pipeline improving prompts and triageSplit stores (match history to a wide-column or columnar store); shard player data or apply ratings per shard idempotently; regional leaderboards plus approximate global ranks; RS256 at a gateway
A two-minute answer you can give

"At a million concurrent players, the simulator's numbers say about 25 events per player per second, which is 25 million events a second. Before adding machines I'd cut the volume: movement is 80% of events and the game server can summarise it per second, and a binary schema shrinks each event about threefold. That brings ingest to roughly half a gigabyte per second, a normal Kafka cluster with partitions sized up front and regional pipelines near the game servers. The Judge hits the provider's rate limits and the budget long before CPU, so the plan there is to judge fewer flags: better detector precision, a cheap classical triage, one case per player, and the LLM only for the ambiguous middle. Arena's bottleneck is a single PostgreSQL primary taking about 17,000 row writes a second and 720 million match-participant rows a day, so I'd time-partition and archive, move match history to an append-optimised store, put the outbox in front of Redis, and add read replicas and PgBouncer."

9. Interview questions

What breaks first in Telemetry if you add 1,000× players? Intermediate

Raw volume. 25 million JSON events a second at ~275 bytes is ~6.9 GB/s of ingest, and roughly 3,300 stream threads at the throughput measured on my laptop. Before scaling the cluster, I'd reduce the work: aggregate movement at the game server (80% of events), switch to Avro or Protobuf, and parse once. Then size partitions up front and run regional pipelines. The integrity processor would also need redesigning, because re-emitting every player every 5 seconds becomes 200,000 records per second on its own.

How many partitions would you choose, and why decide up front? Intermediate

Partitions cap consumer parallelism (one stream task per partition), so I'd divide the expected event rate by measured per-thread throughput, add headroom for failover and growth, and round up to a generous power of two: hundreds to about a thousand at the 1M-player stage. It has to be decided up front because adding partitions later remaps keys, breaking per-player ordering and the co-partitioned join with player profiles.

How do you keep LLM costs under control at scale? Deep

Mostly by sending fewer and cheaper requests: improve upstream precision so fewer honest players are flagged; merge flags into one case per player per window; put a cheap classical model in front so only ambiguous cases reach the LLM; keep model routing and prompt caching, since the cached prefix is shared across all flags; use the batch API for non-urgent work; and enforce a budget with cost metrics, alerts and throttling. The cost model is simply flags per day times cost per verdict, so I reduce both factors.

Your Resilience4j rate limiter is per JVM. What happens with 20 Judge instances? Deep

Each instance would allow the full per-minute budget, so together they'd exceed the provider's limit by 20× and get 429s, which the retry handles but wastes time and capacity. I'd use a shared token bucket (for example in Redis), or split the quota explicitly between instances, and keep honouring Retry-After as the backstop.

What's the human-review problem at scale? Curveball

Any review percentage times millions of flags is more than any team can handle: even 2% of ~15 million flags a day is ~300,000 reviews. So review becomes a prioritised, sampled process: the highest-impact cases first, batches of similar cases, random audits of automatic decisions to measure quality, and every human decision fed back as labelled data to improve prompts and triage, which lowers the review rate over time.

Where does Arena's database break, and what would you do? Deep

At ~833 matches a second, a single PostgreSQL primary takes ~17,000 row writes a second, and match_participants grows by ~720 million rows a day. I'd time-partition the append-heavy tables and archive old partitions, move match history to an append-optimised store, keep the small hot data (players, ratings, statuses) in PostgreSQL, add PgBouncer and read replicas, and eventually shard by player, with ratings applied per shard idempotently from a match event.

Which background jobs would you change first? Intermediate

The percentile refresh, which every instance runs as three full sorts every minute: make it one scheduled job with a distributed lock that stores the cut points centrally, or use incremental approximate quantiles. The leaderboard rebuild, which every instance may run at startup: make it a coordinated single job. And add the missing refresh-token cleanup, ideally by dropping old time partitions. At 1,100 refreshes a second the table would grow by ~95 million rows a day.

How would you go multi-region? Deep

Keep the heavy, latency-sensitive work local: each region runs its own game servers, Kafka and detectors, because detection only needs one player's events. Replicate only the small outputs (flags) to a central cluster for the Judge and reporting. For Arena: regional match recording, a global identity service, regional leaderboards plus an approximate global ranking, and asynchronous replication with clear rules about which region owns which player's writes.

Why is "do less work" often better than "add machines"? Basic

Scaling out multiplies cost and operational complexity linearly or worse, while removing unnecessary work removes cost everywhere downstream. In Fairplay, summarising movement at the game server cuts event volume by about 75%, which shrinks brokers, detectors, storage and network at once. Filtering flags before the LLM cuts tokens, rate-limit pressure and human reviews at once.

10. Quick quiz

1. At ~25 events per player per second, 1M concurrent players produce about:

1,000,000 × 25.

2. Which single change cuts Telemetry's event volume the most?

POSITION is 20 of ~25 events per player per second.

3. Why must partition counts be planned up front?

hash mod n changes when n changes.

4. What limits the Judge first at scale?

The code is cheap; tokens and humans are not.

5. Fifty Arena instances with 20-connection pools each need:

1,000 PostgreSQL connections means 1,000 server processes.

11. Cheat sheet

  • Method: target in numbers → measured baseline → multiply → first bottleneck → fix (often by doing less) → repeat; name trade-offs.
  • Baselines: ~25 events/player/s; ~275 B per JSON event; ~7–8k events/s per stream thread (laptop); ~10 flags/min per 1,000 players (5% cheaters); Redis 5,764 req/s on a shared laptop.
  • 1M concurrent: 25M events/s raw (~6.9 GB/s); 833 matches/s; ~17.5k DB row writes/s; 720M participant rows/day; ~1,100 token refreshes/s; 35–170 flags/s.
  • Telemetry: edge aggregation, Avro, parse once → ~0.5 GB/s; partitions up front (~1,024); N−1 headroom; RF 3 + tiered storage; integrity only on changes; regional pipelines.
  • Judge: rate limits, budget and humans are the bottlenecks; fewer flags (precision, triage model, one case per player); cheaper flags (routing, caching, batch); more partitions or key-ordered parallelism; distributed limiter; bounded ProcessedFlags; prioritised and sampled human review with a feedback loop.
  • Arena: time-partition and archive; split match history out; outbox for Redis; PgBouncer + read replicas; single-run jobs for percentiles and rebuild; token cleanup; RS256 at a gateway; rate limiting; shard by player last.
  • Cross-cutting: Kubernetes + lag-based autoscaling, a schema registry, tracing and SLOs, distributed load tests, routine chaos, data residency, cost per player.
Saved in this browser.