Scaling Fairplay to huge numbers
"How would this work with a million players?" is the most common follow-up to any portfolio project. This module answers it with arithmetic, not hand-waving. It starts from what was measured on one laptop, multiplies to a target, finds what breaks first in each project, and lays out a staged plan (10×, 100×, 1,000×) with the specific changes at each stage. All projections are back-of-envelope estimates, labelled as such, with the formulas shown so you can redo them live.
By the end you can answer
- "How would you scale this to a million concurrent players?"
- "What breaks first?" (for each of the three services)
- "How many partitions or brokers would you need? Show me the maths."
- "How do you keep LLM cost and rate limits under control at scale?"
- "How would you go multi-region?"
1. How to answer any scaling question
- Fix the target in numbers: users, requests per second, events per second, data per day, cost per day.
- Start from measured baselines, not guesses. You have real ones.
- Multiply and find the first bottleneck: the resource that hits its limit first (CPU, a single database primary, a hot key, a rate limit, a budget, human reviewers).
- Fix that bottleneck (scale out, partition, cache, reduce the work, change the data structure), then repeat.
- Name the trade-offs: consistency, latency, cost, operational complexity.
Before adding machines, ask whether the work is necessary: send fewer events, smaller events, fewer flags to the LLM, fewer rows to scan. Several of the biggest wins below come from reducing work, not distributing it.
2. What was measured (the starting point)
| Project | Measured on one laptop (Ryzen 5 3600, Docker Desktop) | What it implies (estimate) |
|---|---|---|
| Arena | Redis rank lookups: 5,764 req/s at p95 63.5 ms (200 virtual users, load generator on the same machine); server-side mean 2.9 ms. Rebuild of 1M players in 22.6 s | One Arena instance with Redis serves thousands of reads per second even while sharing its CPU with the load generator |
| Telemetry | ~24,900 events/s from 1,000 players; 3 detector instances × 2 stream threads; after a kill, 2 instances had ~5,000 events/s of headroom | 2 instances ≈ 30,000 events/s maximum, so roughly 7,000–8,000 events/s per stream thread on that machine |
| Telemetry (load shape) | EventFactory: per player per second, 20 POSITION + ~4.5 SHOT + ~0.3 KILL | ~25 events per player per second; JSON events ≈ 250–300 bytes each |
| Telemetry (flags) | 51 flags in a 5-minute run for 1,000 players with 5% cheaters (31 aimbot + 20 speedhack) | ~10 flags per minute per 1,000 players at that cheater rate |
| Judge | No throughput or cost measured (no API key yet). Configured limiter: 120 model calls per minute | Throughput is bounded by the provider's limits and by budget, not by the code |
3. The target: one million concurrent players
Let's size a big but realistic game: 10 million registered players, 1 million playing at the same time, in 10-player matches of 2 minutes. The formulas:
events/s = concurrent players × events per player per second = 1,000,000 × 25 = 25,000,000 /s
bytes/s (raw) = events/s × bytes per event = 25,000,000 × ~275 B ≈ 6.9 GB/s
matches/s = (concurrent players ÷ players per match) ÷ match seconds = (1,000,000 ÷ 10) ÷ 120 ≈ 833 /s
flags/s = players × flag rate = 1,000 × 10/min ≈ 10,000/min ≈ 170 /s (5% cheaters)
≈ 35 /s (1% cheaters)
token refreshes/s = concurrent players ÷ access-token lifetime = 1,000,000 ÷ 900 s ≈ 1,100 /s
Every number above is an estimate built from your own measurements and code. Now let's see what each one does to each service.
4. Telemetry at scale: the event firehose
4.1 What breaks first
At 25 million JSON events per second and ~7,500 events per second per stream thread, you'd need about 3,300 stream threads (25,000,000 ÷ 7,500), and ingesting ~6.9 GB/s, roughly 20 GB/s across the cluster with replication factor 3. It's possible, but very expensive. The first move is to send less:
| Lever | Effect (estimate) | Trade-off |
|---|---|---|
| Aggregate movement at the edge. 20 POSITION events/s are 80% of the volume. The game server (authoritative over movement anyway) can send one summary per second: samples, moving samples, overspeed samples, max speed | 25 → ~6 events per player per second (−75%) | Detection works on 1 s summaries instead of raw samples; the speedhack window aggregates summaries |
| Binary encoding (Avro or Protobuf with a schema registry, already on the roadmap): field names aren't repeated in every record | ~275 B → ~60–90 B per event | Schema management; less human-readable topics |
| Parse once. Today each event is parsed twice (to avoid the repartition trap). With a binary format and a key-preserving processor, parse once | Noticeably cheaper per event | Move to the new KStream#process overloads that KIP-1128 (a draft under discussion as of November 2026) proposes in place of processValues (banned by the spec), once a release ships them |
| Compression (already lz4) with bigger batches | Fewer bytes on the wire | A few milliseconds of latency |
With edge aggregation and binary encoding: ~6 million events/s × ~80 B ≈ ~0.5 GB/s before replication. That's a normal, mid-sized Kafka cluster rather than a heroic one.
4.2 Partitions, instances and brokers
stream threads ≈ events/s ÷ per-thread throughput ≈ 6,000,000 ÷ ~10,000 (cheaper parsing) ≈ 600
partitions ≥ stream threads (one task per partition) → choose 1,024 up front (room to grow)
instances ≈ 600 threads ÷ 4 threads per instance ≈ 150 detector instances (+ headroom for N−1)
brokers ≈ (bytes/s × replication factor) ÷ practical per-broker write throughput
≈ (0.5 GB/s × 3) ÷ ~0.1–0.3 GB/s ≈ 5–15 brokers (measure! disks, network, compression vary)
- Choose partition counts up front. Adding partitions later remaps keys, which breaks per-player ordering and co-partitioning with
player-profiles(module 09). Pick a generous power of two and keep every player-keyed topic at the same count. - Headroom for failure. The laptop run showed N−1 instances couldn't drain the backlog quickly. Size so that losing one availability zone (say a third of instances) still leaves capacity above the input rate.
- Replication factor 3 with
min.insync.replicas=2, rack- or zone-aware placement, and tiered storage to keep 24-hour (or longer) retention cheap.
4.3 State and the integrity proof
- Window stores: each player has up to 5 live windows per rule. At 1M players that's a few million small aggregates, gigabytes at most, spread over hundreds of tasks. Fine for RocksDB.
- Profiles KTable: 10M profiles at ~150 B ≈ 1.5 GB, spread across partitions. Don't switch to a GlobalKTable at this size (every instance would hold all of it).
- The integrity processor doesn't scale as written. It re-emits every player every 5 s: 1M players means 200,000 extra records per second just for integrity. It was designed as a verification harness for a 1,000-player benchmark. At scale: emit only players that changed, emit per-partition totals instead of per-player, or run the full proof in a sampled canary pipeline.
- Failover gets harder with more state. Keep standby replicas, use static membership (
group.instance.id) so rolling restarts don't rebalance, tune warm-up and probing, and evaluate the newer Streams rebalance protocol. Remember: 14.4 s on a laptop was mostly rebalancing, not detection.
4.4 Multi-region
Players play on regional game servers, and detection only needs one player's events, so run a full Telemetry pipeline per region (regional Kafka, regional detectors) close to the game servers. Replicate only the output (flags, small) to a central cluster for the Judge and global reporting, using MirrorMaker 2 or a managed equivalent. Raw events never cross oceans.
5. The Judge at scale: rate limits, cost and humans
5.1 What breaks first: not the code
flags/s ≈ 170 /s (5% cheaters) ≈ 35 /s (1% cheaters)
model calls/flag ≈ 3 (a tool round or two, plus a verdict; occasionally an escalation) — an assumption
model calls/min ≈ 170 × 3 × 60 ≈ 30,600 /min vs the configured limiter: 120 /min → ~250× too low
cost/day = flags/day × cost per verdict
≈ 14.7 million flags/day × c (e.g. if c were one cent, that's ~$147,000 per day — illustrative only)
human reviews/day = flags/day × review share (even 2% of 14.7M ≈ 294,000 reviews/day — no team can do that)
Three limits arrive long before any CPU limit: the provider's rate limits, the budget, and human review capacity. So the scaling strategy is mostly about judging fewer flags, more cheaply.
5.2 Judge fewer flags
- Better detector precision upstream. Per-tier thresholds (already a planned improvement) and an ML scoring model in the stream would remove most honest-player flags before they cost anything.
- One investigation per player per window, not per flag: merge aimbot and speedhack flags for the same player within the cooldown into a single case.
- A cheap classical triage before any LLM: a gradient-boosted model or simple rules on the long-run stats auto-clears obviously honest players and auto-queues obviously cheating ones for the LLM, so only the ambiguous middle costs tokens.
- Score thresholds and priority queues: judge high-score flags first; low-score flags can wait or be sampled.
- The existing free pre-checks (unknown, already banned, recently judged) matter even more at scale.
5.3 Judge more cheaply
- Keep routing (cheap model first) and prompt caching (the cached prefix is shared across all flags, so at high volume the cache stays warm and most input tokens are cache reads).
- Use the provider's batch API for anything not urgent (re-reviews, evaluation runs): cheaper, with higher throughput, at the cost of latency.
- Budget guards: a cost-per-hour metric (already exported as
judge.verdict.cost.usd) with alerts and automatic throttling.
5.4 Throughput mechanics
- Concurrency is capped by partitions today. One record per poll, one listener thread per partition, 6 partitions: at most 6 investigations at once, each taking seconds. Raise
suspicious-playerspartitions (planned up front, as in Telemetry), or process records concurrently within a partition while keeping per-player ordering (a key-ordered parallel consumer). - Rate limiting must become distributed. Resilience4j's limiter is per JVM. With 20 Judge instances, each would allow 120/min. Use a shared token bucket (for example in Redis) or split the provider's quota between instances explicitly.
ProcessedFlagsmust stop growing. It keeps every flag id forever, and startup replays the whole compacted verdicts topic. Expire entries older than the 30-minute window, delete old records with tombstones (or switch to a TTL key-value store), and snapshot state so startup doesn't replay history.- Arena load from tools is modest (~3 GETs per flag, about 500 requests/s at 170 flags/s) and cacheable for the duration of an investigation, which the memoising executor already does.
5.5 The human bottleneck
Any "send to human review" percentage multiplied by millions of flags is beyond any moderation team. At scale you need review prioritisation (highest impact first: top-tier players, high scores, repeat offenders), batching (similar cases reviewed together), sampling (audit a random fraction of auto-decisions to measure quality), and a feedback loop that turns human decisions into labelled evaluation data, which improves the prompts and the triage model, which lowers the review rate. That loop is the long-term scaling plan for an AI moderation system.
6. Arena at scale: writes, reads, growth
6.1 Write load
matches/s ≈ 833 per match : 1 match insert + 10 participant inserts + 10 player row updates (locked) + 10 ZADDs row writes/s ≈ 833 + 8,330 + 8,330 ≈ 17,500 /s (plus index and WAL writes) participant rows/day ≈ 8,330 × 86,400 ≈ 720 million rows/day refresh-token rows/day ≈ 1,100 /s × 86,400 ≈ 95 million rows/day (and nothing deletes them today)
- A single PostgreSQL primary is the bottleneck. Sustained writes of this order on one primary are at the edge of what's practical, and the data growth is the bigger problem. Options, in order of disruption:
- Time-partition the growing tables (
matches,match_participants,moderation_actions,refresh_tokens) with PostgreSQL declarative partitioning; detach and archive old partitions to cheap storage. This also makes the missing token cleanup a cheapDROPof an old partition. - Separate the stores by access pattern: keep the small, hot, transactional data (players, ratings, moderation status) in PostgreSQL; move append-heavy match history to a store built for it (a wide-column database such as Cassandra or ScyllaDB, or a columnar warehouse for analytics).
- Shard by player id (Citus or application-level). The hard part is that a match touches 10 players who may live on different shards, so the per-match transaction becomes distributed. A common answer is to record the match event once (outbox → Kafka) and let each player's shard apply its own rating update idempotently, accepting eventual consistency for ratings.
- Time-partition the growing tables (
- Lock contention stays small: a player is only in one match at a time, so the sorted
FOR UPDATErarely waits.
6.2 Read load and caching
- Redis: ~8,300 ZADD/s plus rank reads is comfortable for one primary (a single Redis node handles on the order of 100,000 simple operations per second). Add read replicas for rank and page reads. A 10M-member sorted set is roughly 1–1.5 GB, fine for one node; at 100M, shard by region or season (module 08).
- Profiles and history: serve GETs from PostgreSQL read replicas, but route "me" reads to the primary right after a write (read-your-writes), or accept replication lag explicitly.
- Connection pools: 50 Arena instances × 20 connections = 1,000 connections, too many PostgreSQL processes. Put PgBouncer (transaction pooling) in front.
6.3 Background jobs that don't scale as written
| Job | Today | At scale |
|---|---|---|
Percentiles (StatDistributionService) | Every instance runs three percentile_cont sorts over all eligible players every 60 s | Run once (a scheduled job with a distributed lock such as ShedLock), store the 101 cut points in Redis or a table; or maintain approximate distributions incrementally (t-digest) |
| Leaderboard rebuild | Every instance checks at startup and may rebuild (1M in 22.6 s; 10M ≈ 4 minutes by extrapolation) | Run as a single coordinated job with leader election; never on every replica's startup |
| Redis sync | In-memory after-commit events | A transactional outbox to Kafka, keyed by player for ordering, consumed idempotently |
| Refresh-token cleanup | None | Partition by creation time and drop old partitions |
6.4 Auth at scale
- JWT verification is cheap (an HMAC), so stateless request handling scales horizontally. If an API gateway or other services verify tokens, move to RS256 with a JWKS endpoint and key rotation (
kid). - bcrypt is deliberately expensive (tens of milliseconds of CPU per hash), so logins and registrations are CPU-bound. Short access tokens plus refresh tokens mean logins are rare; refreshes (~1,100/s) cost a SHA-256 and a few row writes. Add rate limiting at the edge (already on the roadmap) to stop credential stuffing from becoming a CPU denial of service.
7. Cross-cutting concerns at scale
Orchestration
Kubernetes; autoscale detectors and the Judge on consumer lag (for example with KEDA), Arena on CPU and latency; pod disruption budgets so rolling deploys don't trigger mass rebalances.
Contracts
A schema registry with compatibility checks (backward or full) for every topic; contract tests for the HTTP APIs between services.
Observability
Distributed tracing across HTTP and Kafka headers; SLOs (detection latency, verdict latency, API p99); keep metric tags low-cardinality; sample traces.
Testing at scale
Distributed load generation (multiple k6 runners, never on the system under test); chaos experiments as routine (kill brokers, not just consumers); shadow traffic for new detector rules and prompt versions.
Data governance
Per-region data residency; retention policies; player data out of logs (already a rule); right-to-erasure versus an append-only audit (pseudonymise identities rather than delete history).
Cost
Cost per player per day as a first-class metric: broker storage, compute, LLM tokens and human review time.
8. The staged roadmap: 10×, 100×, 1,000×
| Stage | Telemetry | Judge | Arena |
|---|---|---|---|
| 10× 10k players, ~250k events/s | 3-broker cluster (RF 3); partitions up front (for example 48); ~10–15 detector instances with headroom; static membership | More partitions; higher provider limits; expire ProcessedFlags | 2–3 instances behind a load balancer; PgBouncer; Redis replica; refresh-token cleanup; the refresh race fixed |
| 100× 100k players, ~2.5M events/s raw | Edge aggregation of movement; Avro + schema registry; parse once; integrity proof only on changed players or a canary | Classical triage before the LLM; one case per player per window; a distributed rate limiter; batch API for re-reviews; review prioritisation | Transactional outbox → Kafka; time-partitioned tables; percentiles and rebuild as single coordinated jobs; read replicas |
| 1,000× 1M concurrent, 10M registered | Regional pipelines; flags replicated centrally; tiered storage; hundreds of partitions per region | An ML scoring model in the stream; budget-aware throttling; human-in-the-loop data pipeline improving prompts and triage | Split stores (match history to a wide-column or columnar store); shard player data or apply ratings per shard idempotently; regional leaderboards plus approximate global ranks; RS256 at a gateway |
"At a million concurrent players, the simulator's numbers say about 25 events per player per second, which is 25 million events a second. Before adding machines I'd cut the volume: movement is 80% of events and the game server can summarise it per second, and a binary schema shrinks each event about threefold. That brings ingest to roughly half a gigabyte per second, a normal Kafka cluster with partitions sized up front and regional pipelines near the game servers. The Judge hits the provider's rate limits and the budget long before CPU, so the plan there is to judge fewer flags: better detector precision, a cheap classical triage, one case per player, and the LLM only for the ambiguous middle. Arena's bottleneck is a single PostgreSQL primary taking about 17,000 row writes a second and 720 million match-participant rows a day, so I'd time-partition and archive, move match history to an append-optimised store, put the outbox in front of Redis, and add read replicas and PgBouncer."
9. Interview questions
What breaks first in Telemetry if you add 1,000× players? Intermediate
Raw volume. 25 million JSON events a second at ~275 bytes is ~6.9 GB/s of ingest, and roughly 3,300 stream threads at the throughput measured on my laptop. Before scaling the cluster, I'd reduce the work: aggregate movement at the game server (80% of events), switch to Avro or Protobuf, and parse once. Then size partitions up front and run regional pipelines. The integrity processor would also need redesigning, because re-emitting every player every 5 seconds becomes 200,000 records per second on its own.
How many partitions would you choose, and why decide up front? Intermediate
Partitions cap consumer parallelism (one stream task per partition), so I'd divide the expected event rate by measured per-thread throughput, add headroom for failover and growth, and round up to a generous power of two: hundreds to about a thousand at the 1M-player stage. It has to be decided up front because adding partitions later remaps keys, breaking per-player ordering and the co-partitioned join with player profiles.
How do you keep LLM costs under control at scale? Deep
Mostly by sending fewer and cheaper requests: improve upstream precision so fewer honest players are flagged; merge flags into one case per player per window; put a cheap classical model in front so only ambiguous cases reach the LLM; keep model routing and prompt caching, since the cached prefix is shared across all flags; use the batch API for non-urgent work; and enforce a budget with cost metrics, alerts and throttling. The cost model is simply flags per day times cost per verdict, so I reduce both factors.
Your Resilience4j rate limiter is per JVM. What happens with 20 Judge instances? Deep
Each instance would allow the full per-minute budget, so together they'd exceed the provider's limit by 20× and get 429s, which the retry handles but wastes time and capacity. I'd use a shared token bucket (for example in Redis), or split the quota explicitly between instances, and keep honouring Retry-After as the backstop.
What's the human-review problem at scale? Curveball
Any review percentage times millions of flags is more than any team can handle: even 2% of ~15 million flags a day is ~300,000 reviews. So review becomes a prioritised, sampled process: the highest-impact cases first, batches of similar cases, random audits of automatic decisions to measure quality, and every human decision fed back as labelled data to improve prompts and triage, which lowers the review rate over time.
Where does Arena's database break, and what would you do? Deep
At ~833 matches a second, a single PostgreSQL primary takes ~17,000 row writes a second, and match_participants grows by ~720 million rows a day. I'd time-partition the append-heavy tables and archive old partitions, move match history to an append-optimised store, keep the small hot data (players, ratings, statuses) in PostgreSQL, add PgBouncer and read replicas, and eventually shard by player, with ratings applied per shard idempotently from a match event.
Which background jobs would you change first? Intermediate
The percentile refresh, which every instance runs as three full sorts every minute: make it one scheduled job with a distributed lock that stores the cut points centrally, or use incremental approximate quantiles. The leaderboard rebuild, which every instance may run at startup: make it a coordinated single job. And add the missing refresh-token cleanup, ideally by dropping old time partitions. At 1,100 refreshes a second the table would grow by ~95 million rows a day.
How would you go multi-region? Deep
Keep the heavy, latency-sensitive work local: each region runs its own game servers, Kafka and detectors, because detection only needs one player's events. Replicate only the small outputs (flags) to a central cluster for the Judge and reporting. For Arena: regional match recording, a global identity service, regional leaderboards plus an approximate global ranking, and asynchronous replication with clear rules about which region owns which player's writes.
Why is "do less work" often better than "add machines"? Basic
Scaling out multiplies cost and operational complexity linearly or worse, while removing unnecessary work removes cost everywhere downstream. In Fairplay, summarising movement at the game server cuts event volume by about 75%, which shrinks brokers, detectors, storage and network at once. Filtering flags before the LLM cuts tokens, rate-limit pressure and human reviews at once.
10. Quick quiz
1. At ~25 events per player per second, 1M concurrent players produce about:
1,000,000 × 25.
2. Which single change cuts Telemetry's event volume the most?
POSITION is 20 of ~25 events per player per second.
3. Why must partition counts be planned up front?
hash mod n changes when n changes.
4. What limits the Judge first at scale?
The code is cheap; tokens and humans are not.
5. Fifty Arena instances with 20-connection pools each need:
1,000 PostgreSQL connections means 1,000 server processes.
11. Cheat sheet
- Method: target in numbers → measured baseline → multiply → first bottleneck → fix (often by doing less) → repeat; name trade-offs.
- Baselines: ~25 events/player/s; ~275 B per JSON event; ~7–8k events/s per stream thread (laptop); ~10 flags/min per 1,000 players (5% cheaters); Redis 5,764 req/s on a shared laptop.
- 1M concurrent: 25M events/s raw (~6.9 GB/s); 833 matches/s; ~17.5k DB row writes/s; 720M participant rows/day; ~1,100 token refreshes/s; 35–170 flags/s.
- Telemetry: edge aggregation, Avro, parse once → ~0.5 GB/s; partitions up front (~1,024); N−1 headroom; RF 3 + tiered storage; integrity only on changes; regional pipelines.
- Judge: rate limits, budget and humans are the bottlenecks; fewer flags (precision, triage model, one case per player); cheaper flags (routing, caching, batch); more partitions or key-ordered parallelism; distributed limiter; bounded
ProcessedFlags; prioritised and sampled human review with a feedback loop. - Arena: time-partition and archive; split match history out; outbox for Redis; PgBouncer + read replicas; single-run jobs for percentiles and rebuild; token cleanup; RS256 at a gateway; rate limiting; shard by player last.
- Cross-cutting: Kubernetes + lag-based autoscaling, a schema registry, tracing and SLOs, distributed load tests, routine chaos, data residency, cost per player.