Fairplay Interview Prep
Module 08

Redis, and keeping a cache consistent

Arena's leaderboard went from an 18-second p95 in PostgreSQL to 63 ms in Redis, on a million players. This module explains why: Redis data structures, how a sorted set computes a rank in O(log N), and then the hard part every interviewer will push on: keeping Redis consistent with the database when two systems are written separately.

P1· about 1.5 hours

By the end you can answer

  • "What is Redis and why is it fast?"
  • "How does a sorted set work, and what's the complexity of a rank lookup?"
  • "How do you keep Redis consistent with PostgreSQL? What is the dual-write problem?"
  • "Why update Redis after commit? What can still go wrong?"
  • "How do you rebuild a live leaderboard without readers seeing a half-built one?"
  • "What is pipelining? How is it different from MULTI/EXEC and Lua?"
  • "What's the transactional outbox pattern?"

1. What Redis is

Redis is an in-memory data-structure server. Clients send commands over TCP (ZADD, GET, INCR...) and Redis operates on data held in RAM. It is fast because:

Typical uses: caching, leaderboards, rate limiters, session stores, counters, distributed locks, queues and streams. The trade-off: data is limited by RAM, and durability depends on configuration (section 9).

Data types at a glance

TypeThink of it asTypical use
Stringkey → bytesCached JSON, counters (INCR), flags with expiry
Hashkey → small mapAn object's fields
ListLinked listSimple queues, recent items
SetUnordered unique membersTags, "who is online"
Sorted set (ZSET)Unique members, each with a numeric score, kept orderedLeaderboards, priority queues, time-ordered indexes, sliding-window rate limits
StreamAn append-only log with consumer groupsLightweight event streaming

2. Sorted sets, and how rank is O(log N)

A sorted set maps member → score. Arena uses the key leaderboard:global, with members as player UUID strings and scores as ratings. Members are kept ordered by score, and ties are broken by comparing the member bytes.

Internally (for anything beyond a small set) Redis keeps two structures in sync:

Analogy: the skip list as express trains

Picture a metro line where the local train stops at every station (level 0: every member in order). An express stops at every ~2nd station, a super-express at every ~4th, and so on, with levels chosen randomly. To reach station 700,000 you ride the top express as far as you can without overshooting, drop a level, and repeat. You visit about log N stops instead of N. Redis also stores, on every express link, how many stations it skips (the "span"). Adding up the spans along your route gives your position, your rank, in O(log N) as well.

CommandComplexityArena use
ZADD key score memberO(log N)Set a player's rating (insert or update): LeaderboardSync.onRatingChanged
ZREM key memberO(log N)Remove a banned player: onRemovedFromLeaderboard
ZSCORE key memberO(1)A player's rating
ZREVRANK key memberO(log N)Rank (0-based, highest score first)
ZREVRANGE key start stop WITHSCORESO(log N + M)One leaderboard page
ZCARD keyO(1)Total ranked players; the arena.leaderboard.size gauge

Arena's two read paths

arena/.../leaderboard/LeaderboardService.java (simplified)
// A page of the leaderboard
Set<TypedTuple<String>> tuples = redisTemplate.opsForZSet()
        .reverseRangeWithScores(leaderboardKey, offset, offset + limit - 1);   // ZREVRANGE ... WITHSCORES
Map<UUID, Player> playersById = playerRepository.findAllById(orderedIds) ...; // ONE query for names

// One player's rank
Long zeroBasedRank = redisTemplate.opsForZSet().reverseRank(leaderboardKey, member);  // ZREVRANK
Double score       = redisTemplate.opsForZSet().score(leaderboardKey, member);        // ZSCORE
Long zCard         = redisTemplate.opsForZSet().zCard(leaderboardKey);                // ZCARD
// rank = zeroBasedRank + 1

Redis holds only what is needed to order (id and rating). Display data (username, display name) stays in PostgreSQL and is fetched for the page's ids in one query. Redis is a derived index, not a copy of the database.

A small optimisation to mention

The rank lookup makes three round trips (ZREVRANK, ZSCORE, ZCARD). They could be pipelined into one round trip, and Redis 7.2 added ZREVRANK ... WITHSCORE, which returns rank and score together. Spotting your own possible improvements is a good habit to show.

The measured result

Source (1,000,000 players, 200 virtual users)p50p95p99Throughput
PostgreSQL COUNT(*) + 115.62 s18.01 s29.39 s12.6 req/s
Redis ZREVRANK33.07 ms63.53 ms90.21 ms5,764 req/s

About 280× faster at p95, 460× the throughput. Server-side, the whole lookup (a PostgreSQL eligibility check by primary key plus three Redis commands) averaged 2.9 ms; the rest of the 33 ms the client saw is HTTP, JSON, waiting for a database connection and a request thread, and a load generator sharing the same machine. Method and machine are in Arena's README; module 14 explains how to read percentiles.

3. Spring Data Redis

4. The dual-write problem

Arena writes ratings to PostgreSQL (the truth) and to Redis (the fast index). Any time one operation must update two separate systems, there is no single transaction covering both, and every ordering has a failure mode:

ApproachWhat goes wrong
Update Redis inside the database transactionRedis is updated, then the transaction rolls back (a validation error, a deadlock victim): Redis shows a rating that never existed. Also, Redis latency or an outage now slows or fails database transactions while they hold row locks.
Update Redis first, then the databaseSame phantom-data problem whenever the database write fails.
Update the database, then Redis (after commit)If the process dies, or Redis fails, between the two, Redis is stale (missing an update). This is the least bad option, because the truth is correct and Redis can be repaired from it.

What Arena does: publish an event, apply it after commit

arena/.../leaderboard/LeaderboardSync.java
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void onRatingChanged(RatingChanged event) {
    try {
        redisTemplate.opsForZSet().add(leaderboardKey, event.playerId().toString(), event.rating());
    } catch (Exception ex) {
        syncFailures.increment();                     // metric: arena.leaderboard.sync.failures
        log.warn("Failed to sync rating change to Redis for player {}", event.playerId(), ex);
    }
}
Known limitations: say these before the interviewer does
  1. Lost on crash. Events live in memory. A crash after commit but before the listener runs loses that Redis update.
  2. The startup repair only compares counts. LeaderboardRebuilder skips the rebuild when ZCARD equals the eligible player count, so a missing member is repaired, but a stale score isn't.
  3. Out-of-order application is possible in theory. Two matches for the same player commit one after the other (the second waited on the row lock). Their after-commit listeners run on different threads; if the first is delayed (say, by a GC pause) until after the second, Redis ends with the older rating. The window is tiny, but it exists.
  4. A rebuild racing live updates. The rebuild reads PostgreSQL in batches, then replaces the live key. An update applied to the live key during the rebuild can be overwritten by data read before it.

Fixes, from simplest to strongest: guard updates with a version or timestamp (a small Lua script that only applies a newer value); run periodic reconciliation that compares scores, not just counts; or use the transactional outbox below, which is Arena's documented next step.

The transactional outbox pattern

 ┌─────────────── one database transaction ───────────────┐
 │ UPDATE players SET rating = 1030 ...                    │
 │ INSERT INTO outbox (aggregate_id, type, payload, seq)   │   ← the event is data, committed atomically
 └─────────────────────────────────────────────────────────┘
            │
            ▼  a relay (poller, or CDC such as Debezium reading the WAL)
     publish to Redis / Kafka, in order per player, retry until done, mark sent
            │
            ▼
     consumers apply idempotently (ZADD absolute score; or check seq)

Because the event is written in the same transaction as the change, it exists if and only if the change committed. The relay delivers it at least once, surviving crashes, and idempotent consumers make duplicates harmless. Publishing the outbox to Kafka keyed by player id would also give per-player ordering, and is the third item on Arena's "what I'd do next" list: "publish moderation and match domain events to a message broker".

How to say it

"PostgreSQL is the source of truth; Redis is a derived index. I only touch Redis after commit, so it can never show data that didn't commit, and a Redis failure can't fail a request. The gap is that in-memory events can be lost on a crash, which I cover with a rebuild on startup. For guaranteed delivery I'd move to a transactional outbox: write the event in the same transaction and relay it at least once to idempotent consumers."

5. Caching patterns you should know by name

PatternHowTrade-off
Cache-aside (lazy loading)Read: try the cache; on a miss, read the database and fill the cache with a TTL. Write: update the database, then delete the cache keySimple and common; the first read after a change is slow; brief staleness possible
Write-throughWrites go to the cache and the database togetherThe cache is always warm; writes are slower; still a dual write
Write-behindWrite to the cache; flush to the database asynchronouslyFast writes; risk of data loss
Derived index / read model (Arena)Maintained from database changes (events); rebuildable from the truthNot a "cache miss" model: the structure (ZSET) must be complete to be correct

Arena's leaderboard is not cache-aside. A ranking is only correct if every eligible player is in the set, so you can't lazily fill it on a miss. That's why it is maintained by events and rebuilt in bulk.

Other terms: TTL (expiry); eviction policy when memory is full (allkeys-lru, volatile-ttl, noeviction); cache stampede (many requests miss at once and all hit the database; fix with locking or early refresh); hot key (one key takes most traffic, a concern for one global leaderboard in a cluster).

6. Rebuilding atomically: temp key + RENAME, with pipelining

arena/.../leaderboard/LeaderboardRebuilder.java (condensed)
long eligibleCount = playerRepository.countEligibleForLeaderboard(ELIGIBLE_STATUSES);
Long currentSize = redisTemplate.opsForZSet().zCard(leaderboardKey);
if (currentSize != null && currentSize == eligibleCount) return;      // looks complete: skip

String tempKey = leaderboardKey + ":rebuild";
redisTemplate.delete(tempKey);
UUID lastId = null;
while (true) {
    List<RankedRow> batch = fetchBatch(lastId);                          // keyset: WHERE id > :lastId LIMIT 10000
    if (batch.isEmpty()) break;
    redisTemplate.executePipelined((RedisCallback<Object>) connection -> {
        for (RankedRow row : batch) {
            connection.zSetCommands().zAdd(tempKeyBytes, row.rating(), row.id().toString().getBytes(UTF_8));
        }
        return null;                                                       // 10,000 ZADDs, one round trip
    });
    lastId = batch.getLast().id();
    if (batch.size() < batchSize) break;
}
redisTemplate.rename(tempKey, leaderboardKey);                             // atomic swap
PipelineMULTI/EXEC transactionLua script (EVAL)
PurposeFewer round tripsRun a block of commands without interleavingAtomic logic with conditions
Atomic?No (other clients' commands may interleave)Yes (queued, then executed together); no rollback on errorsYes (the script runs to completion)
Can branch on values?NoNo (only WATCH for optimistic checks)Yes, e.g. "ZADD only if the new version is higher"

7. Durability and memory

8. Scaling Redis

9. Interview questions

Why is Redis fast? Basic

Data lives in memory; it uses purpose-built structures with known complexity; and commands execute on a single thread, so there's no locking or context switching between them, which also makes each command atomic. Network I/O can be multi-threaded in Redis 6+, but command execution stays sequential.

How does a sorted set give you a rank in O(log N)? Intermediate

It combines a hash table (member to score, O(1) lookups) with a skip list ordered by score then member. A skip list is a layered linked list where higher layers skip over many nodes, so a search visits about log N nodes. Each forward pointer stores its span (how many nodes it skips), so summing spans along the search path gives the rank. ZREVRANK is O(log N), versus O(N) for counting rows ahead in SQL.

Why not just add an index to PostgreSQL instead of using Redis? Intermediate

An index finds rows by value quickly, but rank means "how many rows are ahead of me", and a B-tree doesn't store subtree counts, so the database still has to count the rows ahead, which is O(N) per lookup. PostgreSQL's planner can at best scan the index rather than the table. The measured database path was about 1.5 s server-side per lookup on a million rows. A ranked structure like a skip list with spans (or an order-statistic tree) is the right data structure, and Redis provides it.

Explain the dual-write problem and your solution. Deep

Writing to two systems without a shared transaction means one can succeed while the other fails. Updating Redis inside the database transaction risks phantom data on rollback, and couples database latency to Redis. I publish Spring events during the transaction and apply them with @TransactionalEventListener(AFTER_COMMIT), so Redis only ever sees committed data. A Redis failure is logged and counted but doesn't fail the request, because PostgreSQL is the truth. The remaining risk is staleness: the in-memory event can be lost on a crash, which a startup rebuild covers. The robust fix is a transactional outbox.

What is the transactional outbox pattern? Deep

Instead of publishing an event directly, you insert it into an outbox table in the same database transaction as the business change, so the event exists if and only if the change committed. A separate relay (a poller, or change data capture like Debezium on the WAL) publishes outbox rows to Redis or Kafka, retrying until acknowledged, then marks them sent. Delivery is at-least-once, so consumers must be idempotent. ZADD with an absolute score already is. It removes the "lost on crash" gap and gives ordering per key if you partition by player.

How do you rebuild the leaderboard without users seeing a partial one? Intermediate

Build the new sorted set under a temporary key, streaming players from PostgreSQL with keyset pagination in batches of 10,000 and pipelining the ZADDs, then RENAME the temp key over the live key. RENAME is one atomic command, so readers see either the complete old set or the complete new one. It also skips the rebuild when the counts already match. A million rows took 22.6 seconds.

Pipelining vs MULTI/EXEC vs Lua? Intermediate

Pipelining batches commands to save round trips and gives no atomicity. MULTI/EXEC queues commands and executes them as an uninterrupted block, but can't branch on intermediate values (only WATCH for optimistic checks) and doesn't roll back on command errors. A Lua script runs atomically and can contain logic, for example "apply this score only if its version is newer than the stored one", which is how I'd fix out-of-order updates.

Redis loses all its data. What happens to Arena? Intermediate

Leaderboard reads return empty or "not ranked" until restart. At startup, LeaderboardRebuilder sees that ZCARD (0) doesn't match the eligible count and rebuilds from PostgreSQL. No data is lost, because Redis holds only a derived view. That's why Compose gives Redis no persistent volume. In production I'd also trigger a rebuild on detecting an empty key, not only at startup.

Can Redis end up with an older rating than PostgreSQL? Curveball

Yes, in narrow cases: a crash between commit and listener; a Redis error during the listener; or two after-commit listeners for the same player running out of order on different threads. The startup check compares counts, not scores, so it wouldn't catch a stale score. Mitigations: version-guarded updates via a Lua script, periodic reconciliation of scores, or an outbox relay that preserves per-player order. I'd also alert on arena.leaderboard.sync.failures.

How are ties ordered, and how did you keep the database fallback consistent? Deep

Equal scores are ordered by member bytes (lexicographically), and ZREVRANK reverses it, so the larger member string ranks higher. The database rank query counts players with a higher rating, or an equal rating and a larger id::text COLLATE "C". The C collation compares bytes like Redis, so both sources return identical ranks. With about 500 players per rating in the benchmark, this mattered.

How would you scale this to 100 million players? Deep

One ZSET lives on one node, so I'd watch memory and the hot key. Options: replicas for read scaling; shard boards by region or season; keep an exact top-N set and compute approximate ranks below it from per-bucket counts (how many players per rating band); or partition score ranges across nodes and sum the counts above. And I'd move the sync to an outbox via Kafka so many consumers (regional boards, analytics) can subscribe.

10. Quick quiz

1. The complexity of ZREVRANK is:

Skip-list spans give the rank along the search path.

2. Why update Redis in an AFTER_COMMIT listener?

It does not guarantee delivery: a crash can still lose the event.

3. Which gives atomic conditional logic in Redis?

Scripts run atomically and can branch on stored values.

4. RENAME temp live is used because:

Readers never observe a partial rebuild.

5. Why is Arena's leaderboard not a cache-aside cache?

You can't lazily fill a ranking one miss at a time.

6. The outbox pattern makes event publication:

Consumers must still be idempotent.

11. Cheat sheet

  • Redis: in-memory, data structures, single-threaded execution, so each command is atomic.
  • ZSET = hash (member→score, O(1)) + skip list (ordered, spans). ZADD/ZREM/ZREVRANK O(log N), ZREVRANGE O(log N + M), ZCARD O(1). Ties by member bytes.
  • Arena: leaderboard:global, member = UUID, score = rating; names from PostgreSQL in one findAllById.
  • Result: p95 63.5 ms vs 18.0 s; 5,764 vs 12.6 req/s (1M players).
  • Dual write: no shared transaction. Arena: an event during the transaction, applied AFTER_COMMIT; failures logged and counted, the request still succeeds; ZADD absolute is idempotent.
  • Limitations: lost on crash; the startup check compares counts only; possible out-of-order listeners; the rebuild racing live updates. Fixes: a versioned Lua update, reconciliation, the transactional outbox (+ Kafka by player).
  • Rebuild: keyset batches of 10k, pipelined ZADD into a temp key, atomic RENAME; 22.6 s for 1M; skip if counts match.
  • Pipeline (round trips) vs MULTI/EXEC (a block) vs Lua (atomic logic).
  • Durability: RDB/AOF; Arena's Redis is disposable and rebuildable.
  • Scale: replicas, Cluster (one key = one node = a hot key), sharded or bucketed boards.
Saved in this browser.