The Hot Key Problem
Redis is single-threaded per key. A key that receives a disproportionate share of traffic — a "hot key" — becomes a bottleneck regardless of how many Redis nodes you have.
In ad-tech, hot keys are common. A high-spend advertiser's budget key might receive 10,000 writes per second during peak traffic. A popular publisher's impression counter might receive even more.
Pattern 1: Key Sharding
Split one key into N shards. Distribute writes across shards randomly. Aggregate reads across all shards.
func budgetKey(advertiserID string, shards int) string {
shard := rand.Intn(shards)
return fmt.Sprintf("budget:%s:%d", advertiserID, shard)
}func totalBudget(ctx context.Context, advertiserID string, shards int) int64 {
var total int64
for i := 0; i < shards; i++ {
key := fmt.Sprintf("budget:%s:%d", advertiserID, i)
val, _ := rdb.Get(ctx, key).Int64()
total += val
}
return total
}
The trade-off: reads become N times more expensive. For budget pacing, we read infrequently (only for reporting) and write constantly (every auction), so this trade-off is favorable.
Pattern 2: Local Caching with Write Batching
Instead of writing to Redis on every auction, accumulate writes locally and flush in batches.
type BudgetAccumulator struct {
mu sync.Mutex
pending map[string]int64
}func (b *BudgetAccumulator) Decrement(key string, amount int64) {
b.mu.Lock()
b.pending[key] += amount
b.mu.Unlock()
}
func (b *BudgetAccumulator) Flush(ctx context.Context) {
b.mu.Lock()
pending := b.pending
b.pending = make(map[string]int64)
b.mu.Unlock()
pipe := rdb.Pipeline()
for key, amount := range pending {
pipe.DecrBy(ctx, key, amount)
}
pipe.Exec(ctx)
}
This reduces Redis write frequency by 10-100x. The trade-off is that budget state can be stale by up to one flush interval (we use 100ms).
Pattern 3: Read Replicas for Hot Reads
For keys that are read frequently (e.g., feature flags, configuration), use Redis read replicas to distribute read load. Writes still go to the primary, but reads are distributed.
What We Actually Did
For budget keys, we used key sharding (8 shards per advertiser) combined with write batching (100ms flush interval). This reduced peak write load on any single key from ~10,000/s to ~125/s — well within Redis's single-key throughput.
The 100ms staleness in budget state was acceptable — it means we might over-deliver by at most 100ms worth of spend, which is negligible.