paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

September 04, 2019

System Design: Caching Strategies for High-Performance Applications

A practical guide to cache-aside, write-through, write-behind, and eviction policies for engineers building systems that scale

Introduction

Every system that grows past its first few thousand users eventually collides with the same wall: the database becomes the bottleneck. Queries that once returned in single-digit milliseconds start taking hundreds, connection pools saturate, and the application that felt instantaneous during development starts to feel sluggish in production. Caching is one of the oldest and most effective tools engineers have for pushing that wall further away, and it remains one of the most misunderstood.

The appeal of caching is obvious: store a computed or fetched result somewhere fast, and avoid repeating expensive work. The difficulty is that caching introduces a second source of truth into your system, and every second source of truth eventually drifts out of sync with the first. This tension - between speed and correctness - is what makes caching a genuine system design problem rather than a simple performance trick. Choosing the wrong strategy doesn't just cost you performance; it can silently corrupt the data your users see.

This article walks through the caching strategies that show up again and again in production systems: cache-aside, read-through, write-through, write-behind, and the eviction and invalidation policies that keep them healthy. The goal is not to present caching as a checklist of patterns to memorize, but to build the reasoning that lets you pick the right strategy for a given access pattern, consistency requirement, and failure mode.

Context: Why Caching Becomes Necessary

Most applications start with a single database serving every read and write. This works well because databases like PostgreSQL or MySQL are genuinely good at what they do - they use indexes, query planners, and buffer pools to keep most operations fast. The problem is not that databases are slow; it's that they are shared, and contention grows non-linearly with load. A query that takes 5ms in isolation can take 200ms when a thousand other queries are competing for the same disk I/O, lock, or connection slot.

Caching addresses this by inserting a faster, simpler storage layer between the application and the database. Systems like Redis or Memcached serve data from memory, which is orders of magnitude faster than disk-backed storage, and they do so without the overhead of SQL parsing, query planning, or transactional guarantees. A cache does less than a database, and that is precisely what makes it fast. The trade-off is durability and consistency: caches are typically not the system of record, and they can be evicted, restarted, or invalidated without the same guarantees a database provides.

There's a second, less obvious reason caching matters: it reduces the blast radius of expensive operations. If computing a user's dashboard requires joining five tables and running an aggregation, doing that computation once and reusing the result for the next hundred requests is not just faster - it protects the database from being overwhelmed by redundant work. This is especially important for read-heavy workloads, where the same data is requested far more often than it changes. Understanding this ratio, sometimes called the read/write ratio of a dataset, is often the first real signal that caching will help.

Deep Technical Explanation: Core Caching Strategies

There are a handful of strategies that define how an application, its cache, and its backing store interact. Each answers a different question: who is responsible for populating the cache, and who is responsible for keeping it consistent with the source of truth?

Cache-aside (lazy loading) is the most common pattern in practice. The application checks the cache first; on a miss, it reads from the database, then writes the result into the cache before returning it. Subsequent reads hit the cache directly until the entry expires or is invalidated. This pattern is popular because it's simple to reason about and resilient to cache failures - if the cache goes down entirely, the application can still serve requests by falling back to the database, just more slowly. The downside is that the first request after a miss always pays the full latency cost, a phenomenon known as a cold cache or, at scale, a "thundering herd" when many requests miss simultaneously.

Read-through caching moves the population logic into the caching layer itself, so the application only ever talks to the cache, and the cache is responsible for fetching from the database on a miss. This centralizes the loading logic, which is useful when many services share the same cache, but it requires the caching layer to understand how to fetch data, which increases coupling.

Write-through caching writes to the cache and the database synchronously, as part of the same operation. This keeps the cache consistent with the database at all times, at the cost of added write latency, since every write has to succeed in both places before the operation completes. It suits workloads where read-after-write consistency matters and write volume is manageable.

Write-behind (write-back) caching writes to the cache immediately and defers the database write to an asynchronous process, batching or delaying it. This dramatically reduces write latency and database load, but introduces risk: if the cache fails before the deferred write completes, data can be lost. Systems using write-behind caching generally need a durable queue or write-ahead log to make this safe, which is why it appears more often in specialized systems - like write-heavy analytics pipelines - than in general-purpose web applications.

Underneath all of these strategies sits the question of eviction: when the cache is full, or when data becomes stale, how do you decide what to remove? The three dominant approaches are TTL-based expiration (entries expire after a fixed duration), LRU (Least Recently Used) eviction (remove the entry that hasn't been accessed in the longest time), and LFU (Least Frequently Used) eviction (remove the entry accessed the fewest times). Redis, for example, supports several eviction policies including allkeys-lru, volatile-lru, and allkeys-lfu, letting engineers tune eviction behavior to their access patterns rather than relying on a single default.

Implementation: Practical Examples

Abstract descriptions of caching strategies are useful, but the details that matter most - race conditions, TTL jitter, serialization overhead - only become visible in code. The following example implements a cache-aside pattern in TypeScript using Redis, including a defense against the thundering herd problem through a short-lived lock.

import { createClient, RedisClientType } from "redis";

type FetchFn<T> = () => Promise<T>;

class CacheAsideRepository {
  private redis: RedisClientType;
  private readonly defaultTtlSeconds = 300;

  constructor(redis: RedisClientType) {
    this.redis = redis;
  }

  async getOrLoad<T>(
    key: string,
    fetchFn: FetchFn<T>,
    ttlSeconds: number = this.defaultTtlSeconds
  ): Promise<T> {
    const cached = await this.redis.get(key);
    if (cached !== null) {
      return JSON.parse(cached) as T;
    }

    // Attempt to acquire a short-lived lock to prevent a thundering herd
    // of concurrent requests all missing the cache and hitting the DB at once.
    const lockKey = `lock:${key}`;
    const acquiredLock = await this.redis.set(lockKey, "1", {
      NX: true,
      EX: 5,
    });

    if (!acquiredLock) {
      // Another request is already populating this key; wait briefly and retry.
      await new Promise((resolve) => setTimeout(resolve, 50));
      return this.getOrLoad(key, fetchFn, ttlSeconds);
    }

    try {
      const value = await fetchFn();
      // Add jitter to TTL to avoid synchronized mass expirations.
      const jitter = Math.floor(Math.random() * 30);
      await this.redis.set(key, JSON.stringify(value), {
        EX: ttlSeconds + jitter,
      });
      return value;
    } finally {
      await this.redis.del(lockKey);
    }
  }
}

This implementation illustrates two subtle but important details. First, the lock prevents dozens of concurrent requests from all missing the cache simultaneously and independently hammering the database - a scenario that becomes common when a popular key expires under load. Second, the TTL jitter avoids a different failure mode: if many keys are set with identical TTLs, they tend to expire at the same moment, causing a synchronized spike in database load. Adding a small random offset spreads that load out.

The following Python example demonstrates write-through caching with a simple abstraction over a database and a Redis client, ensuring that a write is only considered successful if both operations succeed.

import json
import redis
from typing import Any, Optional

class WriteThroughStore:
    def __init__(self, db_connection, redis_client: redis.Redis, ttl_seconds: int = 600):
        self.db = db_connection
        self.cache = redis_client
        self.ttl_seconds = ttl_seconds

    def get(self, key: str) -> Optional[Any]:
        cached = self.cache.get(key)
        if cached is not None:
            return json.loads(cached)

        row = self.db.query_one("SELECT data FROM records WHERE id = %s", (key,))
        if row is None:
            return None

        self.cache.setex(key, self.ttl_seconds, json.dumps(row["data"]))
        return row["data"]

    def set(self, key: str, value: Any) -> None:
        # Write to the database first; only cache on confirmed success.
        self.db.execute(
            "INSERT INTO records (id, data) VALUES (%s, %s) "
            "ON CONFLICT (id) DO UPDATE SET data = EXCLUDED.data",
            (key, json.dumps(value)),
        )
        self.cache.setex(key, self.ttl_seconds, json.dumps(value))

Notice the ordering in set: the database write happens first, and the cache is only updated after the database confirms success. This ordering matters. If the cache were updated first and the database write failed, subsequent reads would serve incorrect data from the cache while believing it to be authoritative - a much harder bug to trace than a failed write that simply surfaces an error to the caller.

Trade-offs and Common Pitfalls

Caching strategies are rarely wrong in isolation; they become problems when applied without considering how they interact with the rest of the system. The most frequent mistake is treating the cache as a source of truth rather than an accelerator. When engineers begin writing business logic that assumes cached data is always fresh, the system becomes fragile in ways that are difficult to detect during testing, because the cache is usually warm and consistent in development and only diverges under real production load and failure conditions.

Stale reads are the most visible consequence of caching. Every strategy that doesn't write through synchronously introduces a window during which the cache and the database disagree. For most applications - product catalogs, user profiles, article content - a few seconds of staleness is a reasonable trade for a significant reduction in database load. For others - account balances, inventory counts during checkout, permission checks - even brief staleness can cause real harm, and those code paths often need to bypass the cache entirely or use much shorter TTLs.

Cache stampedes, mentioned earlier, occur when a popular key expires and many concurrent requests miss simultaneously, each independently querying the database. Without protection - locking, request coalescing, or probabilistic early expiration - this can turn a cache into a load amplifier rather than a load reducer, especially for "hot key" data accessed by a large fraction of traffic.

Cache invalidation is famously one of the two hard problems in computer science, and for good reason. Deciding when to remove or update a cached value in response to a write is straightforward for simple key-value lookups, but becomes genuinely difficult once caches store derived or aggregated data - a computed leaderboard, a search index, a materialized view. In these cases, a single underlying change can invalidate many cache entries, and tracking those dependencies correctly is often more complex than the caching logic itself. Some teams solve this with tagged invalidation, where cache entries are associated with tags that can be invalidated together, but this adds its own bookkeeping overhead.

Finally, memory pressure and eviction under load deserve attention. A cache is not infinite, and under sustained high write volume, eviction policies like LRU can start removing entries that are still valuable, effectively defeating the purpose of caching. Monitoring cache hit rate is essential here - a declining hit rate under constant traffic is often the first sign that the cache is undersized relative to the working set.

Best Practices for Production Caching

Building a caching layer that survives contact with production traffic requires more discipline than implementing the pattern itself. A few practices consistently separate caching systems that hold up under load from those that quietly become a liability.

Start by choosing TTLs based on actual data volatility rather than convenience. Data that changes rarely - configuration, static reference data - can tolerate long TTLs measured in hours. Data tied to user actions - shopping cart contents, session state - usually needs TTLs measured in minutes or less. Resist the temptation to set a single global TTL across all cached data; it almost always ends up too short for some use cases and too long for others.

Instrument cache hit rate, latency, and eviction counts from day one. Without these metrics, a caching layer is a black box, and diagnosing why performance degraded - whether from a cold cache after deployment, an undersized memory allocation, or a hot key overwhelming a single Redis shard - becomes guesswork. Most caching systems, including Redis, expose these metrics natively; the work is in wiring them into your existing observability stack and setting alerts on meaningful thresholds, not just raw numbers.

Design for cache failure explicitly. A cache should be an optimization, not a dependency the system cannot function without. Applications should degrade gracefully - with higher latency, not errors - when the cache is unavailable. This means avoiding code paths that assume a cache read will always succeed, and load-testing the system with the cache disabled to confirm the database can absorb full traffic, even if only temporarily and at reduced performance.

Finally, be deliberate about what you cache. Not all data benefits equally from caching; the ideal candidates are read-heavy, expensive to compute, and tolerant of some staleness. Caching data that changes on every read, or that is accessed only once per user session, adds complexity and memory overhead without meaningful performance benefit. Profiling actual access patterns - rather than assuming which data is "hot" - consistently produces better caching decisions than intuition alone.

Key Takeaways

Conclusion

Caching is deceptively simple to introduce and genuinely difficult to get right. The mechanics - check a fast store before a slow one, write to both when necessary - are easy to implement in an afternoon. The hard part is reasoning about consistency, failure modes, and the ways a cache's behavior under load differs from its behavior in development. Every strategy discussed here - cache-aside, write-through, write-behind, and the eviction policies that govern them - represents a different point on the trade-off between speed, consistency, and complexity.

There is no universally correct caching strategy, only the one that fits your data's read/write ratio, your tolerance for staleness, and your system's ability to survive the cache being wrong or unavailable. The engineers who build caching layers that hold up in production are not the ones who memorized the most patterns; they're the ones who understood their data well enough to know exactly which trade-offs they were accepting, and who built in the observability to catch it when those trade-offs stopped holding true.

References

A quick flashcard preview - one example of each question type in this post. This is a demo of what you'll find in the full quiz app.

Flashcard preview - 1 / 9Multiple Choice

multiple choice - advanced - auto-graded

Why is cache invalidation described as especially difficult for derived or aggregated cached data, such as a computed leaderboard or search index?

Choose an answer