Basic System Concepts: Latency, Throughput, Scaling, Load Balancing, and Caching

The Building Blocks of System Design

Every system design decision ultimately comes down to trade-offs between a few fundamental concepts. Understanding these deeply is what separates a good engineer from a great one.

Let's start with the numbers that every system designer should have memorized.

---

Latency Numbers Everyone Should Know

These approximate numbers (from Peter Norvig's famous slide and Jeff Dean's talk) give you an intuition for the relative cost of different operations:

Latency vs Throughput

These two terms are often confused:

• Latency — How long it takes to do one operation (measured in time: ms, µs, ns)
• Throughput — How many operations you can do per unit of time (measured in ops/sec, MB/s, req/s)

┌────────────────────────────────────────────┐ │ Highway Analogy │ ├────────────────────────────────────────────┤ │ │ │ Latency = Time for one car to go │ │ from A to B │ │ ───────🚗──────────> │ │ │ │ Throughput = Number of cars passing │ │ a point per hour │ │ 🚗🚗🚗🚗🚗🚗🚗🚗🚗🚗→ │ │ │ │ You can have low latency + low throughput │ │ (one-lane highway, no traffic) │ │ │ │ Or high latency + high throughput │ │ (10-lane highway, but A to B is far) │ └────────────────────────────────────────────┘

---

Vertical vs Horizontal Scaling

When your system can't handle the load, you have two options:

Vertical Scaling (Scale Up)

Buy a bigger machine. More CPU, more RAM, more disk.

Single Server ┌──────────────────────────┐ │ │ │ CPU: 4 cores → 64 cores │ │ RAM: 16 GB → 512 GB │ │ Disk: 500 GB → 10 TB │ │ │ └──────────────────────────┘

Pros:

• Simple — no code changes needed
• No distributed systems complexity
• Works for databases that don't shard easily (PostgreSQL scales very well vertically — see Citus Data's research)

Cons:

• Hardware has a ceiling (you can't buy infinite RAM)
• Expensive — the cost curve is non-linear
• Single point of failure
• Downtime required for upgrades

Horizontal Scaling (Scale Out)

Add more machines. Distribute the load across them.

Pros:

• Theoretically infinite scaling
• Commodity hardware = cheaper
• Fault-tolerant (one node fails, others take over)

Cons:

• Distributed systems complexity (consistency, coordination, debugging)
• Requires architectural changes (statelessness, data partitioning)
• More operational overhead

The Decision Matrix

Real-World Scaling Stories

• Instagram: Ran one PostgreSQL instance for 5+ years before moving to sharding. They proved that vertical scaling with a well-tuned database goes much further than most people think.
• Twitter (X): Migrated from monolithic Ruby on Rails to JVM-based services as horizontal scaling became necessary for their real-time feed.
• Slack: Grew from a single MySQL instance to a sharded architecture as message volume exploded — a classic "start vertical, go horizontal when forced" story.

---

Load Balancing

Once you have multiple servers, you need a way to distribute traffic across them. That's what a load balancer does.

What Is a Load Balancer?

A load balancer sits between clients and your servers, distributing incoming requests across a pool of backend machines.

Layer 4 vs Layer 7 Load Balancing

Load Balancing Algorithms

Consistent Hashing

Consistent hashing is one of the most important algorithms in distributed systems. It maps both servers and data to positions on a hash ring.

When Node A is added: only keys between Node C and Node A move. When Node B is removed: its keys transfer to the next node clockwise.

Unlike simple modulo hashing (key % N), consistent hashing ensures that when you add or remove a server, only a fraction of keys need to be redistributed.

Load Balancers in Practice

---

Caching Basics

Caching is the most effective technique for improving read performance. The core idea: store expensive-to-compute or expensive-to-fetch results in fast storage, and serve them from there on subsequent requests.

Why Cache?

If latency to your database is 10ms, but latency to an in-memory cache (Redis/Memcached) is 0.5ms, caching gives you a 20x speedup for read operations.

Where to Cache

Cache Invalidation Strategies

There are only two hard things in computer science: cache invalidation and naming things. — Phil Karlton

Cache Aside Pattern (Most Common)

Read Request: 1. Check cache for key 2. If HIT → return cached value 3. If MISS → query database 4. Store result in cache (with TTL) 5. Return value Write Request: 1. Update database 2. Delete cache entry (NOT update it — delete!) 3. Next read will be a cache miss and repopulate

Real-World Caching Patterns

• GitHub: Uses a multi-layer caching strategy with Memcached and Redis. Their blog has deep dives on reducing database load through caching.
• Reddit: Relies heavily on Memcached for caching — their open-source architecture has influenced many caching patterns.
• Pinterest: Built a custom caching layer on top of Redis to handle their social graph. Their engineering blog has excellent posts on caching at scale.

---

What to Remember for Interviews

1. Latency numbers — Memorize the orders of magnitude: RAM (100ns) < SSD (150µs) < same-DC network (500µs) < cross-DC (80ms)
2. Latency vs Throughput — Latency is time per operation; throughput is operations per time
3. Scale up vs scale out — Start vertical, go horizontal when forced
4. Load balancing — L4 (fast, dumb) vs L7 (slower, smart); know consistent hashing
5. Caching — Know the cache aside pattern, TTL vs LRU, and the golden rule: delete on write, don't update