Scalability and Availability for Backend Systems

Scalability and Availability

Every high-level design interview and every real production incident eventually comes back to the same two questions: can this system handle more load, and does it keep working when parts of it fail? Scalability and availability are not features you bolt on later — they are architectural decisions baked in from the first component diagram. This guide covers the vocabulary and mechanics a backend engineer with 2-3 years of experience needs to reason about both, and to defend design choices under interview-style questioning.

1. Vertical vs Horizontal Scaling

Vertical scaling (scale up) means adding more resources — CPU, RAM, faster disks — to a single machine. Horizontal scaling (scale out) means adding more machines and distributing load across them. Almost every modern backend system is designed to scale horizontally, because vertical scaling has a hard ceiling and a single point of failure.

Trade-off comparison

Diminishing returns and Amdahl's Law

Horizontal scaling isn't free linear throughput either. If a request has to touch a shared, non-parallelizable resource — a single-writer database, a global lock, a serial log — that portion caps your speedup no matter how many nodes you add. Amdahl's Law formalizes this: if a fraction p of the work is parallelizable and (1 - p) is serial, the maximum speedup with N nodes is:

speedup(N) = 1 / ((1 - p) + p / N)

If 10% of your request path is a serialized write to a single database (p = 0.9), your theoretical maximum speedup caps at 10x, no matter how many app servers you add. This is why "just add more instances" stops working once you hit the database — the real scaling work moves to caching, read replicas, sharding, or async processing.

2. Stateless Service Design

Horizontal scaling only works cleanly if any request can be served by any node. That requires the service tier to be stateless — no request-specific data lives in the memory or local disk of a particular instance between requests.

Where state tries to sneak in

Statelessness is also what makes autoscaling safe: a new instance can join the pool and start serving traffic immediately with zero warm-up beyond process startup, and an instance can be terminated at any time without losing in-flight user context (as long as in-flight requests are drained, which is a separate concern — see failover below).

3. Load Balancers: L4 vs L7

A load balancer distributes incoming traffic across a pool of backend instances. The critical distinction for interviews is which OSI layer it operates at, because that determines what it can see and what decisions it can make.

L4 vs L7 comparison

A minimal L7 config, for concreteness

upstream orders_service {
    least_conn;                          # route to the backend with fewest active connections
    server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
    keepalive 64;                        # reuse connections to upstreams
}
 
server {
    listen 443 ssl;
    server_name api.example.com;
 
    location /orders/ {
        proxy_pass http://orders_service;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_connect_timeout 2s;
        proxy_read_timeout 5s;
        proxy_next_upstream error timeout http_502 http_503;  # retry on next node
    }
}

Load balancing algorithms

4. Reverse Proxies vs Load Balancers vs API Gateways

These three terms get used interchangeably in casual conversation but have distinct primary responsibilities, even though a single piece of software (NGINX, Envoy) often plays more than one role at once.

5. Redundancy and Failover

Redundancy means running more capacity than the minimum needed, specifically so that the failure of any single component doesn't take down the system. Failover is the mechanism that detects a failure and redirects traffic away from it.

Levels of redundancy

Active-passive vs active-active

Graceful degradation over hard failure

Redundancy handles component loss; graceful degradation handles overload or partial dependency failure without a full outage:

• Circuit breakers — stop calling a failing downstream dependency after an error threshold, fail fast instead of piling up threads waiting on timeouts.
• Bulkheads — isolate resource pools (thread pools, connection pools) per dependency so one slow dependency can't starve the whole service.
• Load shedding — reject low-priority requests first when the system is over capacity, to protect the requests that matter.
• Fallback responses — serve stale cached data or a reduced feature set rather than an error page (e.g., show cached recommendations if the personalization service is down).

6. Defining and Measuring Availability

"Highly available" is meaningless without a number attached. Availability is conventionally expressed as a percentage of uptime over a period, and colloquially referred to by its "nines."

The nines table

SLI, SLO, SLA, and error budgets

Availability of a composed system

When a request depends on multiple components in series, availabilities multiply — this is a common interview gotcha.

Overall availability = A1 × A2 × A3 × ... × An   (for components in series)

For example, a request that must succeed through an API gateway (99.99%), an app service (99.95%), and a database (99.95%):

0.9999 × 0.9995 × 0.9995 ≈ 0.9989  →  99.89% overall, NOT 99.95%

Chaining "highly available" components in series reduces overall availability. This is exactly why redundancy at each hop (so each hop's effective availability approaches 100% via failover) matters more as the request path gets longer — and why deeply nested microservice call chains are an availability risk in their own right, independent of any single service's reliability.

For components that are genuinely redundant in parallel (e.g., either replica can serve the request), availability improves:

Availability(parallel) = 1 - [(1 - A1) × (1 - A2)]

Two independent nodes each at 99% availability, deployed in parallel with automatic failover, together achieve 1 - (0.01 × 0.01) = 99.99% — this is the mathematical justification for "just add a standby."

7. Back-of-Envelope Capacity Planning

Scalability decisions should be grounded in numbers, not vibes. A standard capacity estimate for an interview or a real capacity-planning doc:

Given:
  - 10 million daily active users (DAU)
  - Each user makes ~20 requests/day on average
  - Traffic is not uniform: peak = 3x average

Average requests/day = 10,000,000 × 20 = 200,000,000
Average requests/sec = 200,000,000 / 86,400 ≈ 2,315 req/s
Peak requests/sec    = 2,315 × 3 ≈ 6,945 req/s

If each app server instance safely handles 500 req/s at target latency:
  Instances needed at peak = 6,945 / 500 ≈ 14 instances

Add N+2 redundancy for zone loss / rolling deploys → provision ~16-18 instances

8. Production Observations

• Autoscaling reacts to metrics, it doesn't predict them. Scale-out based on CPU or request-rate has a lag (metric collection interval + new instance boot time + warm-up). For predictable traffic spikes (e.g., a sale event at a known time), schedule pre-scaling ahead of the event rather than relying purely on reactive autoscaling.
• Connection pool exhaustion, not CPU, is the most common real-world scaling bottleneck. Doubling app server instances without raising the database's max connection limit (or adding a connection pooler like PgBouncer) just moves the bottleneck to too many connections errors at the database.
• Health checks need to check the right thing. A /health endpoint that returns 200 OK unconditionally provides zero signal. It should verify the dependencies the instance actually needs (DB reachable, cache reachable) without becoming so expensive it causes false failures under load.
• Load balancer timeouts should be shorter than client timeouts, and backend timeouts shorter than load balancer timeouts. Otherwise the client gives up and retries while the backend is still working, doubling load during exactly the period the system is struggling.
• Zone-level redundancy is table stakes; region-level is a deliberate, costly decision. Multi-region active-active systems introduce data replication lag and conflict resolution complexity — don't reach for it unless the business case (regulatory residency, latency to a global user base, disaster recovery RTO) justifies it.
• Every "nines" target implies an operational cost — on-call rigor, deployment safety (canaries, automated rollback), and testing (chaos engineering, game days). Don't commit to 99.99% in a design doc without the operational muscle to back it.

Key takeaways

• Horizontal scaling is the default for stateless service tiers; vertical scaling and replication/sharding carry the data tier.
• Statelessness (externalized sessions, no local-disk dependency, centralized rate limits) is the prerequisite that makes horizontal scaling and autoscaling safe.
• L4 balances connections, L7 balances requests — know which one you're being asked about, since it changes what routing decisions are even possible.
• Redundancy only pays off if health checks and failover actually detect and route around failures fast enough to matter.
• Availability of components chained in series multiplies down; parallel redundancy multiplies failure probabilities down — always distinguish which topology you're computing for.
• SLOs and error budgets turn "how reliable should this be" into a number that governs both incident response and release velocity.
• Capacity numbers should always tie back to a decision (instance count, cache size, shard count) — an estimate that changes nothing wasn't worth doing.

Interview Questions

• What is the difference between vertical and horizontal scaling, and when would you choose one over the other?
• Why does horizontal scaling require the service layer to be stateless? Give three examples of hidden state that breaks it.
• Explain the difference between an L4 and an L7 load balancer. Which would you use for a gRPC service, and why does the connection-vs-request distinction matter there?
• What's the difference between a reverse proxy, a load balancer, and an API gateway?
• How would you design failover for a service with three replicas across two availability zones?
• What's the difference between active-active and active-passive redundancy? What are the trade-offs?
• Define SLI, SLO, SLA, and error budget. How does an error budget change how a team ships features?
• If Service A depends on Service B and Service C in series, and each is individually 99.9% available, what is the overall availability of a request through all three? What does that imply about deep microservice call chains?
• How would you estimate the number of application server instances needed to serve a given peak request rate?
• What is Amdahl's Law, and how does it explain why "just add more servers" eventually stops helping?
• Why is a naive /health endpoint that always returns 200 dangerous in production?
• Describe the difference between graceful degradation and a circuit breaker. How do they work together under partial outage?
• What operational practices are required to responsibly claim a 99.99% availability target?
• How would you detect and mitigate connection pool exhaustion when scaling out an application tier?