Rate Limiting and Throttling: Protecting Systems Under Load

Try it live: See these algorithms in action at the Rate Limiters Simulator — watch token buckets drain, sliding windows shift, and traffic get shaped in real time.

Start With a Black Friday Meltdown

Your e-commerce platform is running a flash sale. Fifty thousand users are refreshing product pages, adding items to cart, and racing to check out.

Then a single scraper — one malicious client — starts hammering the search API at 10,000 requests per second.

There is no rate limiter.

The search service saturates its connection pool. The database CPU hits 100%. The cart service can't fetch inventory. The checkout service can't verify stock. Within 30 seconds, every endpoint is returning 5xx errors. Real customers see spinning spinners. The flash sale becomes a flash outage. Revenue loss: $200,000 in 45 minutes.

A rate limiter would have watched that scraper hit 100 requests in one second — and every subsequent request would have received 429 Too Many Requests while paying customers kept buying.

What Rate Limiting Means

Rate limiting controls how many requests a client, user, tenant, IP, or route can send in a given time period. When the limit is exceeded, the system either rejects the request, delays it, or degrades the response.

Rate Limiting vs Throttling vs Quota vs Backpressure

These four concepts are frequently confused. They solve related but distinct problems.

Why Not Just Trust Clients to Be Nice?

Beginners often ask: "Why not just assume clients will behave reasonably?"

A rate limiter makes decisions at the request boundary, before any expensive work happens. It rejects early, rejects cheaply, and protects downstream resources from ever seeing overload.

Step 1: Choose an Algorithm — Fixed Window Counter

The simplest approach: count requests in fixed calendar windows (e.g., 10:00:00–10:00:59, 10:01:00–10:01:59).

The boundary burst problem: A client can send 100 requests at 10:00:59 and 100 more at 10:01:00 — effectively 200 requests in 2 seconds with no rejection. For some workloads this is acceptable. For others, it defeats the purpose of rate limiting.

Step 2: Sliding Window Log

Store every request timestamp. On each request, remove timestamps older than the window, count remaining ones, and decide.

Step 3: Sliding Window Counter (Approximate)

A hybrid approach: weight the previous window's count by the overlap ratio with the current window. This gives smooth behavior without storing every timestamp.

effective_count =
  previous_window_count * overlap_ratio
  + current_window_count

This is the algorithm behind AWS API Gateway's rate limiting and many production API gateways. It is a strong default for most use cases.

Step 4: Token Bucket

Tokens are added to a bucket at a fixed refill rate. Each request consumes one or more tokens. The bucket has a maximum capacity that limits burst size.

Burst math: A bucket with 100 capacity and 10 req/s refill allows a client to send 100 requests immediately (the burst), then 10 req/s indefinitely. After draining the bucket, it takes 10 seconds to refill to 100 again.

Step 5: Leaky Bucket

Requests enter a queue and leave at a fixed rate. If the queue is full, excess requests are dropped.

When to use leaky bucket:

• Protecting a legacy database with fixed throughput
• Calling a third-party API with strict rate limits
• Processing pipeline where downstream workers have fixed capacity
• Any system where smoothness matters more than responsiveness

Algorithm Selection Guide

Step 6: Distributed Rate Limiting

In a horizontally scaled system, each application instance sees only its own traffic. A client could send 100 req/s to instance A and 100 req/s to instance B — both would allow it, even though the total is 200 req/s.

All instances must share rate limit state.

Redis Design

The key insight: increment, expiry check, and decision must be atomic. A race condition between "check count" and "increment count" lets two requests sneak through when only one should.

Atomicity with Lua

A Lua script running on the Redis server keeps the operation atomic: it reads the current counter, checks whether the limit is exceeded, increments if not, and sets the key expiry — all in one uninterruptible execution. This eliminates the race between checking and incrementing. The script returns the remaining count so the application knows exactly how many more requests the client can send. Redis guarantees no other command executes while a Lua script runs, making this the safest approach for distributed rate limiting.

Key Design Decisions

Local Caching for Hot Paths

For APIs handling 100K+ req/s per instance, every Redis round trip adds latency. A common optimization: cache the rate limit decision locally for a few hundred milliseconds. The application checks an in-memory cache first — if the TTL has not expired, it returns the cached decision without touching Redis. If the cache misses, it falls through to the Redis Lua script, stores the result back in the local cache with a short TTL (50ms), and returns. This reduces Redis load by an order of magnitude. The trade-off: during the 50ms cache window, the instance may allow up to 5% more requests than configured. For most systems, this is acceptable.

This reduces Redis load by an order of magnitude. The trade-off: during the 50ms cache window, the instance may allow up to 5% more requests than configured. For most systems, this is acceptable.

Limit Dimensions

Rate limits are rarely one-dimensional. Good systems apply layered limits:

Step 7: Response Design

A rate-limited client should know exactly what happened and when to retry.

HTTP Status Codes

Standard Headers

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1717305600
Content-Type: application/json

{
  "error": "rate_limit_exceeded",
  "message": "API rate limit exceeded. Retry after 30 seconds.",
  "retry_after_seconds": 30,
  "limit": 100,
  "remaining": 0,
  "reset_at": 1717305600
}

Step 8: Graceful Degradation

When the system is under load, not all features are equally important. Rate limiting should protect critical paths while allowing non-critical features to degrade.

Priority-Based Rate Limiting

Not all requests deserve the same budget. Assign priority levels and enforce different limits.

Common Failure Stories

Story 1: The Fixed Window Flash Sale

An e-commerce site used a fixed window rate limiter: 100 requests per minute. At 10:00:59, the flash sale started. Users hammered the API — 100 requests in the last second of the 10:00 window. At 10:01:00, a fresh window opened. Another 100 requests in the first second. The database received 200 requests in 2 seconds and fell over.

Root cause: Fixed window boundary burst. The rate limiter reported "under limit" for both windows while the database disagreed.

Fix: Switched to sliding window counter. The boundary burst disappeared because the effective rate was calculated continuously, not reset on hard boundaries.

Story 2: The Token Bucket DDoS

A streaming platform used a single token bucket with capacity 10,000 and 100 req/s refill for all API clients sharing a single API key. One misconfigured client sent requests at 200 req/s. The bucket drained from 10,000 to 0 in 50 seconds. Then every other client experienced 429 errors for 100 seconds while the bucket refilled.

Root cause: Shared bucket meant one noisy neighbor starved all others.

Fix: Per-client token buckets. Each client had its own bucket with capacity 1,000 and 10 req/s refill. The noisy neighbor only starved itself.

Story 3: Redis Goes Down, All Requests Pass

A financial API used Redis for distributed rate limiting with "fail open" mode — if Redis was unreachable, the rate limiter allowed the request. A Redis cluster node failed during peak trading hours. The rate limiter silently allowed all requests. The downstream trading engine received 50x normal traffic and crashed.

Root cause: Fail-open was the wrong policy for a financial system. Every rejected trade is better than a crashed system processing half-executed trades.

Fix: Changed to fail-closed for trading endpoints. The API returned 503 during the Redis outage. Trading stopped cleanly instead of crashing.

Story 4: The NAT Trap

A SaaS company rate-limited by IP address at 100 req/min. A university with 5,000 students behind a single NAT IP hit the limit in the first 30 seconds. Every student behind that NAT was blocked for the rest of the minute. Support tickets flooded in.

Root cause: Per-IP rate limiting breaks behind NAT, corporate proxies, and shared infrastructure.

Fix: Rate limit by authenticated user ID for logged-in traffic, and by IP only for anonymous traffic. Added a note: "IP-based limits are a coarse control for unauthenticated traffic."

Story 5: Clock Skew Chaos

A distributed rate limiter used each application server's local clock to compute time windows. Three app servers had clock skews of 2–5 seconds. A client hitting Server A (fast clock) would get rejected, then retry and hit Server B (slow clock), which still had capacity. The effective rate was double the configured limit.

Root cause: Distributed systems need a shared time source for rate limiting windows.

Fix: Used Redis TIME command (which returns the Redis server's clock) instead of time.time() from each app server.

Evaluation: How to Test Rate Limiters

Test Scenarios

Metrics to Track

Load Testing Approach

Send traffic at a controlled rate and measure how many requests are accepted versus rejected. Use a concurrency-limited client that sends exactly limit requests per second for a sustained duration, then compare the effective throughput against the configured limit. The test should verify that every request under the threshold passes, every request above it gets a 429, and recovery after the window reset works correctly. Run the same test from multiple machines to validate distributed consistency — the combined rate from all machines should not exceed the limit, even though each machine routes through different application instances.

End-to-End Protection Flow

Interview Prep

Rate limiting and throttling is a favorite interview topic — especially system design interviews for APIs, distributed systems, and infrastructure roles.

Common Interview Questions

1. Design the rate limiter for a public API. Start with requirements: 10K req/s, 1M clients, free and paid tiers. Walk through algorithm choice, distributed state (Redis), failure modes, and response headers.

2. How would you rate-limit a multi-tenant SaaS platform? Discuss per-tenant limits, shared vs isolated buckets, noisy neighbor protection, and how to enforce tenant isolation at the database level.

3. Your rate limiter is adding 10ms latency. How do you fix it? Local caching with short TTL, batching Redis calls, reducing Lua script complexity, moving rate limiting to the API gateway (L7 proxy).

4. What happens when Redis goes down? Fail-open vs fail-closed trade-off. Per-endpoint policies. Circuit breaker pattern for Redis calls. Local fallback counters as last resort.

5. Compare token bucket and leaky bucket. Token bucket allows bursts up to capacity, refills steadily. Leaky bucket enforces steady output, queues bursts. Token bucket is better for APIs, leaky bucket for downstream processing pipelines.

6. How do you rate-limit across data centers? Global Redis cluster with CRDT-based counters (e.g., Redis Enterprise with active-active geo-replication). Or per-region limits with a shared global budget that syncs asynchronously.

Key Takeaways

1. Choose the algorithm by the behavior you need: fixed window is simple and cheap, token bucket supports bursts with long-term control, leaky bucket smooths traffic for fragile downstreams, sliding window gives the most precise window enforcement.
2. Distributed limits need shared state and atomic operations: Redis + Lua scripts handle both. Never check-then-set without a lock or atomic operation.
3. Limit by multiple dimensions: IP, user, tenant, route, and global. Order checks from cheapest to most expensive.
4. Return useful headers: 429 + Retry-After + rate limit headers. Give clients enough information to self-regulate.
5. Fail gracefully: Decide per-endpoint whether to fail open or fail closed. Protect critical paths and shed non-essential work under load.
6. Test your rate limiter: Boundary bursts, distributed consistency, Redis failure, and recovery scenarios.