Distributed Locks and Rate Limiting with Redis
A staff-engineer guide to Redis-based distributed locks, Redlock trade-offs, and token bucket and sliding window rate limiting.
Distributed Locks and Rate Limiting
Once a backend runs on more than one instance, "just use a synchronized block" stops working — you need coordination that spans processes and machines. This guide covers the two most common Redis-backed coordination primitives: distributed locks for mutual exclusion across instances, and rate limiters for protecting shared resources and APIs from being overwhelmed. Both are deceptively simple to build wrong and worth understanding at the level of what can actually go wrong under real failure conditions.
1. Why You Need a Distributed Lock
A single JVM's synchronized, ReentrantLock, or java.util.concurrent primitives only coordinate threads within that one process. The moment you run 3 replicas of a service behind a load balancer, two different instances can both believe they're the only one processing order #5001 — because from each instance's perspective, no other thread is touching it.
Common use cases: preventing duplicate processing of a queued job across worker instances, ensuring only one instance runs a scheduled cron-style task, serializing access to a non-transactional external API, and guarding critical sections that touch non-database shared state (files, caches).
2. The Basic Redis Lock: SET NX PX
The building block of every Redis lock is a single atomic command: set a key only if it doesn't already exist, with an expiry.
SET lock:order:5001 "worker-7-uuid-abc123" NX PX 10000NX— only set if the key does not already exist (this is the mutual exclusion)PX 10000— expire after 10,000ms (this prevents a crashed holder from locking the resource forever)- The value (
worker-7-uuid-abc123) is a unique token identifying the holder, not just"1"— this matters for safe release.
@Component
public class RedisLockService {
private final StringRedisTemplate redisTemplate;
private static final String UNLOCK_SCRIPT =
"if redis.call('get', KEYS[1]) == ARGV[1] then " +
" return redis.call('del', KEYS[1]) " +
"else " +
" return 0 " +
"end";
private final RedisScript<Long> unlockScript =
RedisScript.of(UNLOCK_SCRIPT, Long.class);
public Optional<String> tryLock(String lockKey, Duration ttl) {
String token = UUID.randomUUID().toString();
Boolean acquired = redisTemplate.opsForValue()
.setIfAbsent(lockKey, token, ttl);
return Boolean.TRUE.equals(acquired) ? Optional.of(token) : Optional.empty();
}
public boolean unlock(String lockKey, String token) {
Long result = redisTemplate.execute(
unlockScript, Collections.singletonList(lockKey), token);
return result != null && result == 1L;
}
}Never release a lock with a plain DEL. If your operation took longer than the TTL, Redis may have already expired the lock and handed it to another holder. A bare DEL from the original (slow) holder would then delete the new holder's lock — two instances end up believing they hold exclusive access simultaneously. The check-token-then-delete must be atomic, which is why it's done via a Lua script (EVAL) rather than a GET followed by a separate DEL from application code.
Why the unlock must be a Lua script
Using GET then DEL as two separate round trips has the same race — another client could acquire the lock in between. The Lua script runs atomically on the Redis server, so the check-and-delete can't be interleaved with another client's operations.
Using a resilience library instead of hand-rolling
For production Spring Boot services, prefer a battle-tested library over hand-rolled lock logic — Redisson is the most common choice and implements the correct token-based unlock, auto-renewal (watchdog), and even Redlock out of the box.
@Service
public class OrderProcessingService {
private final RedissonClient redissonClient;
public void processOrder(String orderId) {
RLock lock = redissonClient.getLock("lock:order:" + orderId);
boolean acquired = false;
try {
// Wait up to 2s to acquire, hold for max 10s unless renewed by the watchdog
acquired = lock.tryLock(2, 10, TimeUnit.SECONDS);
if (!acquired) {
throw new LockAcquisitionException("Could not acquire lock for order " + orderId);
}
doProcessOrder(orderId);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new LockAcquisitionException("Interrupted while acquiring lock", e);
} finally {
if (acquired && lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
}
}Redisson's watchdog auto-extends a lock's TTL while the holder is still alive and hasn't called unlock() — as long as you don't pass an explicit lease time (tryLock(waitTime, unit) with no lease), it renews every ~1/3 of the default 30s lease. This solves the "operation ran longer than expected TTL" problem without you needing to guess a safe TTL upfront.
3. Redlock: Multi-Instance Locking
A single Redis instance is a single point of failure for your lock — if that node crashes before replicating the key to a replica, and a replica gets promoted, the new primary doesn't know the lock exists, and a second client can acquire it. Redlock is an algorithm (proposed by Redis's original author) for acquiring a lock across N independent Redis instances to tolerate this.
The Redlock algorithm
- Get the current time.
- Try to acquire the lock (same key, same random token) on all N independent Redis instances sequentially, using a small timeout per instance well below the lock's TTL.
- Compute elapsed time. The lock is considered acquired only if the client got it on a majority (N/2 + 1) of instances, and the elapsed time is less than the lock's validity time.
- If acquired, the effective validity is the original TTL minus the elapsed acquisition time.
- If not acquired (or validity expired), release the lock on every instance (even the ones that failed) to avoid leaving partial locks behind.
Redlock is genuinely controversial. Martin Kleppmann (author of Designing Data-Intensive Applications) published a well-known critique arguing Redlock doesn't provide the safety guarantees it claims under certain failure/GC-pause scenarios, because it relies on synchronized clocks and bounded process pauses — neither of which distributed systems can actually guarantee. Redis's creator (antirez) published a rebuttal. There's no fully settled resolution; both sides make valid points about different threat models.
When Redlock is (and isn't) the right call
| Scenario | Redlock needed? |
|---|---|
| Best-effort deduplication (avoid duplicate emails, idempotency-ish behavior) | No — single-instance lock is fine, occasional double-fire is tolerable |
| Preventing a scheduled job from running twice on two instances | No — single-instance lock + your job being naturally idempotent is enough |
| Correctness-critical mutual exclusion (e.g. preventing double-spend on a ledger) | Don't rely on Redlock alone — use a strongly consistent store (a database with real transactions, or a consensus system like ZooKeeper/etcd) as the actual source of truth for correctness, and treat any Redis-based lock as a performance optimization on top |
| Reducing contention / avoiding wasted work under normal operation | Yes, single-instance or Redlock both work well — it's an optimization, not a correctness mechanism |
The practical takeaway most teams land on: use Redis locks (single-instance or Redlock) for efficiency — avoiding duplicate/wasted work — but never as the sole guarantee for correctness where money, inventory counts, or irreversible side effects are at stake. Pair Redis locks with idempotency keys, database-level unique constraints, or optimistic concurrency checks (version columns) as the real safety net.
4. Fencing Tokens
Even a correctly-implemented lock has a subtle problem: a client can be paused (GC pause, network partition, OS scheduling delay) for longer than the lock's TTL, wake up believing it still holds the lock, and perform an operation after another client has already acquired the lock and started working. A fencing token — a monotonically increasing number issued with each lock grant — lets the protected resource itself reject stale writes.
public class FencedResource {
private volatile long lastToken = -1;
public synchronized void write(String data, long fencingToken) {
if (fencingToken < lastToken) {
throw new StaleTokenException(
"Rejected write with stale fencing token " + fencingToken +
" (last seen: " + lastToken + ")");
}
lastToken = fencingToken;
// ... apply write
}
}Redis doesn't give you a built-in monotonic fencing token generator out of the box, but INCR on a dedicated counter key does the job — issue the token from INCR lock:order:5001:fence at acquisition time and pass it through to whatever resource you're protecting.
5. Rate Limiting
Rate limiting protects a shared resource — an API, a downstream service, a per-user quota — from being overwhelmed by too many requests in too short a window. Redis is a natural fit because its atomic counters and sorted sets give you exactly the primitives rate limiting algorithms need.
Algorithm comparison
| Algorithm | How it works | Burst handling | Precision | Memory |
|---|---|---|---|---|
| Fixed window counter | INCR a counter per time bucket (e.g. per minute) | Allows 2x burst at window boundary | Low | Minimal (1 key) |
| Sliding window log | Store every request timestamp, count within rolling window | Smooth, accurate | High | High (1 entry per request) |
| Sliding window counter | Weighted average of current + previous fixed window | Smooth, approximate | Medium-high | Minimal (2 counters) |
| Token bucket | Tokens refill at a fixed rate, request consumes a token | Allows configurable bursts up to bucket size | High | Minimal (1 key: tokens + last-refill) |
| Leaky bucket | Requests processed at a fixed output rate, queued otherwise | Smooths bursts into a steady rate | High | Minimal + queue |
Fixed window counter (simplest)
@Component
public class FixedWindowRateLimiter {
private final StringRedisTemplate redisTemplate;
public boolean isAllowed(String userId, int limit, Duration window) {
long windowBucket = Instant.now().getEpochSecond() / window.getSeconds();
String key = "ratelimit:" + userId + ":" + windowBucket;
Long count = redisTemplate.opsForValue().increment(key);
if (count != null && count == 1L) {
redisTemplate.expire(key, window);
}
return count != null && count <= limit;
}
}Fixed window counters allow up to 2x the intended rate at window boundaries. A user can send limit requests in the last millisecond of one window and another limit requests in the first millisecond of the next — 2x the allowed rate within a very short span. Acceptable for coarse protection; not accurate enough for strict SLAs.
Sliding window using a sorted set
@Component
public class SlidingWindowRateLimiter {
private final StringRedisTemplate redisTemplate;
public boolean isAllowed(String userId, int limit, Duration window) {
String key = "ratelimit:sliding:" + userId;
long now = System.currentTimeMillis();
long windowStart = now - window.toMillis();
redisTemplate.execute((RedisCallback<Boolean>) connection -> {
byte[] keyBytes = key.getBytes(StandardCharsets.UTF_8);
// Evict entries outside the window
connection.zSetCommands().zRemRangeByScore(
keyBytes, Range.closed((double) 0, (double) windowStart));
// Count requests still in the window
Long count = connection.zSetCommands().zCard(keyBytes);
if (count != null && count < limit) {
connection.zSetCommands().zAdd(
keyBytes, now, String.valueOf(now).getBytes(StandardCharsets.UTF_8));
connection.keyCommands().expire(keyBytes, window.getSeconds());
return true;
}
return false;
});
return true; // simplified — see note below on atomicity
}
}The sliding-window-log example above has a race condition if not wrapped in a Lua script or MULTI/EXEC transaction — the read (zCard) and write (zAdd) aren't atomic across concurrent requests, which can let more than limit requests through under load. In production, implement this as a single Lua script executed via EVAL so the evict-count-add sequence is atomic. Libraries like Bucket4j (with its Redis backend) or Redisson's RRateLimiter already do this correctly — prefer them over hand-rolled Lua for anything beyond a learning exercise.
Token bucket (the industry-standard choice for APIs)
Token bucket allows configurable burst capacity while enforcing a steady average rate — this is what most public API rate limiters (GitHub, Stripe) effectively implement.
@Component
public class TokenBucketRateLimiter {
private final StringRedisTemplate redisTemplate;
private static final String TOKEN_BUCKET_SCRIPT = """
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refillRate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])
local bucket = redis.call('HMGET', key, 'tokens', 'lastRefill')
local tokens = tonumber(bucket[1])
local lastRefill = tonumber(bucket[2])
if tokens == nil then
tokens = capacity
lastRefill = now
end
local elapsed = math.max(0, now - lastRefill)
tokens = math.min(capacity, tokens + (elapsed * refillRate))
local allowed = 0
if tokens >= requested then
tokens = tokens - requested
allowed = 1
end
redis.call('HMSET', key, 'tokens', tokens, 'lastRefill', now)
redis.call('EXPIRE', key, 3600)
return allowed
""";
private final RedisScript<Long> script =
RedisScript.of(TOKEN_BUCKET_SCRIPT, Long.class);
public boolean tryConsume(String clientId, double capacity, double refillPerSecond) {
double now = System.currentTimeMillis() / 1000.0;
Long allowed = redisTemplate.execute(
script,
Collections.singletonList("ratelimit:bucket:" + clientId),
String.valueOf(capacity),
String.valueOf(refillPerSecond),
String.valueOf(now),
"1"
);
return allowed != null && allowed == 1L;
}
}@RestController
@RequestMapping("/api/v1")
public class SearchController {
private final TokenBucketRateLimiter rateLimiter;
@GetMapping("/search")
public ResponseEntity<SearchResults> search(
@RequestHeader("X-Client-Id") String clientId,
@RequestParam String query) {
// 20 requests/sec sustained, burst up to 50
if (!rateLimiter.tryConsume(clientId, 50, 20)) {
return ResponseEntity.status(HttpStatus.TOO_MANY_REQUESTS)
.header("Retry-After", "1")
.build();
}
return ResponseEntity.ok(performSearch(query));
}
}Why a Lua script for token bucket: the refill calculation and token consumption must happen atomically — otherwise two concurrent requests could both read "5 tokens available" and both deduct, over-allowing consumption. EVAL runs the whole script as a single atomic operation on the Redis server, sidestepping the race entirely without needing a separate lock.
Choosing an algorithm
| Requirement | Recommended algorithm |
|---|---|
| Simple, coarse per-minute limits, low implementation cost | Fixed window counter |
| Strict, accurate limits with no boundary bursts | Sliding window log or Redisson RRateLimiter |
| Public API with configurable burst allowance | Token bucket |
| Smoothing bursty traffic into a steady downstream rate | Leaky bucket (often implemented as a queue + worker) |
6. Putting It Together: Protecting a Downstream Call
A realistic pattern combines a lock (avoid duplicate concurrent work) with a rate limiter (respect a downstream API's quota):
@Service
public class ThirdPartySyncService {
private final RedissonClient redissonClient;
private final TokenBucketRateLimiter rateLimiter;
private final ExternalApiClient externalApiClient;
public void syncCustomer(String customerId) {
// 1. Avoid duplicate concurrent syncs for the same customer
RLock lock = redissonClient.getLock("lock:sync:" + customerId);
boolean locked = false;
try {
locked = lock.tryLock(0, 30, TimeUnit.SECONDS); // don't wait, fail fast
if (!locked) {
return; // another sync already in flight, safe to skip
}
// 2. Respect the third-party API's rate limit
if (!rateLimiter.tryConsume("third-party-api", 10, 5)) {
throw new RateLimitExceededException("Third-party API quota exhausted");
}
externalApiClient.pushCustomerData(customerId);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
if (locked && lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
}
}Key takeaways
- A basic Redis lock is
SET key token NX PX ttl— the token must be unique per holder, and release must be a token-checked atomic Lua script, never a bareDEL. - TTL on a lock is a safety net against crashed holders, not a performance tuning knob — set it generously, or use a watchdog-style library (Redisson) that auto-extends it while the holder is alive.
- Redlock adds resilience against a single Redis node failing, but it's controversial as a correctness guarantee under GC pauses and clock skew — treat it (and single-instance locks) as an efficiency optimization, not a substitute for real consistency guarantees.
- Fencing tokens protect the resource itself from stale writes by a client that paused past its lock's TTL and doesn't know it lost the lock.
- Never build correctness-critical mutual exclusion on Redis locks alone — pair with idempotency keys or database-level constraints for anything involving money or irreversible actions.
- Fixed window counters are simple but allow 2x burst at window boundaries — use sliding window or token bucket when precision matters.
- Token bucket is the standard choice for API rate limiting because it naturally supports configurable burst capacity on top of a steady sustained rate.
- Any rate limiter or lock implemented with multiple Redis round trips (read-then-write) needs to be wrapped in a Lua script or transaction to avoid TOCTOU (time-of-check-to-time-of-use) races under concurrent load.
Interview Questions
- How would you implement a basic distributed lock using Redis? What command and flags does it rely on?
- Why is it unsafe to release a Redis lock with a plain
DELcommand? - Walk through a scenario where a lock's TTL expires while the holder is still working — what goes wrong, and how do fencing tokens fix it?
- What is Redlock, and what problem does it solve that a single-instance Redis lock doesn't?
- What is the core criticism of Redlock, and do you agree with using it for correctness-critical operations?
- What's the difference between using a distributed lock for "efficiency" versus for "correctness"? Give an example of each.
- Compare fixed window, sliding window, and token bucket rate limiting algorithms — what are the trade-offs of each?
- Why does a fixed window counter allow up to 2x the configured rate at window boundaries?
- Why does the token bucket rate limiter implementation need to run inside a Lua script rather than as separate Redis commands from the application?
- How would you rate-limit a public API to allow bursts of 50 requests but sustain only 20 requests/second on average?
- What is a fencing token, and how does the protected resource use it to reject stale operations?
- How would you design a system that must guarantee no double-spend on a wallet balance, using Redis as part of the architecture?
- What happens to in-flight locks if the Redis instance holding them crashes and a replica is promoted without the lock key present?