Stable worker affinity with consistent hashing

The queue is busiest when autoscaling adds a fourth worker. With hash(key) % N, that harmless-looking scale event changes the calculated owner for most open keys. Work moves while old leases are still active, caches disagree about membership, and the system creates contention at the moment it was supposed to relieve pressure.

The people watching the backlog do not need a mathematically perfect distribution. They need scaling to move a predictable slice of work without scrambling every customer’s in-flight stream.

Consistent hashing reduces that movement by placing workers and fixed database buckets on a logical ring. Each bucket belongs to the next worker around the ring, so adding one worker transfers only the nearby ranges instead of recalculating every key against a new worker count. In a balanced pool, the new worker receives roughly 1/(N+1) of the buckets. Heartbeats supply the current membership list by periodically recording which worker processes are still alive.

Do not start here just because the queue has more than one worker:

Workload	Smallest useful coordination model
Jobs are independent	Competing consumers with `SKIP LOCKED`
One key needs strict ordering	Oldest-unfinished sibling guard in the claim query
Keys need stable process affinity during scaling	Consistent hash plus live membership

Pay for a hash ring only when stable affinity solves a measured problem.

A hash ring creates membership, cache, rebalance, and fencing work. Those costs are part of the design, not implementation trivia.

Who owns `order:9182`?

Every row carries a partition_key, such as order:9182, chosen by the producer to group related work. A stable hash maps that key to one of 1,024 database buckets, and the ring assigns each bucket to a live worker. Receipt and refund rows for the same order therefore tend to reach the same process, which can improve cache locality. They are not automatically processed in order, because one worker can still run several handlers concurrently; strict ordering still requires the oldest-unfinished database guard from the previous article.

Affinity keeps work on one process. It does not provide FIFO ordering.

`partition_bucket` vs ring owner

partition_bucket is a stable hash of partition_key, modulo 1,024 in the downloadable example, stored at enqueue time. It is an index-friendly filter for claim queries. Ring ownership decides which live worker may claim those buckets right now. The bucket does not encode the worker that owned it when the row was inserted.

Partition keys, buckets, and worker ownership

Open in Studio

Given a fixed bucket, which worker owns it today and after a scale event? Consistent hashing maps workers and buckets onto a ring. When one worker joins, only buckets transferred to that worker should move. The included test checks that property.

Imagine a circle numbered 0 to 2³²−1. Every worker and every work item gets hashed to a position on that circle. To find the owner of a key, walk clockwise from the key’s position until you hit the first worker.

Virtual nodes (vnodes)

The downloadable example places each worker at 128 points on the ring. These virtual nodes reduce the uneven ownership produced by a single random position. The number is a starting point for this example, not a general tuning rule. Measure the distribution with your bucket and worker counts.

function hash(str) {
  let h = 2166136261;
  for (let i = 0; i < str.length; i++) {
    h ^= str.charCodeAt(i);
    h = Math.imul(h, 16777619);
  }
  return h >>> 0; // unsigned 32-bit
}

function lookupOwner(key, ring) {
  // ring: sorted array of { hash, workerId }
  const keyHash = hash(key);
  for (const node of ring) {
    if (node.hash >= keyHash) return node.workerId;
  }
  return ring[0].workerId; // wrap around
}

Ownership follows live ring membership, not a static config file. At higher vnodes per worker, each worker owns many small ranges interleaved around the ring.

Hash ring demo

Interactive Hash Ring

Vnodes:16

Click a sample key or enter your own, then Lookup Owner. The bright arc on the ring is the hash range that owns that key. Drag vnodes to see how arc size changes.

Ownership arcs Partition keysSelected key's arc

Heartbeat TTL and claim lease measure different things: process liveness vs in-flight work duration.

Heartbeat Timeline: Worker Failure & Recovery

Timing Relationships: Heartbeat, TTL, and Lease

Heartbeat interval (10s) TTL window (30s): mark dead if no heartbeat Claim lease (90s): row locked during processing

Which workers are actually running?

A hash ring is only as accurate as its list of live workers.

A config file that says “three workers” is wrong the moment one process dies or autoscaling adds a fourth. Heartbeats turn the workers table into a live registry.

For one low-latency workload, I would start testing with a 10-second heartbeat, a 30-second TTL, and a 90-second claim lease for handlers that finish in a few hundred milliseconds at p99. Those are example values, not safe defaults. Derive the lease from measured handler time plus network and scheduler margin. Derive heartbeat and TTL from the failure-detection time the service can tolerate, then test pauses and jitter around that boundary.

What gets recorded: Each worker periodically writes last_seen_at, its identity, and optionally load metrics to a shared registry (Postgres table).

Failure detection: If now() - last_seen_at > TTL, the worker is considered dead. Its partitions are reassigned to the next live node on the ring.

Example heartbeat values

Parameter	Example starting value	Why
`heartbeat_interval`	5–15 seconds	Balance DB write load vs detection speed
`ttl (grace period)`	3× interval (e.g. 30s)	Absorb network jitter and avoid false positives
`claim_lease`	60–120 seconds	How long a claimed inbox row stays locked if worker dies mid-process
`housekeeping_interval`	Every worker loop (~30s)	One worker wins advisory lock and runs lease cleanup + stale-worker detection

T+0s: Worker-2 sends a heartbeat: UPDATE workers SET last_seen_at = now() WHERE id = 'worker-2'

T+5s: During normal operation, Worker-2 claims and processes inbox rows for its partition only.

T+12s: Worker-2 crashes and stops sending heartbeats. Other workers continue. Worker-2’s claimed rows sit until their leases expire.

T+42s: The TTL expires, so a worker marks Worker-2 dead: UPDATE workers SET status = 'dead' WHERE last_seen_at < now() - interval '30 seconds'

T+43s: Lease cleanup releases stale claims. Worker-3, the next owner on the ring, picks up the orphaned rows on its next poll.

Simulation: three workers on the ring

Verify the ownership behavior

Before relying on affinity, test these properties with your worker and bucket counts:

Idempotency key has a unique constraint
Claims use row locks or atomic state transitions
Leases expire and are reaped
adding a worker only moves buckets to that worker
removing a worker only moves that worker’s buckets
stale workers cannot complete a newer claim because lease_generation fences them
per-key ordering still comes from the database guard when the business requires it

Source

These companion files collect the SQL and TypeScript used in this article. The link labels distinguish full implementations from focused examples and worker skeletons.