Durable Work in PostgresPart 3
Many workers
Multi-worker contention, delivery guarantees, and per-key ordering before consistent hash.
Traffic doubled. You added a second worker to the pool. Within an hour, two different processes tried to send the receipt for order:9182. Support got a duplicate email.
The bug is claiming without rules. You need clear delivery promises, such as at-least-once delivery and per-key ordering, plus guards that enforce them.
Competing consumers, same inbox
The producer inserts into inbox. Every worker runs the same loop: clean up expired leases, claim with SKIP LOCKED, handle, complete. No central dispatcher. Whoever wins the row lock processes the row.
That pattern is Competing Consumers: many workers, one queue table, and row locks instead of a broker assigning jobs. It scales throughput, but it does not automatically pin order:9182 to one process.
| Concern | Mechanism here |
|---|---|
| Throughput across machines | Many workers + SKIP LOCKED |
| Crash recovery | Leases + lease cleanup |
| Duplicate inbox rows | idempotency_key UNIQUE |
| Duplicate side effects | Idempotent handlers + fencing |
| Related work stays together | Not automatic. Needs consistent hash + heartbeats |
How to scale past one worker
There are three practical ways to coordinate multiple workers on the same inbox, from simplest to most controlled:
- Any worker claims any row: simplest multi-process setup. Throughput increases, but there is no per-key affinity or ordering. Use when jobs are independent.
- Per-key ordering guard: sibling check on claim gives FIFO within one
partition_keywithout a coordinator. Hot keys bottleneck one worker. - Consistent hash + heartbeats: stable affinity so a new worker moves ~1/N keys instead of the whole backlog. Rebalance windows and briefly stale ring caches are the tradeoff. Use when autoscale or deploy churn makes
hash % Ntoo disruptive.
Delivery guarantees to document
You have two workers. Write down what you promise before you ship.
| Promise | Status | Notes |
|---|---|---|
| Row delivery | At-least-once | Leases + lease cleanup. May retry unless dead_letter |
Per-partition_key order | Conditional | Only with ordering guard + idempotent handler |
| Duplicate inbox rows | Prevented | idempotency_key UNIQUE on enqueue |
| Duplicate side-effects | Your responsibility | Idempotent handlers. Fencing at downstream stores |
| Global FIFO | Not provided | By design. Single partition_key or ordered log |
Idempotency at enqueue and in the handler
At enqueue: idempotency_key UNIQUE on INSERT. Protects producer retries.
In the handler: Worker may process the same row twice after reclaim. Check “already done” before side effects.
Per-key ordering
Same partition_key does not imply FIFO unless you opt in. Add a sibling guard on the claim query: refuse to claim if another row for that key is processing. Order by created_at among pending rows.
| Guarantee | How | Breaks when |
|---|---|---|
FIFO per partition_key | NOT EXISTS sibling processing; ORDER BY created_at | Two workers claim different keys concurrently (fine). Same key without guard (bad) |
| No duplicate inbox rows | ON CONFLICT against the idempotency index | Producer omits key |
| No duplicate side-effects | Handler checks processed-keys / fence token | Handler is not idempotent |
| Stable affinity per key | Not in this chapter. Needs consistent hash. |
Ordering is opt-in. The base competing-consumer pattern gives parallelism, not automatic serialization. Add the sibling guard when per-key FIFO matters.
Where this breaks down
- No affinity. Two workers can still race on related keys unless you add ring ownership.
- Hot
partition_key. One tenant stream bottlenecks whichever worker wins the lock. Shard hot keys explicitly. - Rebalance window. When you add consistent hash, keys move while old leases may still be in flight. Plan for fencing during rebalance.
- Split-brain after long pauses. A stale worker must fail
completeand lose fence tokens downstream.
Source
Use the article for explanation, then use these files when you want the complete SQL and TypeScript in one place.