Many workers

Traffic doubles, the queue falls behind, and the obvious fix is a second worker. Soon one worker sends the receipt for order:9182 while another processes a refund for the same order. Both rows were claimed correctly, but the customer receives the messages in the wrong order because nothing kept that order’s stream together.

Adding processes increased capacity, but it also exposed an unanswered question: which work may run concurrently, and which work must stay in order?

Multiple workers can safely claim different rows from one queue. That pattern is called competing consumers. It increases throughput, but it does not provide ordering. This article separates three promises that are easy to blur together: at-least-once delivery, deduplication, and ordering for one business key such as an order.

Competing consumers, same inbox

The producer inserts into inbox, and every worker runs the same loop: clean up expired leases, claim with SKIP LOCKED, handle the payload, and record completion. There is no central dispatcher choosing a worker. Postgres serializes the claim on each row, while SKIP LOCKED lets losing workers continue to other available rows instead of waiting for that row’s lock.

That pattern is Competing Consumers: many workers, one queue table, and row locks instead of a broker assigning jobs. It scales throughput, but it does not automatically pin order:9182 to one process.

Workers registry and shared inbox

Open in Studio

How to scale past one worker

There are three practical ways to coordinate multiple workers on the same inbox, from simplest to most controlled:

1. Any worker claims any row: simplest multi-process setup. Throughput increases, but there is no per-key affinity or ordering. Use when jobs are independent.
2. Per-key ordering guard: sibling check on claim gives FIFO within one partition_key without a coordinator. Hot keys bottleneck one worker.
3. Consistent hash + heartbeats: stable process affinity for cache locality or expensive per-key state. A new worker receives about 1/(N+1) of the keys from the existing pool. This is an optimization, not a replacement for database ordering or idempotency.

Delivery guarantees to document

More workers increase throughput. They do not create ordering.

Once you have two workers, write down the delivery promise before you ship. “The queue is distributed” is not a useful guarantee. The table below separates whether a row will be attempted, whether two rows for one order may overlap, and whether repeating an attempt can duplicate an external effect. Each promise needs a different mechanism.

Idempotency at enqueue and in the handler

At enqueue: a unique constraint on idempotency_key prevents a producer retry from creating two queue rows for the same logical job. This deduplicates admission to the queue; it does not prove the handler ran only once.

In the handler: Worker may process the same row twice after reclaim. Check “already done” before side effects.

Per-key ordering

Same partition_key does not imply FIFO unless you opt in. The claim query must select only the oldest unfinished row for each key. Checking only for a processing sibling is not enough: one batch could otherwise claim several pending rows for the same order at once.

WITH candidates AS (
  SELECT i.id
  FROM inbox i
  WHERE i.status = 'pending'
    AND i.available_at <= now()
    AND NOT EXISTS (
      SELECT 1
      FROM inbox busy
      WHERE busy.partition_key = i.partition_key
        AND busy.status = 'processing'
    )
    AND NOT EXISTS (
      SELECT 1
      FROM inbox earlier
      WHERE earlier.partition_key = i.partition_key
        AND earlier.status = 'pending'
        AND (earlier.created_at, earlier.id) < (i.created_at, i.id)
    )
  ORDER BY i.created_at, i.id
  LIMIT 25
  FOR UPDATE SKIP LOCKED
)
UPDATE inbox i
SET status = 'processing',
    claimed_by = $1,
    lease_expires_at = now() + interval '90 seconds',
    lease_generation = i.lease_generation + 1,
    attempts = i.attempts + 1
FROM candidates c
WHERE i.id = c.id
RETURNING i.*;

The (created_at, id) comparison gives ties a stable order. Including both pending and processing siblings means a retrying head item blocks later work for the same key. That is strict FIFO. Do not pay for it when later work is allowed to pass a delayed retry.

Ordering is opt-in. Competing consumers provide parallelism, not automatic serialization.

Add the sibling guard when per-key FIFO matters.

Simulation: two workers, no ring yet

Where this breaks down

I do not add affinity until the workload needs it, but I make these limits explicit:

• No process affinity. Correctly ordered work may move between workers. That only matters when the handler benefits from local cache or other process-local state.
• Hot partition_key. One tenant stream bottlenecks whichever worker wins the lock. Shard hot keys explicitly.
• Rebalance window. When you add consistent hash, keys move while old leases may still be in flight. Plan for fencing during rebalance.
• Split-brain after long pauses. A stale worker must fail complete and lose fence tokens downstream.

Source

These companion files collect the SQL and TypeScript used in this article. The link labels distinguish full implementations from focused examples and worker skeletons.

• SQL for this article
• TypeScript worker-pool skeleton
• Shared TypeScript helper