Production controls

The first production deploy interrupts twenty workers at once. A two-minute partner call outlives its 90-second lease. One large customer fills a partition faster than it can drain. None of those failures appeared while a developer ran one worker against a small test backlog.

Production is where deploy waves, lease length, and hot keys reveal which assumptions in the basic claim loop no longer hold. A lease that was comfortably longer than a local test handler may expire during a real partner call. A rolling deploy may stop many workers before their claims finish. A partitioning scheme that looked even in test data may route a large customer’s work through one key. Each failure needs a specific control because they do not share one cause.

Add those controls in response to a named failure, not as a bundle of “enterprise-ready” features. This post connects each symptom to one mechanism: lease renewal for long jobs, fence tokens for stale workers, graceful drain for deploys, elected housekeeping for cleanup, key sharding for hot streams, and notifications for idle workers. The goal is an operating model the team can explain during an incident, not a queue with every possible knob.

Add controls when the failure appears

Add each production control in response to a failure you can name and test.

The claim loop remains the core. Do not add every control on day one. Add the one that matches the failure you have or the failure you are about to create with deploys, longer handlers, more workers, or higher traffic.

Lease and fencing columns on inbox

Open in Studio

Keep long handlers from losing ownership

The lease duration is a failure-detection window, not a maximum job runtime. Its deadline answers “how long will we wait before assuming this owner is gone?” If a healthy handler can outlive that window, it must renew the lease before expiry. Otherwise cleanup can return the row to pending and a second worker can begin the same side effect while the original worker is still running.

-- Call every 30s for long-running jobs (must be < lease duration)
UPDATE inbox
SET lease_expires_at = now() + interval '90 seconds'
WHERE id = $1
  AND claimed_by = $2
  AND status = 'processing'
  AND lease_expires_at > now();  -- only if we still hold valid lease

Make row ownership explicit

While a row is processing, only the current owner should renew or complete it. Postgres row locks and guarded UPDATE … WHERE clauses enforce that rule. The only risky period is the overlap after a lease expires and another worker claims the row.

Renew before the lease dies

If processing can outlast the lease, renew it before expiry. Choose the interval from measured handler time and the longest scheduler or network pause you expect to tolerate. The handler must also respond to cancellation. Detecting a lost lease without stopping the work is not enough.

async function processWithLease(row, workerId) {
  const controller = new AbortController();

  const renewUntilCancelled = async () => {
    while (!controller.signal.aborted) {
      await sleep(RENEW_INTERVAL_MS, controller.signal);
      if (controller.signal.aborted) return;

      const renewed = await db.renewLease(row.id, workerId, row.lease_generation);
      if (!renewed) throw new LeaseLostError("another worker owns this row");
    }
  };

  try {
    await Promise.race([
      processPayload(row.payload, row.lease_generation, controller.signal),
      renewUntilCancelled(),
    ]);
    const completed = await db.complete(row.id, workerId, row.lease_generation);
    if (!completed) throw new LeaseLostError("completion was fenced");
  } finally {
    controller.abort();
  }
}

processPayload must honor the abort signal where cancellation is possible. For an external call that may already have succeeded, cancellation cannot undo the effect. The provider idempotency key or downstream fence still protects the retry.

A fence token makes lost ownership enforceable, even when the stale worker is still running.

Stop stale workers from committing

ALTER TABLE inbox ADD COLUMN lease_generation BIGINT NOT NULL DEFAULT 0;

-- On claim: bump generation
WITH picked AS (
  SELECT id
  FROM inbox
  WHERE status = 'pending'
    AND available_at <= now()
  ORDER BY created_at, id
  LIMIT 25
  FOR UPDATE SKIP LOCKED
)
UPDATE inbox i
SET
  status = 'processing',
  claimed_by = $worker_id,
  lease_expires_at = now() + interval '90 seconds',
  lease_generation = i.lease_generation + 1,
  attempts = i.attempts + 1
FROM picked
WHERE i.id = picked.id
RETURNING i.*, i.lease_generation AS fence_token;

-- Downstream write must include fence_token; reject if stored token > incoming

Drain workers during deploys

A deploy should follow a different path than a crash. On SIGTERM, the worker should leave the live ring, stop claiming new rows, and finish or intentionally release the rows it already owns. That keeps a rolling deploy from creating a wave of expired leases.

UPDATE workers SET status = 'draining' WHERE id = $worker_id;

-- Worker loop: skip claiming when draining; exit when in-flight = 0
SELECT count(*) FROM inbox
WHERE claimed_by = $worker_id AND status = 'processing';

Run housekeeping without a separate scheduler

You do not need pg_cron, a sidecar, or a separate scheduler. Every worker already polls Postgres, so piggyback two cheap maintenance tasks on that loop and use an advisory lock so only one worker runs them per cycle.

Lease cleanup: Reset processing rows past lease_expires_at back to pending (or dead_letter if max attempts exceeded).

Stale worker sweep: Mark workers dead when last_seen_at exceeds TTL so the ring rebuilds without them.

const HOUSEKEEPING_LOCK = 847291;
let lastHousekeeping = 0;
const HOUSEKEEPING_MS = 30_000;

async function maybeRunHousekeeping(db) {
  if (Date.now() - lastHousekeeping < HOUSEKEEPING_MS) return;
  lastHousekeeping = Date.now();

  try {
    await db.query("BEGIN");
    const { rows } = await db.query("SELECT pg_try_advisory_xact_lock($1) AS acquired", [
      HOUSEKEEPING_LOCK,
    ]);
    if (!rows[0].acquired) {
      await db.query("COMMIT");
      return; // another worker is housekeeping
    }
    await db.query(`UPDATE workers SET status = 'dead'
      WHERE status IN ('alive','draining')
        AND last_seen_at < now() - interval '30 seconds'`);
    await db.query(`UPDATE inbox
      SET status = 'pending',
          claimed_by = NULL,
          claimed_at = NULL,
          lease_expires_at = NULL,
          available_at = now() + (LEAST(POWER(2, attempts)::int, 3600) * interval '1 second')
      WHERE status = 'processing'
        AND lease_expires_at < now()
        AND attempts < max_attempts`);
    await db.query(`UPDATE inbox
      SET status = 'dead_letter',
          claimed_by = NULL,
          claimed_at = NULL,
          lease_expires_at = NULL,
          last_error = COALESCE(last_error, 'max attempts during lease cleanup')
      WHERE status = 'processing'
        AND lease_expires_at < now()
        AND attempts >= max_attempts`);
    await db.query("COMMIT");
  } catch (e) {
    await db.query("ROLLBACK");
    throw e;
  }
}

Wake idle workers without trusting notifications for durability

Polling is the reliable baseline. LISTEN/NOTIFY is a latency optimization: it wakes idle workers quickly after enqueue, but workers should still poll as a fallback because notifications are not durable storage.

CREATE OR REPLACE FUNCTION notify_inbox_insert() RETURNS TRIGGER AS $$
BEGIN
  PERFORM pg_notify('inbox_new', NEW.partition_key);
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER inbox_notify
  AFTER INSERT ON inbox
  FOR EACH ROW EXECUTE FUNCTION notify_inbox_insert();

Shard hot keys when one stream dominates

When one partition_key dominates volume, for example a single large tenant, I shard that key explicitly: tenant:123#shard-0 … tenant:123#shard-7. Producers choose the shard. Ordering still holds within each shard, but not across all shards for that tenant.

Sharding a hot key buys throughput by giving up global ordering for that key.

Source

These companion files collect the SQL and TypeScript used in this article. The link labels distinguish full implementations from focused examples and worker skeletons.

• SQL for this article
• Housekeeping implementation
• Shared TypeScript helper