Durable Work in PostgresPart 7

Cross-service handoffs

Transactional outbox, transport choices, webhooks, contracts, and when brokers fit. Work that leaves your database commit.

Payment for order:9182 commits while the search service is unavailable. The order service can truthfully tell the customer that payment succeeded, but search still shows the order as pending. Once those records live in databases owned by different services, no local transaction can make both changes happen at the same instant or roll one back when the other is temporarily unreachable.

This is where a local queue becomes a cross-service handoff. Inside one database, Postgres can commit the business change and follow-up row atomically. Across databases and deployable services, that guarantee ends. What you can preserve is responsibility: the producer records durable intent, the transport keeps attempting delivery, and the receiver applies the event idempotently.

I draw that boundary explicitly because phrases like “eventually consistent” are too vague for the people affected. Product needs a lag promise, support needs a status to inspect, and engineers need to know where retries and deduplication live. This post builds that contract with a transactional outbox, a transport, and an inbox on the receiving side.

When the commit boundary ends

Inside one database, the story is clean: one commit can update an order and enqueue the follow-up work. The moment work leaves that database or deployable unit, you no longer have one transaction. You have a handoff.

From that point on, you need transport: a way for intent to cross from your Postgres into someone else’s process. The important boundary is not necessarily a network hop. It is the point where one transaction and one team can no longer control both sides, such as a different database, deployable service, broker, or external API. On each side of that boundary, a durable row records what the local system still owes.

The transport delivers bytes. Each service still owns its durable state and recovery.

Use an outbox relay on the producer and an inbox on the receiver. Transport does not replace leases or claim loops on either side.

What each part can guarantee

Your database can make the order update and outbox insert succeed or fail together. It cannot make a broker and search index join that same local transaction, so “paid” and “visible in search” become different guarantees. The first is true when the producer commits. The second becomes true later, after transport and the consumer succeed, and should have its own measurable lag objective.

Transport and downstream consumers are always eventual. Only the producer database gets ACID on write.

LayerWhat holdsConsistencyPostgres tools
Inside your DBBusiness row + outbox/inbox row commit together. Claims are exclusiveStrong (ACID, row locks)Transactional outbox, SKIP LOCKED, leases
Your worker layerEach row processed at least once. Optional per-key FIFOAt-least-once + idempotent effectsHash ring, heartbeats, ordering guard, fencing
TransportMessage or HTTP delivery may retry, reorder (across keys), or lagEventual, at-least-once typicalBroker, webhook, outbox relay
Downstream consumerTheir read model, inbox, or side effect catches up laterEventual: bounded lag is the service-level objective (SLO)Their idempotency + your contract

Cross-boundary architecture

Guarantees stop at each service boundary. Make the handoff explicit: durable intent on your side, idempotent consumption on theirs, and transport in between that may retry.

Three zones: producer DB (ACID), transport (at-least-once), consumer (catches up later).

Producer outbox and consumer inbox

Open in Studio
Each database owns a durable boundary. The transport connects them but does not make the transaction distributed.

Transactional outbox for order:9182

Make the intent durable. When payment succeeds, you need search to know eventually. You cannot call search inside the transaction. You can insert an outbox row that says “publish OrderPaid” in the same commit as UPDATE orders.

The Transactional Outbox pattern is the bridge: write intent in the same transaction as the domain mutation, then let a relay worker publish to your transport. In this implementation, the relay is the same competing-consumers loop from earlier chapters, applied to outbound events. You get:

  1. Atomic intent: the paid order and OrderPaid outbox row share one commit
  2. Relay: a worker claims the outbox row and publishes it to the transport
  3. At-least-once publish: a crash after publish but before completion causes a retry
  4. Consumer deduplication: the subscriber uses event_id as its idempotency key
BEGIN;
UPDATE orders SET status = 'paid' WHERE id = 9182;  -- business fact the API must not lie about
INSERT INTO outbox (partition_key, event_type, payload, idempotency_key)
VALUES (
  'order:9182',
  'OrderPaid',
  '{"order_id":9182,"paid_at":"2026-07-01T12:00:00Z"}'::jsonb,
  'evt-order-9182-paid-v1'
);  -- durable publish intent, not the wire publish
COMMIT;  -- both rows exist or neither. No ghost OrderPaid events

Follow the same event across the boundary:

TimeStateWhat support can say
12:00:00.000Order and outbox row commitPayment is recorded and search update is owed
12:00:00.120Relay claims evt-order-9182-paid-v1Delivery is in progress
12:00:03.120Publish times outOutcome is ambiguous, so the same event id will retry
12:00:05.400Relay publishes the retryTransport accepted the event
12:00:05.470Search inbox deduplicates and updatesSearch has caught up in 5.47 seconds

That timeline gives product a lag measure and gives support one id to follow. It also shows why a timeout cannot create a new event id.

Choosing a transport

After the outbox row is durable, choose transport based on the consumer, the SLO, and who operates the middleware.

KindGood whenOptimizes forBreaks when
Message brokerService-to-service delivery, routing keys, ops-owned middlewareFast handoff and per-queue routingYou need long retention replay for many independent consumers
Event log / streamMany subscribers, replay, high-volume shared logFan-out and retentionPoint-to-point only with one consumer team
Managed queueCloud-native, minimal ops, push to one consumer groupManaged durability and scalingStrict per-key ordering without paying for FIFO SKUs
HTTP / webhooksOne partner or SaaS exposes a URLSimple integration surfacePartner outage blocks your relay unless you queue and retry
Poll / RPC pullNo push. Batch export or client-pull APIsReceiver controls cadenceLow latency SLOs without aggressive polling
CDCDownstream needs row changes, not domain eventsDecoupling from app publish codeSchema refactors and domain event contracts

Inbound: webhooks and broker consumers become inbox rows

The receiver applies the mirror image. It verifies the transport, inserts an inbox row using event_id as a unique key, and returns 2xx only after the row commits. The claim loop handles the business effect later. For the full HTTP version, including signatures, timeouts, and retry policy, see Build your own webhook transport.

Contracts across boundaries

Shared transactions are gone. Shared vocabulary is not.

When two teams own two databases, coordination becomes documentation plus stable ids. The table below is the minimum contract for OrderPaid on order:9182 to land safely in search.

Contract pieceProducer owesConsumer owes
event_id / idempotency keyStable, unique per logical eventDedupe store or unique constraint
partition_key / message keySame key for all ordered events in a streamSingle consumer per key (or ordering guard)
Schema versionBackward-compatible changes. Bump version on breaksReject or dead-letter unknown versions
Delivery semanticsDocument at-least-once. No silent dropsIdempotent handlers. Expose lag metrics
ReplayRetention policy or archive for re-publishSafe to re-process historical events

Sagas and long flows across services

Some workflows outlive one request. Charging a card, reserving inventory, shipping, and updating search are several local commits linked by durable facts, not one distributed lock. Keep each step idempotent, give it a timeout, and model compensation when a committed effect must be countered. Use a saga or workflow record to track business progress instead of stretching the transport into a business state machine.

Explaining lag to product

Payment in your DB can be immediate while search and analytics catch up on a separate timeline. Use this distinction when someone asks why search lags payment by thirty seconds.

  • User-facing write path. Strong in your DB: “Payment recorded” is true when API returns success.
  • Derived views. Eventual. Search, analytics, recommendations catch up in seconds (or minutes: define SLO).
  • External systems. Eventual with retries. A webhook may arrive twice. Partner ledger may lag. Contract defines max delay. Same mechanics whether the other party is another team or Stripe.
  • Failure mode: At-least-once everywhere outside ACID. Duplicates are normal. Idempotency is not optional.

Where this implementation fits

Match the tool to the constraint. Postgres coordinates work inside a service. Brokers and queues help at the boundary when fan-out, retention, or cross-team delivery require them.

Strong fit

  • Durable work with claims, leases, and retries
  • Postgres already in the stack (or inbox/outbox tables are acceptable)
  • Per-entity ordering or partition affinity matters
  • Producers and workers share a database. Or outbox/inbox at each service boundary
  • Backlog depth and stuck rows should be visible in SQL

Weaker fit: consider other tools

  • Shared event log with long retention and many independent replay consumers → Kafka / Pulsar as platform infra
  • Fan-out streaming is the product, not per-service claim coordination
  • Jobs-only background work, no custom ordering → pg-boss, Graphile Worker, or River
  • No database on either end can participate in inbox/outbox
  • Postgres is saturated after tuning and coordination itself must move off-DB