Laravel and Kafka Without Lost Events: The Transactional Outbox, Idempotent Consumers, and PostgreSQL

A common first attempt at event-driven Laravel code looks innocent:

$order = Order::create($data);
$kafka->publish('orders.placed.v1', $order->toArray());

What if the database insert succeeds and the publish fails? The order exists, but downstream inventory never hears about it. Reversing the calls merely reverses the failure: the event may be visible even though the order transaction rolls back. Publishing inside DB::transaction() does not make Kafka part of PostgreSQL's transaction.

The transactional outbox solves the handoff between the database and a publisher. It does not magically deliver an event exactly once to every external system. It gives you a durable record to publish and a clean place to retry. Consumers still need duplicate protection. Let's build the pattern for a Laravel order service and PostgreSQL.

Define the contract before the integration

Our event means the order was accepted and committed, not merely that an HTTP request arrived:

{
  "eventId": "evt_01JQ8X8F",
  "eventType": "OrderPlaced",
  "schemaVersion": 1,
  "occurredAt": "2026-09-29T10:30:00Z",
  "orderId": "ord_782",
  "customerId": "cus_17",
  "totalMinor": 12900,
  "currency": "USD"
}

eventId is the deduplication identity. The Kafka record key is orderId, which keeps events for the same order together under a stable partitioning policy. The payload is a public contract. Do not serialize an Eloquent model wholesale: hidden fields, relations, and column names change for reasons unrelated to the event. Decide which data consumers need and how you will add fields or a new schema version.

Topic names such as orders.placed.v1 can encode a major contract generation, but the topic name alone is not a schema registry. Document ownership, retention, key, fields, compatibility, and intended consumers. If personal data appears in an event, think about retention and access before publishing it.

The outbox table

A minimal PostgreSQL outbox table can hold:

CREATE TABLE outbox_events (
    id uuid PRIMARY KEY,
    aggregate_type text NOT NULL,
    aggregate_id text NOT NULL,
    topic text NOT NULL,
    event_key text NOT NULL,
    payload jsonb NOT NULL,
    occurred_at timestamptz NOT NULL,
    published_at timestamptz NULL,
    attempts integer NOT NULL DEFAULT 0,
    last_error text NULL
);

CREATE INDEX outbox_unpublished_idx
    ON outbox_events (occurred_at, id)
    WHERE published_at IS NULL;

This is illustrative DDL. In a real service, add operational fields for worker ownership or retry scheduling if needed, control permissions, and decide a retention policy for published rows. A partial index keeps polling efficient as history grows. Avoid storing credentials or verbose sensitive responses in last_error.

Inside one PostgreSQL transaction, create the order and its outbox row. This PHP is a design sketch using ordinary Laravel database operations; adapt types and validation to your application:

DB::transaction(function () use ($input) {
    $order = Order::create([
        'customer_id' => $input->customerId,
        'total_minor' => $input->totalMinor,
        'currency' => $input->currency,
    ]);

    $eventId = (string) Str::uuid();
    DB::table('outbox_events')->insert([
        'id' => $eventId,
        'aggregate_type' => 'order',
        'aggregate_id' => (string) $order->id,
        'topic' => 'orders.placed.v1',
        'event_key' => (string) $order->id,
        'payload' => json_encode([
            'eventId' => $eventId,
            'eventType' => 'OrderPlaced',
            'schemaVersion' => 1,
            'occurredAt' => now()->toIso8601String(),
            'orderId' => (string) $order->id,
            'customerId' => (string) $order->customer_id,
            'totalMinor' => $order->total_minor,
            'currency' => $order->currency,
        ], JSON_THROW_ON_ERROR),
        'occurred_at' => now(),
    ]);
});

If the transaction rolls back, neither row survives. If it commits, the outbox event is available for a separate publisher even if the web process crashes immediately afterward. For a high-traffic API, keep this transaction short; do not perform a network publish or slow downstream call while holding database locks.

Publish from the outbox

A worker selects a small batch of unpublished rows, claims them safely, sends each record to Kafka with the stored key, waits for producer acknowledgement according to its reliability settings, and marks it published. Multiple workers need a claim strategy. PostgreSQL FOR UPDATE SKIP LOCKED is one option when used in short transactions, but do not hold a database row lock while waiting on a slow broker. A lease column or short claim transaction lets the network operation happen outside the lock. Recover expired leases after a crashed worker.

There is an unavoidable crash window: Kafka accepts a record, then the worker dies before setting published_at. The next worker sends it again. Therefore the outbox provides at-least-once publication, and the event ID must remain the same across retries. Creating a new event ID on each retry would defeat consumer deduplication.

Producer idempotence helps avoid duplicates caused by producer retries to Kafka itself. It does not make your PostgreSQL update and Kafka acknowledgement one atomic transaction. Kafka transactions are powerful when consuming from Kafka and producing back to Kafka, but they do not automatically wrap arbitrary PostgreSQL writes. Keep the boundaries explicit.

Whether the publisher is a Laravel command, a queue worker, or a CDC pipeline is an implementation choice. Laravel's built-in queue drivers cover backends such as Redis, SQS, and databases; Kafka is not the built-in queue driver you get simply by changing QUEUE_CONNECTION. A community Kafka client or managed connector needs its own compatibility, security, and operations review. This article deliberately keeps the broker client behind an adapter rather than pretending a package-specific method is universal.

Make the consumer idempotent

Suppose inventory consumes OrderPlaced and reserves stock. It can process the event and crash before committing its Kafka offset. On restart it reads the same record again. The business side effect must not reserve stock twice.

Use a processed_events table with a uniqueness constraint scoped to the consumer's logical operation:

CREATE TABLE processed_events (
    consumer_name text NOT NULL,
    event_id text NOT NULL,
    processed_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, event_id)
);

Within one inventory database transaction, try inserting (inventory-reservation, eventId) and apply the reservation. A duplicate insert means this consumer already completed that event, so it can skip the side effect. After the transaction commits, advance the Kafka offset. The exact PHP depends on the selected client and whether it commits per record or batch; the ordering principle matters more than a fictional one-line API.

If the inventory transaction commits and the process crashes before offset commit, the event is read again and the unique constraint blocks the second reservation. If the transaction rolls back, the offset should not advance as if processing succeeded. Never commit an offset before a non-idempotent external effect merely to avoid duplicates; that can lose work.

For side effects outside the same database, such as email or payment, a local dedupe row alone is not necessarily atomic with the remote service. Use the remote provider's idempotency key if available, an outgoing outbox, or a compensating design. State the guarantee for each side effect, not for the pipeline in general.

Handle poison events and retries deliberately

A malformed event that always fails can block a partition indefinitely. A transient database outage should retry with backoff; a schema violation needs inspection and a decision about skip, repair, or quarantine. A dead-letter topic is one possible quarantine mechanism, but publishing to it and committing the original offset creates its own failure window. Define the sequence, alerting, retention, access, and replay procedure.

Do not swallow exceptions and commit the offset because a log line was written. Include eventId, topic, partition, offset, consumer group, attempt count, and a safe error category in diagnostics. Avoid logging full sensitive payloads. Use a bounded retry policy so one record does not silently stall an entire partition forever.

A replay must be an intentional operation. If inventory has already reserved stock, processing from the beginning should reconstruct a projection or skip known event IDs rather than repeat payments and emails. Keep enough dedupe history for the event retention and replay horizon you promise.

Schema evolution and cross-service ownership

Producers own the event meaning, but consumers need advance notice of incompatible changes. Prefer additive, optional fields with defined defaults for routine evolution. If a field's meaning changes, create a new version or migration path. Validate the contract in CI with representative events, including older versions consumers may still see during replay.

Avoid the “shared database via Kafka” anti-pattern where one service emits every row change and consumers reconstruct all internal tables. Publish domain facts that downstream teams can understand. If the integration genuinely needs change data capture, distinguish raw CDC from a stable business event contract.

Production test plan

Test the failure windows, not just a mocked publish() call:

1. Roll back the order transaction: neither order nor outbox row remains.
2. Commit the order and stop the publisher: the event remains pending.
3. Publish successfully, then simulate a crash before published_at: a retry uses the same event ID.
4. Deliver the same event twice to inventory: exactly one reservation is applied.
5. Crash after the inventory transaction but before offset commit: replay remains safe.
6. Send invalid schema and transient database errors: the retry and quarantine policies behave as documented.
7. Run two publisher workers and concurrent consumers: claims and unique constraints hold.

Observe outbox age, pending count, publish error rate, consumer lag, duplicate counts, and quarantine volume. A “healthy” HTTP endpoint can coexist with an outbox that has stopped draining, so put these signals on a dashboard and alert on sustained age, not just one momentary spike.

When this design is worth it

For one background email, Laravel's ordinary queue may be enough. The outbox and Kafka become valuable when the order fact must reach multiple independent systems, replay is useful, and loss or duplication has meaningful business cost. The reliable design is not a single library installation; it is the combination of a database transaction, durable publication, stable event identity, idempotent consumers, and observable failure handling.

Further reading

• Apache Kafka delivery semantics and transactions
• Kafka producer configuration
• Kafka consumer configuration
• Laravel queue documentation