Transactional Outbox Pattern with Spring Boot and Kafka

Your order saved. Your Kafka event didn't.

@Transactional can't fix that. It opens a JDBC connection and wraps your database writes in a single atomic unit — but it knows nothing about Kafka. The moment you call kafkaTemplate.send() inside a @Transactional method, you have left ACID territory. Kafka is a separate system. It has no concept of your transaction, and no way to participate in it.

The failure mode is quiet. The database commit succeeds. Then Kafka times out, or the broker is briefly unavailable, or the JVM crashes at exactly the wrong millisecond. The event is gone. No exception propagates back to the caller. No retry happens. Downstream services never learn that the order was created, and your data silently drifts out of sync. This is the dual-write problem.

If you haven't seen it break in a running demo yet, start with the Dual Write lab first. It shows exactly where the gap opens and ends with: "The Transactional Outbox Pattern solves this." This lab is that solution — a complete Spring Boot 3.4 implementation with PostgreSQL 16, Redpanda (Kafka-compatible), exponential backoff retry, and three Testcontainers integration tests that prove the guarantees hold against real infrastructure.

What this lab covers:

• The outbox_events schema and why each column exists
• Five annotated classes: OrderService, OutboxWriter, OutboxProcessor, KafkaOutboxEventHandler, and RetryPolicy
• FOR UPDATE SKIP LOCKED and why it prevents duplicate Kafka publishes when multiple scheduler replicas run in Kubernetes
• Three Testcontainers integration tests proving atomicity, full relay, and broker-failure resilience against real PostgreSQL 16 and Kafka

How the Transactional Outbox Pattern Works

The fix is simpler than it sounds. Instead of writing to Kafka inside the business transaction, write an event row to the database inside the business transaction. Then a background scheduler reads those rows and forwards them to Kafka independently.

That's it. The insight is that the database — not Kafka — is your source of truth for what needs to be published. The outbox_events table is the durable buffer between your domain logic and the message broker.

sequenceDiagram
    participant C as Client
    participant S as OrderService
    participant DB as PostgreSQL
    participant P as OutboxProcessor
    participant K as Kafka / Redpanda

    C->>S: POST /orders
    S->>DB: BEGIN TRANSACTION
    S->>DB: INSERT INTO orders
    S->>DB: INSERT INTO outbox_events (PENDING)
    S->>DB: COMMIT
    S-->>C: 201 Created

    loop every 5 s
        P->>DB: SELECT FOR UPDATE SKIP LOCKED
        P->>K: publish event
        alt Kafka available
            P->>DB: UPDATE status = COMPLETED
        else Kafka down
            P->>DB: UPDATE retry_count++, next_retry_at
        end
    end

Because both rows are written in a single database transaction, they commit or roll back together. The three failure modes that break the naive approach are all handled:

The Schema

Flyway creates the outbox_events table in V2__create_outbox_events_table.sql. If you're new to Flyway migrations, Spring Boot + Flyway — Database Migrations Step by Step covers the setup in detail.

CREATE TABLE outbox_events (
    id              UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
    event_type      VARCHAR(255) NOT NULL,
    aggregate_id    VARCHAR(255) NOT NULL,
    aggregate_type  VARCHAR(255) NOT NULL,
    topic           VARCHAR(255) NOT NULL,
    payload         JSONB        NOT NULL,
    status          VARCHAR(20)  NOT NULL DEFAULT 'PENDING',
    retry_count     INT          NOT NULL DEFAULT 0,
    last_error      TEXT,
    next_retry_at   TIMESTAMPTZ,
    processed_at    TIMESTAMPTZ,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW()
);

-- Partial index: only covers rows the processor actually needs to scan
CREATE INDEX idx_outbox_pending ON outbox_events (created_at)
    WHERE status = 'PENDING';

Three deliberate choices in this schema:

JSONB payload. Storing the event payload as native JSON means PostgreSQL can index or query it if you ever need to inspect events by field values. It also validates structure at write time — malformed JSON fails immediately.

next_retry_at. Null on fresh rows. Bumped by exponential backoff on each failed delivery attempt. The scheduler ignores rows where next_retry_at > NOW(), so a failing entry backs off without blocking the rest of the batch.

Partial index on PENDING. The scheduler's WHERE status = 'PENDING' query only ever touches unprocessed rows. The index only covers them — completed rows, which are the majority over time, don't bloat it. On a high-volume table with millions of historical events, this keeps query plans fast as the table grows.

Code Walkthrough

1. OrderService — the atomic write

OrderService.createOrder() is the educational centerpiece of the entire pattern. Two writes, one transaction:

@Transactional
public Order createOrder(String customerId, BigDecimal amount) {
    // Step 1: persist the domain entity
    Order order = orderRepository.save(Order.create(customerId, amount));

    // Step 2: write the outbox entry — SAME transaction
    //
    // OutboxWriter has NO @Transactional of its own. It joins the transaction
    // opened by THIS method. If this method rolls back, the outbox row rolls
    // back too — no phantom events ever reach Kafka.
    outboxWriter.write(
            "OrderCreated",
            order.getId().toString(),
            "Order",
            TOPIC,
            toJson(Map.of(
                    "orderId",    order.getId(),
                    "customerId", order.getCustomerId(),
                    "amount",     order.getAmount(),
                    "status",     order.getStatus()
            ))
    );

    return order;
}

Notice what is not here: no kafkaTemplate.send(), no async callback, no try/catch around a broker call. The method's only concern is persisting data. If a validation error later in this method causes a rollback, both the orders row and the outbox_events row disappear together. No event is emitted for a transaction that never committed.

2. OutboxWriter — no transaction boundary of its own

@Component
public class OutboxWriter {

    private final OutboxRepository outboxRepository;

    public OutboxWriter(OutboxRepository outboxRepository) {
        this.outboxRepository = outboxRepository;
    }

    public OutboxEntry write(
            String eventType,
            String aggregateId,
            String aggregateType,
            String topic,
            String payload) {
        OutboxEntry entry = OutboxEntry.create(eventType, aggregateId, aggregateType, topic, payload);
        return outboxRepository.save(entry);
    }
}

OutboxWriter has no @Transactional annotation — deliberately. It participates in whatever transaction is active when it is called. When OrderService.createOrder() calls it, it joins that transaction. This is the whole mechanism: the writer must never open its own boundary, because that would make the outbox write independent of the domain write, which is exactly the problem we are solving.

3. OutboxProcessor — FOR UPDATE SKIP LOCKED

The scheduler runs every 5 seconds by default. It fetches a batch of pending rows and attempts to publish each one:

@Scheduled(fixedDelayString = "${outbox.poll-interval-ms:5000}")
@Transactional
public void processOutbox() {
    List<OutboxEntry> entries = outboxRepository.findPendingWithLock(batchSize);

    if (entries.isEmpty()) return;

    for (OutboxEntry entry : entries) {
        try {
            eventHandler.handle(entry);
            entry.markCompleted();
        } catch (Exception ex) {
            Instant nextRetry = retryPolicy.nextRetryAt(entry.getRetryCount());
            entry.scheduleRetry(nextRetry, ex.getMessage());
        }
        outboxRepository.save(entry);
    }
}

The query behind findPendingWithLock uses PostgreSQL's FOR UPDATE SKIP LOCKED:

@Query(value = """
        SELECT *
        FROM   outbox_events
        WHERE  status = 'PENDING'
          AND  (next_retry_at IS NULL OR next_retry_at <= NOW())
        ORDER  BY created_at ASC
        LIMIT  :limit
        FOR UPDATE SKIP LOCKED
        """, nativeQuery = true)
List<OutboxEntry> findPendingWithLock(@Param("limit") int limit);

FOR UPDATE locks each selected row. SKIP LOCKED skips any row that is already locked by another session rather than waiting on it. In a Kubernetes deployment with three replicas all running the scheduler, each pod grabs a different batch of rows — no waiting, no duplicate publishes to Kafka. Without SKIP LOCKED, all three pods would queue up on the same rows and likely publish duplicates.

Every mutation in processOutbox() commits in the same transaction as the fetch: if the process crashes mid-batch, in-flight rows revert to PENDING and are retried on the next poll.

4. KafkaOutboxEventHandler — sync-over-async

@Override
public void handle(OutboxEntry entry) throws Exception {
    kafkaTemplate
            .send(entry.getTopic(), entry.getAggregateId(), entry.getPayload())
            .get(10, TimeUnit.SECONDS); // block — turns async send into a checked exception
}

Spring's KafkaTemplate.send() is asynchronous by default — it returns a CompletableFuture and moves on. Calling .get(10, TimeUnit.SECONDS) blocks the thread until the broker acknowledges or the timeout fires. This converts a silent async failure into a thrown exception that OutboxProcessor's catch block handles and converts into a scheduled retry. Without the .get(), a Kafka timeout would go undetected and the row would be incorrectly marked COMPLETED.

The message key is aggregateId — all events for the same order land on the same Kafka partition, preserving per-aggregate ordering.

KafkaOutboxEventHandler implements the OutboxEventHandler interface, making it swappable. Swap in a different handler for RabbitMQ, SNS, or any other transport without touching OutboxProcessor.

5. RetryPolicy — exponential backoff

public Instant nextRetryAt(int retryCount) {
    long delayMs = Math.min(
            (long) (initialDelayMs * Math.pow(multiplier, retryCount)),
            maxDelayMs
    );
    return Instant.now().plusMillis(delayMs);
}

With the defaults (initialDelayMs=2000, multiplier=2.0, maxDelayMs=60000), the backoff sequence is:

When Kafka is down for an extended period, entries back off to a 60-second retry cycle rather than hammering the broker on every 5-second poll. All three parameters are configurable in application.yml:

outbox:
  poll-interval-ms: 5000
  batch-size: 100
  retry:
    initial-delay-ms: 2000
    multiplier: 2.0
    max-delay-ms: 60000

Integration Tests

OutboxIntegrationTest spins up real PostgreSQL 16 and Kafka via Testcontainers and proves all three guarantees against the actual implementation — no mocks, no embedded brokers, no spring test overrides that paper over production-incompatible behaviour.

The three tests cover:

• Atomicity: both orders and outbox_events rows are always written together, or not at all
• Full relay: the scheduler transitions PENDING rows to COMPLETED after Kafka acknowledges
• Broker isolation: the business transaction completes regardless of Kafka availability

@SpringBootTest
@Testcontainers
class OutboxIntegrationTest {

    @Container
    static final PostgreSQLContainer<?> postgres =
            new PostgreSQLContainer<>(DockerImageName.parse("postgres:16"))
                    .withDatabaseName("outbox_demo")
                    .withUsername("outbox")
                    .withPassword("outbox");

    @Container
    static final KafkaContainer kafka =
            new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.6.1"));

    @DynamicPropertySource
    static void overrideProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.datasource.url",      postgres::getJdbcUrl);
        registry.add("spring.datasource.username", postgres::getUsername);
        registry.add("spring.datasource.password", postgres::getPassword);
        registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
        registry.add("outbox.poll-interval-ms", () -> "500"); // fast poll in tests
    }
    // ...
}

Test 1: Atomic write

@Test
void createOrder_persistsOrderAndOutboxEntryInSameTransaction() {
    Order order = orderService.createOrder("customer-1", BigDecimal.valueOf(49.99));

    assertThat(order.getId()).isNotNull();

    List<OutboxEntry> entries = outboxRepository.findAll();
    assertThat(entries)
            .anyMatch(e ->
                    e.getAggregateId().equals(order.getId().toString()) &&
                    e.getEventType().equals("OrderCreated") &&
                    e.getStatus() == OutboxStatus.PENDING);
}

After createOrder returns, both the orders row and the outbox_events row exist in the database. The outbox entry is PENDING — the scheduler hasn't run yet. This test verifies that the two writes are truly atomic: if the database persisted the order, the outbox row is always there alongside it.

Test 2: Full relay path

@Test
void afterSchedulerRuns_outboxEntryIsCompletedAndKafkaReceivesMessage() throws Exception {
    Order order = orderService.createOrder("customer-2", BigDecimal.valueOf(99.99));

    // poll-interval-ms = 500 ms in tests — wait up to 10 s
    await()
            .atMost(Duration.ofSeconds(10))
            .untilAsserted(() -> {
                List<OutboxEntry> entries = outboxRepository.findAll();
                assertThat(entries)
                        .anyMatch(e ->
                                e.getAggregateId().equals(order.getId().toString()) &&
                                e.getStatus() == OutboxStatus.COMPLETED);
            });

    try (KafkaConsumer<String, String> consumer = createConsumer()) {
        consumer.subscribe(List.of("orders.events"));
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));

        assertThat(records.count()).isGreaterThan(0);

        boolean found = false;
        for (ConsumerRecord<String, String> record : records) {
            if (order.getId().toString().equals(record.key())) {
                assertThat(record.value()).contains(order.getId().toString());
                found = true;
                break;
            }
        }
        assertThat(found)
                .as("Expected to find Kafka message for order %s", order.getId())
                .isTrue();
    }
}

This test proves the full path end-to-end: business transaction commits → scheduler polls → Kafka receives message with the correct key → row transitions to COMPLETED. Awaitility handles the asynchronous gap between the commit and the scheduler run without brittle Thread.sleep calls.

Test 3: Dual-write problem solved

@Test
void whenKafkaIsDown_orderIsPersistedAndOutboxRemainsForRetry() {
    // createOrder() never touches Kafka — only the background processor does.
    // Kafka availability is irrelevant to the business transaction.
    Order order = orderService.createOrder("customer-3", BigDecimal.valueOf(19.99));

    // Order must be persisted regardless of broker availability
    assertThat(order.getId()).isNotNull();

    // Outbox entry persisted — PENDING, waiting for retry
    List<OutboxEntry> entries = outboxRepository.findAll();
    assertThat(entries)
            .anyMatch(e ->
                    e.getAggregateId().equals(order.getId().toString()) &&
                    e.getStatus() == OutboxStatus.PENDING);
}

This is the most pedagogically important test. It demonstrates the core guarantee: the business transaction is completely isolated from Kafka availability. The order is safe in the database. The outbox row is the durable buffer — it stays PENDING until the broker recovers, at which point the scheduler picks it up and publishes it. No events are silently dropped.

Tradeoffs and Alternatives

Polling latency

The default poll interval is 5 seconds. For most workloads — order processing, notification queues, async workflows — a 5-second event lag is acceptable. If you need sub-second latency, two options:

Reduce the poll interval. Set outbox.poll-interval-ms to 500 ms or lower. This increases database polling load but requires no additional infrastructure.

CDC with Debezium. Capture the outbox_events table's PostgreSQL write-ahead log and stream changes directly to Kafka. Lag drops to milliseconds. The tradeoff: you add a Debezium connector to your infrastructure and take a dependency on PostgreSQL's replication slot mechanism.

At-least-once delivery

The outbox pattern delivers events at least once — never zero, but potentially more than once. If the scheduler publishes to Kafka successfully but crashes before marking the row COMPLETED, the next poll will publish the same event again. Consumers must be idempotent. Use the aggregateId + eventType combination as a deduplication key on the consumer side.

In production, add a max_retry_count check inside OutboxProcessor: when retry_count exceeds your threshold (e.g., 10), transition the row to a DEAD status and alert on it. DEAD rows represent events that require manual intervention or a separate replay process — leaving retry_count unbounded means a persistently broken consumer poisons the outbox queue indefinitely.

Spring Modulith

Spring Modulith 1.x ships a built-in outbox backed by ApplicationModuleListener and Spring Application Events. It is a lower-ceremony option if you are already using Spring Modulith and want the outbox without writing the relay yourself. The tradeoff: you take a framework dependency and give up direct control over the polling query, batch size, retry logic, and handler interface. This implementation gives you all of that — at the cost of roughly 200 lines of code you now own and can tune.

For the broader context of why data consistency in event-driven systems matters — event sourcing, CQRS, the difference between events and commands — see Event-Driven Architecture: Events vs Commands Explained.

Dedicated relay service

For high-volume systems, running the scheduler in the same JVM as the API creates backpressure coupling: a slow Kafka broker can saturate the scheduler's thread pool and affect API latency. Extracting the scheduler into a separate Spring Boot application — with its own resource limits, deployment cadence, and scaling policy — decouples the two concerns completely.

Clone and Run

Prerequisites: Docker, Java 21, Maven 3.9.

# Clone
git clone https://github.com/andrelucasti/outbox-pattern-spring-boot.git
cd outbox-pattern-spring-boot

# Start PostgreSQL 16 + Redpanda
docker compose up -d

# Run the application (starts on :8080)
./mvnw spring-boot:run

Create an order:

curl -s -X POST http://localhost:8080/orders \
     -H "Content-Type: application/json" \
     -d '{"customerId": "mars-001", "amount": 99.99}' | jq .

Watch the outbox row transition from PENDING to COMPLETED within 5 seconds:

psql -h localhost -U outbox -d outbox_demo \
  -c "SELECT id, event_type, status, retry_count, processed_at \
      FROM outbox_events ORDER BY created_at DESC LIMIT 5;"

To see the retry logic in action, stop Redpanda while creating orders and watch retry_count and next_retry_at increment in the outbox table. Restart Redpanda — the scheduler will drain the backlog automatically.

# Simulate broker failure
docker compose stop redpanda

curl -s -X POST http://localhost:8080/orders \
     -H "Content-Type: application/json" \
     -d '{"customerId": "retry-demo", "amount": 29.99}'

# Order is persisted. Outbox entry is PENDING. Check it:
psql -h localhost -U outbox -d outbox_demo \
  -c "SELECT status, retry_count, last_error, next_retry_at FROM outbox_events;"

# Recover the broker — events will publish automatically
docker compose start redpanda

Run the integration tests (Testcontainers manages its own containers — no manual setup):

./mvnw test

Clone it, run it, break it. The full source is at andrelucasti/outbox-pattern-spring-boot — schema, domain service, scheduler, retry policy, integration tests, all runnable in under 5 minutes. No Spring Modulith lock-in, no CDC infrastructure, no proprietary relay. Just a database table, a background scheduler, and careful placement of a single @Transactional annotation.

The next experiment: simulate a broker failure with docker compose stop redpanda, watch retry_count increment in the outbox table, then bring it back and watch the backlog drain automatically.