Spring Cloud Stream Solace Binder Stripe Integration: How requeueRejected=false Routes the Billing Message to Solace DMQ After ch_A Commits Making On-Call Re-Enqueue Generate UUID_B Committing ch_B, How transactedSession=true JCSMP Commit Fails After Spring @Transactional Commits Redelivering the Message with No Content-Hash Guard Committing ch_B, and How Solace Message Replay Re-Delivers 1,247 Already-Charged Events Generating UUID_B × 1,247 Committing ch_B × 1,247

The Spring Cloud Stream Solace Binder introduces three layers of Solace-specific abstraction over the raw JCSMP SDK — dead-message routing with requeueRejected=false as the default, an optional transacted JCSMP session that shares no transaction coordinator with Spring’s @Transactional, and broker-level Message Replay that re-delivers already-acknowledged messages from an append-only replay log — and each layer has a failure mode that produces duplicate Stripe charges in ways that are invisible to developers coming from Kafka, RabbitMQ, or Azure Service Bus.

This post covers all three failure modes end to end: the exact SCS Solace Binder configuration that enables each one, the step-by-step timeline showing when ch_A commits and when UUID_B is generated, the specific misconception each one exploits about the Solace Binder’s behavior, and the idempotency fix that makes your billing consumer safe regardless of DMQ routing, transacted session configuration, or replay scope. Code samples use Spring Cloud Stream 4.x, spring-cloud-stream-binder-solace 5.x, Spring Boot 3.x, WireMock, Testcontainers, and JUnit 5.

Background: how the SCS Solace Binder differs from other SCS binders at the acknowledgement and recovery layers

To understand why these three failure modes exist at the SCS Solace Binder layer rather than at the raw JCSMP SDK layer, you need to understand three structural differences the Solace Binder introduces on top of com.solacesystems:sol-jcsmp.

The SCS Solace Binder defaults requeueRejected=false, meaning rejected messages go to DMQ, not back to the queue. In Solace PubSub+, a consumer that cannot process a message has two settlement options: requeue the message (NACK with requeue=true, which returns the message to the head of the original queue for immediate redelivery) or route to the dead-message queue (NACK with requeue=false, which moves the message to the provisioned DMQ if one exists). The raw JCSMP SDK leaves this decision entirely to the application. The SCS RabbitMQ Binder defaults requeueRejected=true, so rejected messages return to the queue. The SCS Solace Binder defaults requeueRejected=false — rejected messages (after all max-attempts retries are exhausted) are routed to the DMQ. This is a deliberate design choice to avoid poison-pill loops, but it means that every exception the consumer throws after max-attempts exhaustion silently moves the message to a DMQ rather than requeueing it. Stripe charges committed within those attempts are already durable in Stripe’s system. The DMQ message carries no information about whether Stripe was called during those attempts.

The SCS Solace Binder’s transactedSession=true option creates a JCSMP transacted session that is not a JTA participant and does not share a transaction coordinator with Spring’s @Transactional. A Solace JCSMP TransactedSession is a local Solace-layer transaction: when you call transactedSession.commit(), the JCSMP client sends a COMMIT frame to the Solace broker over the data plane connection, and the broker moves all in-transaction messages from the consumer’s in-progress state to acknowledged. This is not a JTA XAResource. It does not register with the Spring JtaTransactionManager. It does not receive rollback callbacks from TransactionSynchronizationManager. When the SCS Solace Binder calls transactedSession.commit() after the consumer function returns normally, that call is made after the Spring @Transactional has already committed the database write. If the JCSMP commit() call fails (broker restart, network loss, ClosedFacilityException after transport acknowledge timeout), the database write is already durable — it cannot be rolled back. The Solace broker, having never received the COMMIT frame, requeues the message for redelivery.

Solace PubSub+ Message Replay is a broker-level feature that re-delivers messages directly from an append-only replay log to a consumer, bypassing the normal queue delivery mechanics and delivery-count tracking. When a Solace endpoint (queue or topic endpoint) has replay enabled, the Solace broker records every published message in a persistent replay log, independent of whether the message is acknowledged in the queue. An administrator or application can instruct the Solace broker to replay all messages from a given timestamp or replay ID, re-delivering them to the currently-subscribed consumer. Unlike Kafka consumer group offset resets (which move the consumer’s position in the partition log) or RocketMQ’s resetOffsetByTime (which resets the consumer group commit offset), Solace Message Replay delivers messages as if they were freshly published. Crucially, replayed messages have their JMSXDeliveryCount set to 1 and the solace_redelivered header set to false — they are indistinguishable from first-time deliveries to the SCS Solace Binder and to your consumer function. If your consumer function calls UUID.randomUUID(), every replayed message generates a fresh idempotency key, producing a fresh Stripe charge regardless of how recently the original charge was committed.

With these three structural differences as context, the three failure modes follow directly from specific configurations and operational patterns.

Failure mode 1: requeueRejected=false + autoBindDmq=true — SCS exhausts max-attempts=3, NACKs to Solace DMQ after ch_A through ch_C commit; on-call re-enqueue generates UUID_D committing ch_D

The setup

A team runs a usage-based billing service for a developer-tools platform. The message broker is Solace PubSub+ Cloud. The Spring Boot 3.x billing service uses the SCS Solace Binder to consume metered usage events from the billing/created topic (bound to a persistent queue billing-processor) and convert them to Stripe charges. The team uses the SCS Solace Binder defaults for acknowledgement:

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing/created
          group: billing-processor
          consumer:
            max-attempts: 3
            back-off-initial-interval: 1000
            back-off-multiplier: 2.0
      solace:
        bindings:
          billing-in-0:
            consumer:
              autoBindDmq: true          # provision DMQ automatically
              # requeueRejected: false   # this is the DEFAULT — not set explicitly

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // BUG: fresh random UUID on every consumer invocation (retry or redelivery)
        String idempotencyKey = UUID.randomUUID().toString();

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        // DB write — can fail under connection pool exhaustion
        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            idempotencyKey,
            charge.getId()
        ));
    };
}

The timeline

The billing service is under elevated load during end-of-month billing. The DB connection pool is at 95% utilization. A billing event M1 arrives for account acct_42, billing period 2024-03, amountCents=29900:

Attempt 1 (t=0ms): SCS delivers M1 to the consumer function. UUID.randomUUID() → idem_A = "f47ac10b-58cc-4372-a567-0e02b2c3d479". Stripe call begins. At t=430ms Stripe commits ch_A (HTTP 200, charge id ch_stripe_abc123). At t=435ms billingRecordRepo.save() throws CannotGetJdbcConnectionException (connection pool exhausted). Consumer function throws. ch_A is committed in Stripe.

Attempt 2 (t=1000ms, after 1s back-off): SCS RetryTemplate re-invokes the consumer function with the same M1 message. UUID.randomUUID() → idem_B = "9d2c83e1-4f12-4b8e-bce0-7a1d6f8e2c45". Stripe call begins. At t=1430ms Stripe commits ch_B. At t=1435ms billingRecordRepo.save() throws CannotGetJdbcConnectionException again. Consumer function throws. ch_A and ch_B are now both committed in Stripe.

Attempt 3 (t=3000ms, after 2s back-off): SCS RetryTemplate re-invokes one final time. UUID.randomUUID() → idem_C = "1a3b5c7d-9e2f-4a6b-8c0d-2e4f6a8b0c2d". Stripe commits ch_C at t=3430ms. billingRecordRepo.save() throws again. SCS has now exhausted max-attempts=3. The exception propagates to the SCS Solace Binder’s error handler.

NACK to DMQ (t=3440ms): The SCS Solace Binder calls messageConsumer.settle(message, Outcome.FAILED) on the underlying JCSMP consumer. Because requeueRejected=false, the JCSMP consumer tells the Solace broker to route M1 to the provisioned DMQ (the auto-bound DMQ queue is named billing/created/!q/#billing-processor by default). M1 disappears from the billing-processor queue and appears in the DMQ. The on-call alerting system detects a non-zero DMQ depth and pages the on-call engineer.

On-call re-enqueue (t=2h later): The on-call engineer opens the Solace Console, sees M1 in the DMQ with headers JMSXDeliveryCount=3 and the original message body intact. The on-call runbook reads: “DMQ messages represent billing events that failed to process. Check the reason for failure (usually a DB connectivity issue). Once the underlying issue is resolved, re-enqueue the message from the DMQ to the billing-processor queue to retry.” The engineer re-enqueues M1. The consumer receives it. UUID.randomUUID() → idem_D = "3c5e7f91-a2b4-4c6d-8e0f-2a4c6e8f0a2c". The DB connection pool has recovered. Stripe commits ch_D. billingRecordRepo.save() succeeds. The engineer marks the incident resolved.

Account acct_42 has been charged four times: ch_A ($299.00), ch_B ($299.00), ch_C ($299.00), and ch_D ($299.00). Total: $1,196.00 for a $299.00 billing event. The engineer and the on-call runbook are unaware of this — no error was logged for the Stripe calls, only for the DB save that followed each one.

The misconception

“Messages in the Solace DMQ were never successfully processed. If a billing message is in the DMQ, it means my consumer function crashed before any external side effect was committed — either before the Stripe call was made, or the Stripe call itself failed with a network error, or the Stripe charge was declined. Re-enqueuing a billing message from the DMQ is therefore safe: whatever the original attempt tried to do, it failed, and re-enqueuing gives the message a fresh start. The JMSXDeliveryCount=3 header tells me the message was attempted three times, which means three failures, not three successful Stripe charges alongside a failed DB write.”

This is wrong on two counts. First, the Solace DMQ receives messages that exhausted all retries — it says nothing about the outcome of any individual attempt. A message reaching the Solace DMQ could have spent those three attempts committing three Stripe charges and failing only at the last DB write after each one. The DMQ header JMSXDeliveryCount=3 records the number of delivery attempts, not the number of Stripe calls and not the number of failures at the Stripe layer vs. the DB layer. Second, the on-call runbook’s premise — “DMQ = pre-Stripe failure” — is derived from the behavior of systems where retries are not wired to Stripe (e.g., a consumer that validates input, writes to DB, and then publishes a downstream event). When Stripe is in the hot path of the consumer function and is called before the final DB write, any exception after the Stripe call but before the DB write will land the message in the DMQ with the Stripe charge already committed.

This failure mode differs from the equivalent failure in other SCS binders in a Solace-specific way: the final resting place of the message is a DMQ, not a requeue. In the SCS RabbitMQ Binder with requeueRejected=true (the default), a failed message returns to the RabbitMQ queue for immediate redelivery — the developer sees the message cycling back and eventually routes it to the dead-letter exchange after x-max-delivery-count. In the SCS Kafka Binder, a failed message is published to the dead-letter topic by the binder’s error handler — the original topic partition offset is committed regardless. In the SCS Solace Binder with requeueRejected=false, the message is silently NACKed to the DMQ on the first retry exhaustion. The engineer never sees the message bouncing. The Solace Console shows billing-processor queue depth: 0 and DMQ depth: 1. The operator interprets this as “one message failed to process” rather than “one message failed to write to DB after three successful Stripe charges.”

The fix

Use a content-hash idempotency key derived from the billing event’s immutable fields, and write the BillingRecord guard before making the Stripe call rather than after. With a stable key, every retry attempt and every DMQ re-enqueue uses the same idempotency key string — Stripe’s 24-hour cache returns the existing charge data without committing a new charge:

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        String idempotencyKey = sha256Hex(
            event.getAccountId() + "|" +
            event.getBillingPeriod() + "|" +
            event.getAmountCents() + "|" +
            event.getUnitsConsumed()
        );

        // Content-based guard: blocks on (accountId, billingPeriod), not on UUID
        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            return; // already processed — retry or DMQ re-enqueue, no action needed
        }

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            idempotencyKey,
            charge.getId()
        ));
    };
}

With this fix, the first SCS retry that reaches Stripe sends the same idempotencyKey as the first attempt. Stripe’s API returns the same ChargeCreateResponse from its idempotency cache — no new charge is committed. When the DMQ message is re-enqueued and the consumer function runs again, billingRecordRepo.existsByAccountIdAndBillingPeriod() returns true (if the DB write succeeded on any prior attempt) and the function returns immediately. If the DB write never succeeded despite 3 Stripe calls all returning ch_A from cache, the re-enqueue succeeds at the DB write and saves the correct ch_A charge id — no duplicate charges.

Additionally, set maxDeliveryAttempts=1 in the SCS Solace Binder to prevent the max-attempts multiplier from making multiple Stripe calls within a single delivery:

spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          consumer:
            max-attempts: 1              # don't retry within a delivery; let Solace redeliver
      solace:
        bindings:
          billing-in-0:
            consumer:
              autoBindDmq: true
              requeueRejected: true      # requeue on first failure; let stable key deduplicate
              maxDeliveryAttempts: 5     # broker-level redelivery limit before DMQ routing

With requeueRejected=true and maxDeliveryAttempts=5, the Solace broker itself manages redelivery, and each redelivery uses the same stable idempotency key. The DMQ then only receives messages that truly cannot be processed after 5 broker-managed redeliveries — at which point the stable key ensures that whatever Stripe charges were committed are all identical (ch_A).

Failure mode 2: transactedSession=true — JCSMP session.commit() called after Spring @Transactional DB commit; JCSMP commit fails; Solace redelivers; UUID.randomUUID() generates UUID_B; Stripe commits ch_B alongside existing BillingRecord written with UUID_A

The setup

A team migrates their billing service from ActiveMQ (where they use Spring’s JmsTransactionManager to coordinate JMS session commits with database transactions) to Solace PubSub+. They know that transactional messaging is important for billing correctness and look for the equivalent of JmsTransactionManager in the SCS Solace Binder. They find transactedSession=true and enable it, believing it coordinates the Solace ACK with the Spring @Transactional the same way JmsTransactionManager coordinates a JMS session commit.

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing/created
          group: billing-processor
          consumer:
            max-attempts: 1   # no SCS-level retries; rely on Solace redelivery
      solace:
        bindings:
          billing-in-0:
            consumer:
              transactedSession: true   # enable JCSMP transacted session

@Bean
public Consumer<BillingEvent> billing() {
    return event -> billingService.chargeAndRecord(event);
}

@Service
public class BillingService {

    @Transactional  // manages the PostgreSQL transaction
    public void chargeAndRecord(BillingEvent event) {
        // BUG: UUID.randomUUID() generates a different key on every call
        String idempotencyKey = UUID.randomUUID().toString();

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            idempotencyKey,
            charge.getId()
        ));
        // @Transactional commits here — BillingRecord is durable in PostgreSQL
    }
    // SCS Solace Binder calls transactedSession.commit() after this method returns
}

The timeline

The service is running in Kubernetes. A Solace PubSub+ Cloud maintenance window causes a brief TCP RST on broker connections at t+441ms — timed to coincide exactly with the JCSMP COMMIT frame in-flight:

Delivery 1 (t=0ms): Solace delivers M1 for acct_42, billingPeriod=2024-03, amountCents=29900. The SCS Solace Binder starts JCSMP transacted session T1. The consumer function is invoked.

t=5ms: UUID.randomUUID() → idem_A = "a1b2c3d4-e5f6-7890-abcd-ef1234567890".

t=10ms: stripeClient.charges().create() called with idempotencyKey=idem_A.

t=430ms: Stripe commits ch_A (HTTP 200, id="ch_stripe_XYZ789"). The Stripe charge is durable.

t=435ms: billingRecordRepo.save() called. JPA/Hibernate inserts the BillingRecord row into PostgreSQL. The INSERT is in the open @Transactional transaction, not yet committed.

t=440ms: chargeAndRecord() returns. Spring’s @Transactional proxy calls DataSourceTransactionManager.commit(). PostgreSQL commits the transaction. The BillingRecord row (accountId=acct_42, billingPeriod=2024-03, idempotencyKey=idem_A, chargeId=ch_stripe_XYZ789) is now durable in PostgreSQL.

t=441ms: The billing() consumer function returns normally. The SCS Solace Binder calls jcsmpSession.getTransactionSession().commit(). This sends a JCSMP COMMIT frame to the Solace broker data plane connection.

t=441ms (concurrent): The Solace PubSub+ Cloud maintenance operation sends a TCP RST on the broker connection. The JCSMP COMMIT frame is lost in transit. The Solace broker never receives it.

t=10441ms (10 seconds later): The JCSMP transport acknowledge timeout (com.solacesystems.jcsmp.JCSMPProperties.TRANSPORT_ACKNOWLEDGE_TIMEOUT_MS, default 10,000ms) expires. JCSMP throws ClosedFacilityException: Transacted Session is closed. The SCS Solace Binder catches this and logs ERROR ... Failed to commit transacted session. The consumer function has already returned normally; there is nothing for the SCS Binder to do about the DB write — @Transactional committed 10 seconds ago.

Solace broker side: The broker never received the COMMIT frame for transacted session T1. After the connection drops and the TCP session is cleaned up, the broker marks T1’s in-progress messages as unacknowledged and requeues them. M1 returns to the billing-processor queue with its delivery count incremented to 2.

Delivery 2 (t=10500ms): Solace redelivers M1. The SCS Binder starts a new transacted session T2. The consumer function is invoked again.

t=10505ms: UUID.randomUUID() → idem_B = "f9e8d7c6-b5a4-3210-fedc-ba9876543210". Completely different string from idem_A.

t=10510ms: stripeClient.charges().create() called with idempotencyKey=idem_B. Because idem_B != idem_A, Stripe’s idempotency cache does not match. Stripe processes this as a new charge request.

t=10940ms: Stripe commits ch_B (HTTP 200, id="ch_stripe_ABC456"). A second Stripe charge of $299.00 is durable.

t=10945ms: billingRecordRepo.save() throws DataIntegrityViolationException: duplicate key value violates unique constraint billing_record_account_id_billing_period_key. The row (acct_42, 2024-03) already exists from delivery 1. @Transactional rolls back the delivery-2 DB write. But ch_B is already committed in Stripe and cannot be rolled back.

The account is charged $598.00: ch_A ($299.00) from the first delivery and ch_B ($299.00) from the second delivery. The DB has only one BillingRecord row, correctly recording idem_A and ch_A. The second charge ch_B is invisible to the application — it exists only in Stripe.

The misconception

“Enabling transactedSession=true in the SCS Solace Binder coordinates the Solace message acknowledgement with my Spring @Transactional database transaction, similar to how JmsTransactionManager coordinates a JMS session commit with the database transaction. If the @Transactional commits — meaning the database write succeeded — the Solace JCSMP transacted session will also commit atomically. If the JCSMP commit fails for any reason, the @Transactional will roll back the database write, leaving the system in a consistent state where either both the DB row and the Solace ACK committed or neither did. This makes my billing consumer safe against double-charges: if the database never got the row, Solace will redeliver the message, and a fresh Stripe charge will be the correct action.”

This is wrong on three counts. First, the JCSMP TransactedSession is not a JTA XAResource. It does not register with Spring’s JtaTransactionManager. It does not receive TransactionSynchronization.afterCompletion() callbacks. The Spring @Transactional and the JCSMP TransactedSession are managed by two completely independent transaction managers: DataSourceTransactionManager (or JpaTransactionManager) for the DB, and the SCS Solace Binder’s internal JCSMP session management for the Solace ACK. There is no shared coordinator.

Second, the ordering of operations in the SCS Solace Binder with transactedSession=true is:

1. Consumer function is called.
2. Consumer function calls @Transactional service method.
3. @Transactional proxy commits the DB transaction (PostgreSQL transaction committed, row is durable).
4. Consumer function returns.
5. SCS Solace Binder calls jcsmpSession.getTransactionSession().commit().

Steps 3 and 5 are sequential, not atomic. If step 5 fails, step 3 cannot be reversed. The PostgreSQL transaction commit at step 3 is a durable operation — once PostgreSQL returns COMMIT, no subsequent exception in the same JVM process can undo it.

Third, the JmsTransactionManager comparison is misleading in a specific way. Spring’s JmsTransactionManager manages both the JMS session and the ConnectionFactory as transactional resources together within a JmsTransactionManager-managed transaction. When you use JmsTransactionManager with a JMS-aware DataSourceTransactionManager, you are typically using a ChainedTransactionManager or a single JmsTransactionManager that does not coordinate with the DB transaction at all. True atomic coordination of JMS + DB requires JTA with XA (using JtaTransactionManager with XAConnectionFactory and XADataSource). Most applications that used JmsTransactionManager with ActiveMQ and a JDBC datasource were NOT getting XA atomicity — they were getting best-effort coordination. The Solace JCSMP TransactedSession provides even less: it provides a Solace-internal transaction that allows you to receive multiple messages and commit or roll back their acknowledgement atomically on the Solace side, but it has no coupling to the DB whatsoever.

The fix

Use a stable content-hash idempotency key and a content-based guard that checks (accountId, billingPeriod) rather than a per-invocation UUID. The guard must run before the Stripe call:

@Service
public class BillingService {

    @Transactional
    public void chargeAndRecord(BillingEvent event) {
        // Stable key: same string on every invocation for the same billing event
        String idempotencyKey = sha256Hex(
            event.getAccountId() + "|" +
            event.getBillingPeriod() + "|" +
            event.getAmountCents() + "|" +
            event.getUnitsConsumed()
        );

        // Content-based guard: blocks on business identity, not UUID
        // If JCSMP commit failed and Solace redelivers, this guard catches it
        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            return; // BillingRecord already exists from a prior successful delivery
        }

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            idempotencyKey,
            charge.getId()
        ));
    }
}

With this fix, the JCSMP commit failure re-delivery follows this path: existsByAccountIdAndBillingPeriod(acct_42, 2024-03) → true (the row was committed at t=440ms in delivery 1) → function returns immediately → no Stripe call → no ch_B. The SCS Solace Binder calls transactedSession.commit() again after the consumer function returns normally, this time on the reconnected broker session, and the commit succeeds. M1 is acknowledged. The account is charged exactly once.

The stable key provides defense in depth: even if the guard is bypassed (race condition between two concurrent deliveries of the same message to two different pods), the Stripe idempotency key ensures both pods’ Stripe calls return ch_A from the Stripe cache rather than committing ch_A and ch_B independently.

Whether to keep transactedSession=true after applying the fix is a separate question. With a stable key and content-based guard, the transacted session provides no additional billing safety — the fix makes the consumer idempotent regardless of how many times it is invoked. The transacted session does provide one Solace-specific benefit: it allows multiple messages to be acknowledged atomically in a single commit() call (useful for batch consumers). If your consumer processes messages one at a time, the default (transactedSession=false) with stable key + guard is equally safe and simpler.

Failure mode 3: Solace Message Replay re-delivers 1,247 pre-outage billing events; UUID.randomUUID() × 1,247 generates UUID_B_1 through UUID_B_1247; Stripe commits ch_B_1 through ch_B_1247

The setup

A team runs a high-volume billing service on Solace PubSub+ Cloud with Message Replay enabled on the billing/created queue. The Solace replay log retains 7 days of messages. The billing consumer calls UUID.randomUUID() per delivery and has been running in production without incident for months — the normal steady-state has no Solace redeliveries, so the UUID instability has never caused a problem. The operations team has a documented disaster-recovery runbook: “In the event of a downstream outage (database, Stripe API, etc.), start Solace Message Replay from 10 minutes before the outage window to ensure full recovery with a safety buffer.”

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // BUG: fresh UUID on every consumer invocation
        String idempotencyKey = UUID.randomUUID().toString();
        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );
        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(), event.getBillingPeriod(),
            idempotencyKey, charge.getId()
        ));
    };
}

The timeline

Normal processing (11:30:00 UTC – 11:47:23 UTC): The billing consumer processes messages normally. Between 11:30:00 and 11:47:23, it processes 2,094 billing events. Of these, 1,247 events fall in the pre-outage window 11:40:00–11:47:23 (relevant because of the 10-minute replay buffer in the runbook). Each is processed with a unique UUID.randomUUID() key: ch_A_1 through ch_A_1247 are committed in Stripe. BillingRecord rows are written to PostgreSQL for all 1,247 events.

PostgreSQL outage (11:47:23 UTC – 12:04:51 UTC, 17m 28s): A cascading connection pool failure takes PostgreSQL offline. The billing consumer’s calls to billingRecordRepo.save() throw CannotGetJdbcConnectionException. With max-attempts=3 and requeueRejected=true, the consumer retries each message three times and then requeues it to the billing-processor queue. Because the Stripe call happens before the DB write, the consumer is also calling Stripe three times per message with fresh UUIDs — but under the short 17-minute outage, only messages that were in-flight at the moment of the outage are affected (max-attempts keeps them cycling). 847 billing events from 11:47:23 to 12:04:51 are cycling in the queue with unacknowledged Stripe calls.

PostgreSQL recovery (12:04:51 UTC): The SRE resolves the outage. The 847 cycled messages are redelivered from the queue (with incremented JMSXDeliveryCount). Because they used UUID.randomUUID() on each cycle, the SCS Solace Binder re-calls Stripe with new UUIDs. Some of these already had Stripe charges committed (pre-DB-write failure); some had not. The SRE cannot determine which.

SRE triggers Solace Message Replay (12:08:00 UTC): Following the disaster recovery runbook, the SRE uses the Solace Java API to trigger replay from 10 minutes before the outage start: ReplayStartLocation.createByDate(Instant.parse("2024-03-15T11:37:23Z")) (10 minutes before 11:47:23). The SCS Solace Binder receives the REPLAY_STARTED event and begins re-delivering messages from the replay log starting at 11:37:23 UTC.

The replay delivers:

• Messages from 11:37:23–11:40:00 UTC: 412 events (pre-buffer; the runbook’s 10-minute buffer extends to here). These were already processed at 11:30–11:47, so they all have existing BillingRecord rows. UUID.randomUUID() generates fresh keys; Stripe commits new charges.
• Messages from 11:40:00–11:47:23 UTC: 1,247 events already processed successfully. Each generates a fresh UUID. Stripe commits new charges ch_B_1 through ch_B_1247.
• Messages from 11:47:23–12:04:51 UTC: 847 events from the outage window. These are the ones the SRE intended to replay. Each generates a fresh UUID. Stripe commits charges for the outage-window events correctly (these are genuinely new charges).
• Messages from 12:04:51 UTC onwards: Already processed by the recovered consumer from the queue. Replay re-delivers these too, causing additional doubles.

Total: 1,247 + 412 = 1,659 duplicate charges committed by Stripe, plus duplicate charges for the post-recovery messages. The SRE intended to replay 847 events; the replay produced 1,659 unintended additional charges.

The misconception

“Solace Message Replay re-delivers the same message bytes that were originally published. Since Stripe stores idempotency keys for 24 hours, the replayed messages will hit the Stripe idempotency cache and return the existing charge data rather than committing new charges — as long as we run the replay within 24 hours of the original processing. The replay is therefore safe to run with a wide safety buffer: the pre-outage events will be deduplicated by Stripe, and only the outage-window events will actually create new charges.”

“Also, the SCS Solace Binder sets solace_redelivered=true or increments JMSXDeliveryCount on replayed messages, so I can detect them in my consumer function and skip the Stripe call for messages that were already processed.”

Both parts of this misconception are wrong in Solace-specific ways.

On the Stripe 24-hour idempotency cache: Stripe’s idempotency cache deduplicates by the exact value of the Idempotency-Key HTTP header. It does not inspect the request body, the charge amount, the customer ID, or any other field — it deduplicates by key string only. If the consumer calls UUID.randomUUID() on each invocation and the replay delivers the same message bytes, the consumer will generate a completely different UUID string on the replayed delivery. Stripe has no way to match UUID_B_1 = "3d9f2a81-c4e5-4b7d-a8f3-1c2d3e4f5a6b" against UUID_A_1 = "f47ac10b-58cc-4372-a567-0e02b2c3d479". They are different strings; Stripe treats them as different charge requests. The Stripe 24-hour cache only helps if you send the same idempotency key string as the original request — which requires a stable content-hash key derived from the message content, not a random UUID generated at call time.

On detecting replayed messages via Solace headers: This is wrong about how Solace Message Replay interacts with delivery headers. In Solace PubSub+:

• JMSXDeliveryCount is incremented each time the broker redelivers a message that was previously NACKed or that had its lock expire (i.e., normal redelivery within the queue). For messages delivered via the Message Replay feature, the delivery is sourced from the append-only replay log, not from the queue’s in-flight state. Replay messages have JMSXDeliveryCount=1 — they are treated as first-time deliveries from the broker’s perspective.
• solace_redelivered (the SCS Solace Binder’s SolaceHeaders.REDELIVERED header) reflects the JCSMP XMLMessage.getRedelivered() flag. This flag is set to true only for messages that are redelivered due to a NACK within the same consumer session (i.e., the same JCSMP FlowReceiver was the original recipient and is now receiving the redelivery). For Message Replay, the messages come from the replay log as fresh deliveries — XMLMessage.getRedelivered() returns false. The SCS Solace Binder sets SolaceHeaders.REDELIVERED=false on all replayed messages.

There is one Solace header that can detect replayed messages in some configurations: SolaceHeaders.IS_REPLAY, which the Solace JCSMP SDK sets on messages sourced from the replay log if the endpoint is configured to surface it. However, this header is not reliably present in all Solace PubSub+ versions and configurations, and relying on it creates a fragile detection layer that breaks silently when the broker is upgraded or the endpoint configuration changes. The robust solution — as with all idempotency problems in this series — is to use a stable content-hash key that makes the consumer idempotent regardless of how it detects or fails to detect replays.

The fix

Use a stable content-hash idempotency key. With a stable key, every replayed message sends the same idempotency key string to Stripe as the original delivery. Stripe’s idempotency cache returns the existing charge data. No new charges are committed for any pre-outage event:

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // Stable key: same string for the same billing event, always
        String idempotencyKey = sha256Hex(
            event.getAccountId() + "|" +
            event.getBillingPeriod() + "|" +
            event.getAmountCents() + "|" +
            event.getUnitsConsumed()
        );

        // Content-based guard (fast path for known-processed events)
        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            return;
        }

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(), event.getBillingPeriod(),
            idempotencyKey, charge.getId()
        ));
    };
}

With this fix, the SRE’s replay from 11:37:23 UTC processes all 2,094 + safety-buffer events as follows:

• 1,659 pre-outage events with existing BillingRecord rows: existsByAccountIdAndBillingPeriod() → true → early return → no Stripe call. Zero duplicate charges.
• 847 outage-window events without BillingRecord rows: existsByAccountIdAndBillingPeriod() → false → Stripe called with stable key → new charge committed → BillingRecord saved. Correct recovery.
• Any post-recovery events replayed again: Already have BillingRecord rows → early return → zero additional charges.

The replay is fully idempotent. The SRE can run it multiple times with any start timestamp — wider safety buffers produce no additional charges. This also means the disaster-recovery runbook can be simplified: “If DB is recovered and you are unsure which events were processed, replay from any timestamp in the last 7 days. The consumer is idempotent — no duplicate charges will result.”

The stable key fix also handles the secondary problem from this scenario: the 847 outage-window events that were cycling in the queue with requeueRejected=true and max-attempts=3 were also generating fresh UUIDs on each SCS retry cycle. With a stable key, those cycled deliveries all use the same idempotency key → Stripe charges each account once → no additional doubles from the cycling.

Cross-layer comparison: raw Solace JCSMP, SCS Solace Binder, Spring Integration Solace, SCS Kafka

Diagnostic checklist: 8 questions before deploying a Solace-backed billing consumer

1. Is your Stripe idempotency key derived from the billing event’s immutable content fields (account ID, billing period, amount cents, units consumed), or from UUID.randomUUID()? If the latter, every SCS retry attempt, DMQ re-enqueue, transacted session redelivery, and Message Replay will generate a new key and a new Stripe charge.
2. Is your pre-flight guard based on (accountId, billingPeriod) with a unique constraint in the database, or on the Stripe idempotency key string? A guard based on the idempotency key string only works if the key is stable. If you switch from UUID keys to hash keys without updating the guard’s lookup column, the guard will miss duplicates until the column is re-indexed on the new hash.
3. Is requeueRejected set explicitly in your SCS Solace Binder consumer configuration? The default is false, meaning failed messages go to DMQ silently. If your on-call runbook includes “re-enqueue from DMQ,” that runbook must be updated to state that DMQ re-enqueues can duplicate Stripe charges unless stable keys and content guards are in place.
4. Does your Solace billing queue have Message Replay enabled? If yes, your disaster recovery runbook’s replay instructions will cause mass duplicate charges unless the consumer uses stable idempotency keys. Run a test replay in a staging environment against a Stripe test-mode account and verify the charge count.
5. Is transactedSession=true configured in your SCS Solace Binder consumer? If yes, verify that you are not using it under the assumption that it coordinates with Spring @Transactional. The JCSMP TransactedSession is not a JTA participant. A content-based guard handles the JCSMP commit failure re-delivery scenario regardless of transacted session configuration.
6. Does your consumer function return normally when the billing event has already been processed (guard returns true)? Returning normally causes the SCS Solace Binder to ACK (or commit the transacted session for) the duplicate delivery. If the consumer throws an exception instead, the binder may redeliver or DMQ-route the already-processed message, creating unnecessary DMQ pressure.
7. Is the setIdempotencyKey() call on the Stripe RequestOptions builder actually wired to the content-hash string and not to a field that defaults to null or empty? Stripe accepts requests with no idempotency key — they are treated as non-idempotent. A null or empty idempotency key silently disables deduplication. Log the key at DEBUG level and verify it is non-null and non-empty in your integration tests.
8. Have you verified that the unique constraint on (accountId, billingPeriod) in your billing_records table is enforced at the database level (DDL UNIQUE constraint) and not only at the application level (existsByAccountIdAndBillingPeriod() check)? Under concurrent delivery (two pods receiving M1 simultaneously), the application-level check can pass on both pods before either completes the DB write, resulting in two inserts and two Stripe charges before the constraint violation is surfaced. The DDL constraint ensures that one insert wins and one throws DataIntegrityViolationException, making the stable-key Stripe deduplication the only active charge.

Tests: JUnit 5 + WireMock + Testcontainers + Solace PubSub+ emulator

The following tests use testcontainers-solace (the Solace PubSub+ Standard image available as a Testcontainers module), WireMock for Stripe, and JUnit 5. Each test isolates one failure mode and verifies both the broken behavior (without stable key) and the fixed behavior (with stable key).

@SpringBootTest
@Testcontainers
class SolaceStripeIdempotencyTest {

    @Container
    static SolaceContainer solace = new SolaceContainer("solace/solace-pubsub-standard:latest")
        .withExposedPorts(55555, 8080, 5672);

    @Container
    static WireMockContainer wireMock = new WireMockContainer("wiremock/wiremock:3.x")
        .withMappingFromResource("stripe-charge-success.json");

    @Autowired SolaceAdmin solaceAdmin;
    @Autowired BillingRecordRepository billingRecordRepo;
    @Autowired SolaceTemplate solaceTemplate;

    static final String BILLING_TOPIC = "billing/created";
    static final String BILLING_QUEUE = "billing-processor";
    static final String BILLING_DMQ   = "billing/created/!q/#billing-processor";

    // ── Mode 1: requeueRejected=false sends to DMQ ─────────────────────────

    @Test
    @DisplayName("Mode 1: DMQ re-enqueue after ch_A commits → UUID_B → ch_B (without stable key)")
    void dmqReenqueue_withoutStableKey_chargesDuplicate() throws Exception {
        // Arrange: Stripe always returns 200 regardless of idempotency key
        List<String> capturedKeys = Collections.synchronizedList(new ArrayList<>());
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody(chargeJson("ch_A"))));

        // Simulate DB failure on first 3 attempts, success on 4th
        AtomicInteger saveCount = new AtomicInteger(0);
        when(billingRecordRepo.save(any())).thenAnswer(inv -> {
            if (saveCount.incrementAndGet() < 4) {
                throw new CannotGetJdbcConnectionException("pool exhausted");
            }
            return inv.getArgument(0);
        });

        // Act: publish M1, let 3 SCS retries exhaust, capture DMQ message, re-enqueue
        BillingEvent event = new BillingEvent("acct_42", "2024-03", 29900L, 1000);
        solaceTemplate.publish(BILLING_TOPIC, event);

        await().atMost(Duration.ofSeconds(10))
               .until(() -> solaceAdmin.getQueueDepth(BILLING_DMQ) == 1);

        // Simulate on-call re-enqueue from DMQ
        solaceAdmin.moveFromDmqToQueue(BILLING_DMQ, BILLING_QUEUE);

        await().atMost(Duration.ofSeconds(10))
               .until(() -> saveCount.get() >= 4);

        // Assert: 4 Stripe calls made (3 retries + 1 re-enqueue)
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSize(4);
        // Each request has a different Idempotency-Key
        Set<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(keys).hasSize(4); // 4 different keys = 4 potential charges
    }

    @Test
    @DisplayName("Mode 1: DMQ re-enqueue after ch_A commits → stable key → idempotent (with fix)")
    void dmqReenqueue_withStableKey_deduplicates() throws Exception {
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200).withBody(chargeJson("ch_A"))));

        AtomicInteger saveCount = new AtomicInteger(0);
        when(billingRecordRepo.save(any())).thenAnswer(inv -> {
            if (saveCount.incrementAndGet() < 4) {
                throw new CannotGetJdbcConnectionException("pool exhausted");
            }
            return inv.getArgument(0);
        });

        BillingEvent event = new BillingEvent("acct_42", "2024-03", 29900L, 1000);
        solaceTemplate.publish(BILLING_TOPIC, event);

        await().atMost(Duration.ofSeconds(10))
               .until(() -> solaceAdmin.getQueueDepth(BILLING_DMQ) == 1);

        solaceAdmin.moveFromDmqToQueue(BILLING_DMQ, BILLING_QUEUE);

        await().atMost(Duration.ofSeconds(10))
               .until(() -> saveCount.get() >= 4);

        // Assert: all 4 Stripe requests used the SAME idempotency key
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSize(4);
        Set<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(keys).hasSize(1); // one stable key across all 4 attempts
    }

    // ── Mode 2: transactedSession=true JCSMP commit failure ────────────────

    @Test
    @DisplayName("Mode 2: JCSMP commit failure redelivery → UUID_B → ch_B (without stable key)")
    void jcsmpCommitFailure_withoutStableKey_chargesDuplicate() throws Exception {
        // Arrange: Stripe accepts any idempotency key
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200)
                .withBody(chargeJson("ch_" + UUID.randomUUID()))));

        // Simulate JCSMP commit failure on first delivery:
        // publish M1, let first delivery succeed at DB but fail at JCSMP commit
        solaceAdmin.simulateJcsmpCommitFailureOnNextDelivery(BILLING_QUEUE);

        BillingEvent event = new BillingEvent("acct_42", "2024-03", 29900L, 1000);
        solaceTemplate.publish(BILLING_TOPIC, event);

        // Wait for 2 deliveries (original + redelivery after JCSMP commit failure)
        await().atMost(Duration.ofSeconds(15))
               .until(() -> billingRecordRepo.count() == 1); // only one DB row

        // Assert: 2 Stripe calls with different keys = 2 charges
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSize(2);
        Set<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(keys).hasSize(2); // 2 different keys = 2 charges
    }

    @Test
    @DisplayName("Mode 2: JCSMP commit failure redelivery → stable key + guard → idempotent (with fix)")
    void jcsmpCommitFailure_withStableKey_deduplicates() throws Exception {
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200).withBody(chargeJson("ch_A"))));

        solaceAdmin.simulateJcsmpCommitFailureOnNextDelivery(BILLING_QUEUE);

        BillingEvent event = new BillingEvent("acct_42", "2024-03", 29900L, 1000);
        solaceTemplate.publish(BILLING_TOPIC, event);

        await().atMost(Duration.ofSeconds(15))
               .until(() -> billingRecordRepo.count() == 1);

        // Assert: guard catches redelivery; only 1 Stripe call made
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSize(1);
    }

    // ── Mode 3: Solace Message Replay ──────────────────────────────────────

    @Test
    @DisplayName("Mode 3: Message Replay of 10 pre-outage events → UUID_B × 10 → ch_B × 10 (without stable key)")
    void messageReplay_withoutStableKey_chargesAllEventsAgain() throws Exception {
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200).withBody(chargeJson("ch_NEW"))));

        // Arrange: publish and process 10 billing events normally
        for (int i = 1; i <= 10; i++) {
            solaceTemplate.publish(BILLING_TOPIC,
                new BillingEvent("acct_" + i, "2024-03", 29900L, 1000));
        }
        await().atMost(Duration.ofSeconds(10))
               .until(() -> billingRecordRepo.count() == 10);

        long stripeCallsBefore = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges"))).size();
        assertThat(stripeCallsBefore).isEqualTo(10); // 10 original charges

        // Act: trigger Solace Message Replay from before the 10 events
        Instant replayFrom = Instant.now().minusSeconds(60);
        solaceAdmin.startReplay(BILLING_QUEUE, replayFrom);

        // Wait for replay to complete
        await().atMost(Duration.ofSeconds(15))
               .until(() -> wireMock.findAll(
                   postRequestedFor(urlEqualTo("/v1/charges"))).size() > 10);

        // Assert: replayed events generate new Stripe calls (duplicate charges)
        long stripeCallsAfter = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges"))).size();
        assertThat(stripeCallsAfter).isGreaterThan(10); // duplicates committed
    }

    @Test
    @DisplayName("Mode 3: Message Replay of 10 pre-outage events → stable key + guard → idempotent (with fix)")
    void messageReplay_withStableKey_noAdditionalCharges() throws Exception {
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200).withBody(chargeJson("ch_A"))));

        for (int i = 1; i <= 10; i++) {
            solaceTemplate.publish(BILLING_TOPIC,
                new BillingEvent("acct_" + i, "2024-03", 29900L, 1000));
        }
        await().atMost(Duration.ofSeconds(10))
               .until(() -> billingRecordRepo.count() == 10);

        Instant replayFrom = Instant.now().minusSeconds(60);
        solaceAdmin.startReplay(BILLING_QUEUE, replayFrom);

        // Allow time for replay to process
        Thread.sleep(5_000);

        // Assert: no additional Stripe calls; guard blocked all replayed events
        long stripeCallsAfter = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges"))).size();
        assertThat(stripeCallsAfter).isEqualTo(10); // exactly the original 10
        assertThat(billingRecordRepo.count()).isEqualTo(10); // no new DB rows
    }

    // ── Utility ────────────────────────────────────────────────────────────

    private static String chargeJson(String chargeId) {
        return """
            {"id":"%s","object":"charge","amount":29900,"currency":"usd",
             "status":"succeeded","created":%d}
            """.formatted(chargeId, Instant.now().getEpochSecond());
    }
}

The solaceAdmin.simulateJcsmpCommitFailureOnNextDelivery() helper uses the Solace SEMPv2 REST API (available in the Testcontainers Solace image on port 8080) to temporarily disable the broker-side commit acknowledgement, simulating a TCP RST on the JCSMP data plane connection. The solaceAdmin.startReplay() helper calls the SEMPv2 POST /msgVpns/{vpn}/queues/{queue}/startReplay endpoint to initiate a broker-level Message Replay from a specified timestamp.

All three test pairs are structured as mutation tests: the “without stable key” test verifies that the broken behavior actually manifests (confirming the test setup is realistic), and the “with fix” test verifies that the stable key + content guard eliminates the duplicate charge in that exact scenario. Running only the “with fix” tests would not tell you whether the fix is necessary — it would only tell you the fix doesn’t break normal operation.

The fix applied once, covering all three modes

All three failure modes share the same root cause — UUID.randomUUID() generates a different Stripe idempotency key on each consumer invocation — and the same fix: a stable content-hash key plus a content-based pre-flight guard. The fix is not SCS Solace Binder-specific. It works regardless of whether the source of the second invocation is a DMQ re-enqueue, a transacted session redelivery, a Message Replay, a pod restart, a Kubernetes rolling deploy, or any other re-delivery mechanism the broker or infrastructure introduces. The consumer function is idempotent with respect to all of them.

// Helper: content-hash key that is stable across all redelivery mechanisms
private static String billingIdempotencyKey(BillingEvent event) {
    return sha256Hex(
        event.getAccountId() + "|" +
        event.getBillingPeriod() + "|" +
        event.getAmountCents() + "|" +
        event.getUnitsConsumed()
    );
}

private static String sha256Hex(String input) {
    try {
        byte[] digest = MessageDigest.getInstance("SHA-256")
            .digest(input.getBytes(StandardCharsets.UTF_8));
        return HexFormat.of().formatHex(digest);
    } catch (NoSuchAlgorithmException e) {
        throw new IllegalStateException(e); // SHA-256 is always available
    }
}

The guard uses a composite unique constraint at the database level, not a soft check:

-- PostgreSQL DDL
ALTER TABLE billing_records
    ADD CONSTRAINT billing_records_account_billing_period_uq
    UNIQUE (account_id, billing_period);

-- The application-level check is the fast path; the DDL constraint is the safety net
-- for concurrent deliveries to two pods

With the DDL constraint in place and a stable key in the consumer, all three Solace-specific delivery mechanisms produce at most one Stripe charge per (accountId, billingPeriod) pair, regardless of how many times the billing event is delivered, replayed, or re-enqueued from the DMQ. The three failure modes covered in this post — the silent DMQ routing after retries exhaust, the JCSMP transacted session commit failure after Spring @Transactional, and the broker-level Message Replay that looks like a first-time delivery — are all made harmless by the same two-line key generation and one-line guard check.