Keybrake

Spring Cloud Stream Azure Service Bus Stripe Integration: How SCS ASB Binder AUTO Checkpoint Mode Retries the Consumer 3 Times Per Delivery Before Abandoning — With MaxDeliveryCount=10 That Is 30 UUID.randomUUID() Calls Generating ch_B Through ch_Z for One Billing Event, How MANUAL Mode Checkpointer.success() Called Before BillingRecord Guard Insert ACKs the Message Before the Guard Exists Enabling ch_B on DLQ Replay, and How Session-Aware Consumer Session Lock Expiry Mid-3DS Reassigns the Session to a Second Pod Generating UUID_B Committing ch_B

By Keybrake —

The Spring Cloud Stream Azure Service Bus Binder introduces three layers of SCS-specific abstraction over the raw Azure Service Bus SDK — application-level retry, checkpoint mode, and session lock management — and each layer has a failure mode that produces duplicate Stripe charges in ways that the raw SDK does not. The binder’s AUTO checkpoint mode silently multiplies the SCS application-level retry count by the Azure Service Bus MaxDeliveryCount, turning a three-retry billing consumer into one that can commit thirty Stripe charges for a single billing event. Its MANUAL checkpoint mode makes Checkpointer.success() appear to be a transactional operation when it is in fact an independent asynchronous Azure Service Bus API call that does not roll back if the Spring transaction does. And its session-aware consumer manages a session lock renewal thread whose failure during a slow Stripe 3DS call reassigns the session to a second pod with no error logged and no grace period.

This post covers all three failure modes end to end: the exact Spring Cloud Stream Azure Service Bus Binder configuration that enables each one, the step-by-step timeline showing when ch_A commits and when UUID_B is generated, the specific misconception each one exploits about the SCS ASB Binder’s behavior, and the idempotency fix that makes your billing consumer safe regardless of checkpoint mode or session configuration. Code samples use Spring Cloud Stream 4.x, spring-cloud-azure-stream-binder-servicebus 5.x (Spring Cloud Azure), Spring Boot 3.x, WireMock, Testcontainers, and JUnit 5.

Background: how the SCS Azure Service Bus Binder checkpoint model differs from the raw Azure Service Bus SDK and from other SCS binders

To understand why these three failure modes exist at the SCS layer rather than the raw SDK layer, you need to understand two structural differences the SCS ASB Binder introduces on top of the raw azure-messaging-servicebus SDK.

The SCS ASB Binder inserts its own application-level retry template between the raw SDK delivery and the consumer function. When using the raw azure-messaging-servicebus SDK directly, a ServiceBusProcessorClient invokes your Consumer<ServiceBusReceivedMessageContext> callback exactly once per delivery. Whether you retry the operation is your decision: you call context.abandon() explicitly, or you let the lock expire, or you call context.complete() if you decide the error is not retriable. The raw SDK has no built-in application-level retry loop. In contrast, the SCS ASB Binder wraps your consumer function with a Spring RetryTemplate that re-invokes the consumer function up to max-attempts times before propagating the exception to the binder’s acknowledgement layer. These retries happen within a single message delivery — the lock token does not change between retries, and context.complete() or context.abandon() is not called until all retries are exhausted. If UUID.randomUUID() is called inside the consumer function, each retry generates a different UUID and a different Stripe idempotency key, producing a new Stripe charge on every attempt.

The SCS ASB Binder’s acknowledgement modes map to Azure Service Bus message disposition methods, but the mapping is not self-evident from the checkpoint mode names. The SCS ASB Binder’s checkpoint-mode property controls when and how the binder calls the underlying Azure Service Bus disposition methods (complete(), abandon(), deadLetter()). In AUTO mode (the default), the binder calls complete() when the consumer function returns normally after all retries succeed, and calls abandon() when all retries are exhausted without success. In MANUAL mode, the binder injects a Checkpointer object into the consumer function via a message header, and the consumer is responsible for calling Checkpointer.success() (backed by context.complete()) or Checkpointer.failure() (backed by context.abandon()). The MANUAL mode name echoes the SCS Kafka Binder’s MANUAL_IMMEDIATE ack mode, which does participate in the Kafka consumer’s offset commit protocol in a controllable way, but the SCS ASB Binder’s Checkpointer does not participate in the Spring transaction lifecycle.

With these two structural differences as context, the three failure modes follow directly from specific configurations.

Failure mode 1: AUTO checkpoint mode — SCS max-attempts=3 retries within each delivery, then abandon(); ASB redelivers up to MaxDeliveryCount=10 times; 3 × 10 = 30 Stripe calls for one billing event

The setup

A team runs a usage-based billing service for a developer platform. The message broker is Azure Service Bus. The Spring Boot 3.x billing service uses the Spring Cloud Stream Azure Service Bus Binder to consume metered usage events from the billing queue and convert them to Stripe charges. The team uses the SCS ASB Binder default checkpoint-mode=AUTO and the SCS default retry settings:

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing           # Azure Service Bus queue name
          group: billing-consumer-group
          consumer:
            max-attempts: 3             # SCS application-level retries (default)
            back-off-initial-interval: 1000
            back-off-multiplier: 2.0
      servicebus:
        bindings:
          billing-in-0:
            consumer:
              checkpoint-mode: auto     # complete() on success, abandon() on exhaustion (default)
    azure:
      servicebus:
        connection-string: "${AZURE_SERVICEBUS_CONNECTION_STRING}"

The billing queue was created with Azure Service Bus’ default MaxDeliveryCount=10. This means Azure Service Bus will redeliver a message up to 10 times before moving it to the Dead-Letter Queue (DLQ). The team did not review this setting when configuring the queue, because they expected the SCS max-attempts=3 to be the effective retry limit.

The consumer is written as a standard Spring functional bean:

@Component
public class BillingConsumer implements Consumer<BillingEvent> {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    @Override
    public void accept(BillingEvent event) {

        // BUG: fresh UUID per invocation — different value on every retry and redelivery
        String idempotencyKey = UUID.randomUUID().toString();

        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getStripeCustomerId())
            .setDescription("Usage " + event.getAccountId()
                + " period " + event.getBillingPeriod()
                + " units " + event.getUnitsConsumed())
            .build();

        Charge charge = stripe.charges().create(params,
            RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build());

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            charge.getId(),
            idempotencyKey
        ));
    }
}

The failure timeline

During an end-of-month billing run, the Azure Service Bus queue delivers a BillingEvent for account acct_007, billing period 2026-10, USD 149 in usage charges. The billing_records table has a unique constraint on (account_id, billing_period). A partial billing run from a failed job three weeks ago inserted a record for acct_007 in period 2026-10, but that run crashed before the Stripe call was made — the record has stripe_charge_id = NULL. The DataIntegrityViolationException will fire from the unique constraint regardless.

Outcome: 30 Stripe charges for account acct_007, billing period 2026-10, USD 149 each — USD 4,470 charged against USD 149 owed. No BillingRecord exists for any of the 30 charges (the DB write failed every time). The message sits in the DLQ with DeadLetterReason=MaxDeliveryCountExceeded.

Because there is no BillingRecord, an on-call engineer looking at the DLQ message sees DataIntegrityViolationException in the logs and a message in the DLQ with no associated charge in the billing records table. The engineer concludes that Stripe was never charged and re-enqueues the message from the DLQ. This generates attempt 31 — UUID.randomUUID() → ch_X — which again commits at Stripe but fails at the DB.

The misconception

“SCS max-attempts=3 and ASB MaxDeliveryCount=10 are two views of the same retry counter. max-attempts=3 controls how many times the SCS retry framework re-invokes the consumer before giving up; MaxDeliveryCount=10 is the Azure Service Bus-level equivalent of the same limit. If max-attempts=3 is configured in SCS, the MaxDeliveryCount=10 setting only applies if the SCS binder itself crashes or the JVM is killed mid-retry — under normal error handling, the message is dead-lettered after 3 SCS retries, not redelivered 10 times at the ASB layer.”

This is wrong. max-attempts and MaxDeliveryCount operate at different layers and they multiply, not override. max-attempts controls how many times the SCS RetryTemplate re-invokes the consumer function within a single ASB message delivery — no lock token change, no Azure Service Bus redelivery between SCS attempts. MaxDeliveryCount controls how many times Azure Service Bus will deliver the message before dead-lettering it. When the SCS binder exhausts all max-attempts retries and calls abandon(), Azure Service Bus increments its own delivery counter and — if DeliveryCount < MaxDeliveryCount — redelivers the message as if it were arriving for the first time. The SCS RetryTemplate then starts its retry loop over again from attempt 1.

This differs from the SCS Kafka Binder. In the Kafka Binder, when max-attempts is exhausted and a dead-letter topic (DLT) is configured, the binder publishes to the DLT and commits the consumer offset — Kafka does not have an independent redelivery mechanism equivalent to ASB’s MaxDeliveryCount. Without a DLT, the Kafka Binder’s default behavior is to seek back to the failed record and re-deliver it indefinitely, but this is a single continuous retry loop controlled entirely by the SCS Kafka Binder, not a layered system where a message-broker-level counter multiplies with an application-level counter.

For the SCS ASB Binder in AUTO mode, the effective retry count per billing event is:

Setting Layer Default value What it controls
max-attempts SCS application 3 Re-invocations of consumer function within one delivery, without releasing the lock
MaxDeliveryCount Azure Service Bus 10 Number of times ASB delivers the message before dead-lettering it
Total Stripe calls (worst case) Both layers multiplied 3 × 10 = 30 One UUID.randomUUID() per call = 30 distinct Stripe idempotency keys = 30 charges

Diagnosing this failure in production

The AUTO mode retry-multiplication pattern has a specific Stripe dashboard signature: charges for the same account and billing period arrive in bursts. Each burst of 3 corresponds to one ASB delivery (3 SCS retries), and each burst is spaced approximately 3 seconds apart (1s + 2s back-off with multiplier=2.0). With MaxDeliveryCount=10, there are 10 bursts over roughly 30 seconds, producing 30 charges.

The fastest SQL to identify accounts affected by this pattern:

-- Accounts with more than 3 charges for the same billing period
-- (3 = max-attempts, so any count > 3 indicates ASB redelivery cycles)
SELECT
    stripe_customer_id,
    billing_period,
    COUNT(*) AS charge_count,
    MIN(created_at) AS first_charge_at,
    MAX(created_at) AS last_charge_at,
    EXTRACT(EPOCH FROM (MAX(created_at) - MIN(created_at))) AS window_seconds
FROM stripe_charges_audit    -- external audit table populated by Stripe webhook
WHERE billing_period = '2026-10'
GROUP BY stripe_customer_id, billing_period
HAVING COUNT(*) > 3
ORDER BY charge_count DESC;

Note that this query requires an external audit table populated from Stripe webhooks, because the billing_records table will be empty for the affected accounts — every DB write failed. The absence of BillingRecord rows is itself a diagnostic signal: if Stripe shows charges but the billing records table has no rows for those accounts, the likely cause is that the guard’s DB write is failing after every Stripe commit, preventing the guard from being installed.

The fix

Two changes make the billing consumer safe under AUTO checkpoint mode regardless of how many SCS retries or ASB redeliveries occur.

First: derive the idempotency key from stable payload fields, not from UUID.randomUUID(). The SHA-256(accountId + billingPeriod + amountCents + unitsConsumed) value is the same on every SCS retry, every ASB redelivery, and every future manual replay. Stripe’s idempotency cache returns ch_A on every subsequent call with the same key:

@Component
public class BillingConsumer implements Consumer<BillingEvent> {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    @Override
    public void accept(BillingEvent event) {

        // Guard: check before calling Stripe
        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            return;  // already charged — all SCS retries and ASB redeliveries skip Stripe
        }

        // Stable idempotency key — same value across all retries and redeliveries
        String idempotencyKey = Hashing.sha256()
            .hashString(
                event.getAccountId()
                    + "|" + event.getBillingPeriod()
                    + "|" + event.getAmountCents()
                    + "|" + event.getUnitsConsumed(),
                StandardCharsets.UTF_8
            )
            .toString();

        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getStripeCustomerId())
            .setDescription("Usage " + event.getAccountId()
                + " period " + event.getBillingPeriod()
                + " units " + event.getUnitsConsumed())
            .build();

        Charge charge = stripe.charges().create(params,
            RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build());

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            charge.getId(),
            idempotencyKey
        ));
    }
}

Second: set MaxDeliveryCount=1 on the billing queue if SCS’s max-attempts is already handling retries. With a stable idempotency key, ASB redeliveries are safe — Stripe returns the cached result. But there is no reason to let ASB redeliver 10 times if the SCS retry framework already handles the application-level retry loop. Setting MaxDeliveryCount=1 ensures that after SCS exhausts its retries and calls abandon(), ASB immediately dead-letters the message rather than redelivering it. This makes the retry count explicit and auditable in one place:

# Azure CLI: set MaxDeliveryCount=1 to avoid silent multiplication
az servicebus queue update \
  --resource-group billing-rg \
  --namespace-name billing-ns \
  --name billing \
  --max-delivery-count 1

Test: AUTO mode retry multiplication does not commit duplicate Stripe charges

@SpringBootTest
@Testcontainers
class AutoModeRetryMultiplicationTest {

    @Container
    static PostgreSQLContainer<?> postgres =
        new PostgreSQLContainer<>("postgres:16")
            .withDatabaseName("billing_test")
            .withUsername("billing")
            .withPassword("billing");

    @Autowired
    private BillingConsumer billingConsumer;

    @Autowired
    private BillingRecordRepository billingRecordRepo;

    private WireMockServer wireMock;

    @BeforeEach
    void setUp() {
        wireMock = new WireMockServer(WireMockConfiguration.wireMockConfig().port(9099));
        wireMock.start();
        wireMock.stubFor(WireMock.post(WireMock.urlEqualTo("/v1/charges"))
            .willReturn(WireMock.aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_A_auto\",\"object\":\"charge\","
                        + "\"amount\":14900,\"status\":\"succeeded\"}")));
    }

    @Test
    void autoModeRedelivery_alreadyChargedAccountSkippedOnRedelivery() {
        BillingEvent event = BillingEvent.builder()
            .accountId("acct_auto_001")
            .billingPeriod("2026-10")
            .amountCents(14900L)
            .unitsConsumed(75L)
            .stripeCustomerId("cus_auto_001")
            .build();

        // First delivery: Stripe called once, BillingRecord saved
        billingConsumer.accept(event);
        assertThat(wireMock.getAllServeEvents()).hasSize(1);
        assertThat(billingRecordRepo.existsByAccountIdAndBillingPeriod(
            "acct_auto_001", "2026-10")).isTrue();

        // Simulate ASB redelivery (DeliveryCount=2..10): guard skips Stripe each time
        for (int delivery = 2; delivery <= 10; delivery++) {
            billingConsumer.accept(event);
        }

        // Stripe still called exactly once — guard blocked all 9 redeliveries
        assertThat(wireMock.getAllServeEvents()).hasSize(1);

        // One BillingRecord — no duplicates
        assertThat(billingRecordRepo.countByAccountIdAndBillingPeriod(
            "acct_auto_001", "2026-10")).isEqualTo(1);
    }

    @AfterEach
    void tearDown() { wireMock.stop(); }
}

Failure mode 2: MANUAL checkpoint mode — Checkpointer.success() called after Stripe but before BillingRecord guard insert; DB write fails; message ACKed before guard exists; DLQ replay generates UUID_B committing ch_B

The setup

A team migrates the billing consumer from checkpoint-mode=AUTO to MANUAL after a performance review recommends reducing message lock hold time. The reasoning: in AUTO mode, the lock is held until the consumer function returns, which includes both the Stripe call (up to 500ms) and the database write. By calling Checkpointer.success() immediately after the Stripe call completes — before the DB write — the team believes they can release the ASB lock faster, increasing throughput for other consumers waiting on the same queue. The configuration:

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing
          group: billing-consumer-group
          consumer:
            max-attempts: 1             # Disable SCS retries; handle retry at application level
      servicebus:
        bindings:
          billing-in-0:
            consumer:
              checkpoint-mode: manual   # Consumer must call Checkpointer explicitly
    azure:
      servicebus:
        connection-string: "${AZURE_SERVICEBUS_CONNECTION_STRING}"

The consumer is rewritten to inject the Checkpointer via the SCS ASB Binder’s message header:

@Component
public class BillingConsumer {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    // Spring Cloud Stream functional model with headers injection
    @Bean
    public Consumer<Message<BillingEvent>> billing() {
        return message -> {
            BillingEvent event = message.getPayload();
            Checkpointer checkpointer = (Checkpointer) message.getHeaders()
                .get(ServiceBusMessageHeaders.CHECKPOINTER);

            // Guard check (pre-flight)
            if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                    event.getAccountId(), event.getBillingPeriod())) {
                checkpointer.success().block();
                return;
            }

            // Stable idempotency key
            String idempotencyKey = Hashing.sha256()
                .hashString(
                    event.getAccountId() + "|" + event.getBillingPeriod()
                        + "|" + event.getAmountCents() + "|" + event.getUnitsConsumed(),
                    StandardCharsets.UTF_8)
                .toString();

            ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build();

            Charge charge = stripe.charges().create(params,
                RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());

            // BUG: ACK the message before saving the BillingRecord guard
            checkpointer.success().block();   // <-- called before billingRecordRepo.save()

            billingRecordRepo.save(new BillingRecord(
                event.getAccountId(),
                event.getBillingPeriod(),
                charge.getId(),
                idempotencyKey
            ));
            // If save() throws here, the message is already ACKed — no retry, no redelivery
        };
    }
}

The failure timeline

During an end-of-month billing run, the billing queue delivers a BillingEvent for account acct_012, billing period 2026-10, USD 249. The pre-flight guard passes: no BillingRecord exists for this account and period. The consumer calls Stripe and commits ch_A (USD 249) at t=480ms. The consumer calls checkpointer.success().block() at t=490ms, which dispatches context.complete() to Azure Service Bus. At t=495ms, Azure Service Bus confirms the complete: the message is removed from the queue and will not be redelivered.

At t=500ms, the consumer calls billingRecordRepo.save(). The PostgreSQL connection pool is exhausted during a peak billing run (24 other consumer threads have acquired connections and are awaiting Stripe responses). The save() call waits for a connection. After a 30-second timeout, it throws CannotGetJdbcConnectionException: Unable to acquire JDBC Connection. The exception propagates out of the lambda, but the message has already been ACKed. ASB does not redeliver it. The SCS ASB Binder’s max-attempts=1 configuration means no SCS-level retry is attempted. There is no error channel configured. The exception is logged and the consumer moves to the next message.

The state of the system at t=530s:

Three days later, the on-call billing reconciliation job runs a nightly check: it queries SELECT account_id FROM billing_invoices WHERE billing_period = '2026-10' AND charged_at IS NULL. acct_012 appears because the billing invoice was never marked as charged (that happens in the same DB transaction as the BillingRecord save). The reconciliation job re-enqueues a billing message for acct_012 to the billing queue.

The billing consumer receives the re-enqueued message. Pre-flight guard: existsByAccountIdAndBillingPeriod("acct_012", "2026-10") returns false (no record exists). Consumer computes the same stable idempotency key. But it is now 3 days later: Stripe’s 24-hour idempotency cache for the key has expired. Stripe creates a new charge: ch_B, USD 249. checkpointer.success().block() succeeds. billingRecordRepo.save() succeeds this time (connection pool has capacity). BillingRecord saved. acct_012 is now charged USD 498 for billing period 2026-10.

The misconception

“Checkpointer.success() in SCS ASB Binder MANUAL mode participates in the Spring @Transactional boundary. If the billing() function is annotated with @Transactional, or if billingRecordRepo.save() is wrapped in a @Transactional service method, and that transaction rolls back due to a connection timeout, the Checkpointer.success() call will also be rolled back — the message will not be ACKed, and Azure Service Bus will redeliver it. Calling Checkpointer.success() before billingRecordRepo.save() is therefore safe: the ACK and the DB write succeed or fail together as part of the same distributed transaction.”

This is wrong for two reasons. First, Checkpointer.success() in the SCS ASB Binder calls ServiceBusReceivedMessageContext.complete() on the underlying azure-messaging-servicebus ServiceBusReceivedMessage. This is an independent Azure Service Bus API call that goes over the network to the Azure Service Bus management plane. It is not a local in-memory operation that can be rolled back. Once complete() has dispatched and received acknowledgement from Azure Service Bus, the message is gone from the queue, regardless of what happens to the Spring transaction.

Second, even if the consumer is annotated with @Transactional, the Spring transaction manager has no knowledge of the Checkpointer.success() call. Checkpointer is not a Spring TransactionSynchronization. It does not register itself with the current transaction to receive rollback callbacks. It is a thin wrapper around ServiceBusReceivedMessageContext.complete(), which is an Azure SDK reactive call (Mono<Void>). Calling .block() on it executes it eagerly and immediately, disconnected from any Spring transaction lifecycle event.

This is a meaningful difference from the SCS Kafka Binder’s MANUAL_IMMEDIATE ack mode. In the Kafka Binder, Acknowledgment.acknowledge() queues an offset commit to be processed by the Kafka consumer’s commit callback — while it is not transactional in the two-phase-commit sense, its behavior is closer to an in-process queue rather than an out-of-process network call. In the SCS ASB Binder, Checkpointer.success() is a network call to an external service that produces a durable side effect immediately upon execution. It cannot be undone.

Binder Checkpoint/Ack call Transactional rollback? What happens if DB throws after ack?
SCS Kafka (MANUAL_IMMEDIATE) Acknowledgment.acknowledge() No — commits Kafka offset in-process Message not redelivered; offset committed; DB write lost
SCS RabbitMQ (MANUAL) Channel.basicAck() No — sends AMQP frame to broker Message not redelivered; ACK sent to broker; DB write lost
SCS ASB (MANUAL) Checkpointer.success() → context.complete() No — Azure Service Bus API call Message not redelivered; ASB complete confirmed; DB write lost
SCS AEH (MANUAL) Checkpointer.success() → Blob Storage write No — Blob Storage write independent of Spring TX Checkpoint committed; pod restart replays from committed position; DB write lost

No SCS binder checkpoint or ack call participates in the Spring transaction. The safe ordering is always: DB guard write first, then checkpoint/ack call, never the reverse.

The fix

Move Checkpointer.success() to after the billingRecordRepo.save() call. The stable idempotency key ensures that if the message is redelivered (due to lock expiry before complete()) and the DB write actually succeeded, the pre-flight guard will catch it on the next invocation:

@Bean
public Consumer<Message<BillingEvent>> billing() {
    return message -> {
        BillingEvent event = message.getPayload();
        Checkpointer checkpointer = (Checkpointer) message.getHeaders()
            .get(ServiceBusMessageHeaders.CHECKPOINTER);

        // Pre-flight guard: skip if already charged
        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            checkpointer.success().block();
            return;
        }

        String idempotencyKey = Hashing.sha256()
            .hashString(
                event.getAccountId() + "|" + event.getBillingPeriod()
                    + "|" + event.getAmountCents() + "|" + event.getUnitsConsumed(),
                StandardCharsets.UTF_8)
            .toString();

        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getStripeCustomerId())
            .build();

        Charge charge = stripe.charges().create(params,
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());

        // Save the guard FIRST — then ACK the message
        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            charge.getId(),
            idempotencyKey
        ));

        // Only ACK after the BillingRecord guard is durably written
        checkpointer.success().block();
    };
}

With this ordering, if checkpointer.success() fails or times out (e.g., ASB lock expired between the DB write and the complete() call), the message will be redelivered. The pre-flight guard now finds the existing BillingRecord, skips the Stripe call, and calls checkpointer.success() again successfully. This is the correct ordering regardless of Stripe’s 24-hour cache expiry: the guard is what protects against redeliveries after the idempotency window closes.

Test: MANUAL mode guard-before-ack ordering does not produce duplicate charges on reconnect replay

@SpringBootTest
@Testcontainers
class ManualCheckpointGuardOrderTest {

    @Container
    static PostgreSQLContainer<?> postgres =
        new PostgreSQLContainer<>("postgres:16")
            .withDatabaseName("billing_test")
            .withUsername("billing")
            .withPassword("billing");

    @Autowired
    private ApplicationContext context;

    @Autowired
    private BillingRecordRepository billingRecordRepo;

    private WireMockServer wireMock;

    @BeforeEach
    void setUp() {
        wireMock = new WireMockServer(WireMockConfiguration.wireMockConfig().port(9099));
        wireMock.start();
        wireMock.stubFor(WireMock.post(WireMock.urlEqualTo("/v1/charges"))
            .willReturn(WireMock.aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_A_manual\",\"object\":\"charge\","
                        + "\"amount\":24900,\"status\":\"succeeded\"}")));
    }

    @Test
    void manualMode_guardInsertedBeforeAck_redeliverySkipsStripe() throws Exception {
        BillingEvent event = BillingEvent.builder()
            .accountId("acct_manual_001")
            .billingPeriod("2026-10")
            .amountCents(24900L)
            .unitsConsumed(100L)
            .stripeCustomerId("cus_manual_001")
            .build();

        Consumer<Message<BillingEvent>> billingConsumer =
            context.getBean("billing", Consumer.class);

        // Simulate first delivery with a Checkpointer that records success() calls
        AtomicInteger checkpointSuccessCalls = new AtomicInteger(0);
        Checkpointer checkpointer = () -> {
            checkpointSuccessCalls.incrementAndGet();
            return Mono.empty();
        };

        Message<BillingEvent> firstDelivery = MessageBuilder.withPayload(event)
            .setHeader(ServiceBusMessageHeaders.CHECKPOINTER, checkpointer)
            .build();

        billingConsumer.accept(firstDelivery);

        // Stripe called once, BillingRecord saved, checkpoint called once
        assertThat(wireMock.getAllServeEvents()).hasSize(1);
        assertThat(billingRecordRepo.existsByAccountIdAndBillingPeriod(
            "acct_manual_001", "2026-10")).isTrue();
        assertThat(checkpointSuccessCalls.get()).isEqualTo(1);

        // Simulate ASB redelivery (lock expired before complete() reached ASB)
        Message<BillingEvent> redelivery = MessageBuilder.withPayload(event)
            .setHeader(ServiceBusMessageHeaders.CHECKPOINTER, checkpointer)
            .build();

        billingConsumer.accept(redelivery);

        // Stripe still called only once — guard blocked the redelivery
        assertThat(wireMock.getAllServeEvents()).hasSize(1);
        // Checkpoint called twice (once per delivery), but Stripe only once
        assertThat(checkpointSuccessCalls.get()).isEqualTo(2);
        // Only one BillingRecord
        assertThat(billingRecordRepo.countByAccountIdAndBillingPeriod(
            "acct_manual_001", "2026-10")).isEqualTo(1);
    }

    @AfterEach
    void tearDown() { wireMock.stop(); }
}

Failure mode 3: session-aware consumer — session lock expiry mid-3DS call reassigns the session to Pod 2; Pod 2 re-receives the in-flight billing message; generates UUID_B committing ch_B

The setup

A team adopts Azure Service Bus message sessions on the billing queue to guarantee per-account ordered processing. Each billing event carries a SessionId equal to the accountId, ensuring that all billing messages for a given account are processed sequentially by one consumer at a time. The SCS ASB Binder is configured to use the session-aware ServiceBusSessionProcessorClient:

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing
          group: billing-consumer-group
          consumer:
            max-attempts: 3
      servicebus:
        bindings:
          billing-in-0:
            consumer:
              checkpoint-mode: auto
              session-aware: true       # enables ServiceBusSessionProcessorClient
    azure:
      servicebus:
        connection-string: "${AZURE_SERVICEBUS_CONNECTION_STRING}"
        processor:
          session:
            max-concurrent-sessions: 10   # up to 10 sessions processed in parallel
            max-concurrent-calls-per-session: 1

The billing queue is a session-enabled Azure Service Bus queue with the default session lock duration of 30 seconds (configurable up to 5 minutes; the default is significantly shorter than message lock duration defaults in other systems). When the SCS ASB Binder acquires a session, it receives both a session lock (held by the pod’s ServiceBusSessionReceiver) and a message lock (held per received message). The session lock and the message lock have separate renewal mechanisms. The SCS ASB Binder manages session lock renewal via a background thread that calls sessionReceiver.renewSessionLock() every N seconds, where N is calculated from the session lock duration. The message lock is renewed separately.

The billing consumer is straightforward:

@Component
public class BillingConsumer implements Consumer<BillingEvent> {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    @Override
    public void accept(BillingEvent event) {

        // BUG: UUID.randomUUID() generates a different value on session reassignment
        String idempotencyKey = UUID.randomUUID().toString();

        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getStripeCustomerId())
            .setDescription("Session billing: " + event.getAccountId()
                + " period " + event.getBillingPeriod())
            .build();

        // This Stripe 3DS call can take 35 seconds for international card issuers
        Charge charge = stripe.charges().create(params,
            RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build());

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            charge.getId(),
            idempotencyKey
        ));
    }
}

The failure timeline

Pod 1 in the billing consumer group acquires session sess_acct_017 and receives message M1: a billing event for acct_017, billing period 2026-10, EUR 299, an international card requiring Stripe 3DS authentication.

Pod 1’s billingRecordRepo.save() may or may not succeed depending on whether it was called before Pod 2’s save. If Pod 1 saves first, Pod 2 encounters a unique constraint violation and its DataIntegrityViolationException causes the SCS binder to call abandon() on Pod 2’s lock token. Azure Service Bus may then redeliver M1 to Pod 3, continuing the cascade. If Pod 2 saves first, Pod 1’s save succeeds but its context.complete() fails, leaving Pod 1’s BillingRecord (with ch_A) alongside Pod 2’s BillingRecord (with ch_B) if the unique constraint allows it. Under a strict unique constraint, exactly one save succeeds, but the Stripe charges for both ch_A and ch_B are already committed and cannot be rolled back.

The misconception

“Session lock expiry in the SCS ASB Binder session-aware consumer triggers the binder’s error handler before the session is reassigned to another pod. There is a grace period between when the session lock expires and when another pod can acquire the session — similar to how Kubernetes node eviction has a grace period before the workload is rescheduled. During this grace period, the binder detects the expiry, logs an exception, and the current consumer function can still call context.complete() to finalize the message before the session moves. The binder manages this internally; developers do not need to handle session lock expiry explicitly.”

This is wrong in three ways. First, there is no grace period between Azure Service Bus session lock expiry and session reassignment. The moment the session lock expires, the session becomes immediately available for acquisition by any competing ServiceBusSessionProcessorClient. Azure Service Bus does not wait for the current consumer to finish processing. Second, the SCS ASB Binder’s session lock renewal thread failure does not trigger the consumer function’s error handler. The renewal thread failure is logged at WARN level, but it does not call the consumer’s error channel, does not throw an exception into the consumer thread, and does not interrupt the Stripe call in progress. The consumer function has no way to know that the session lock renewal failed unless it explicitly checks for ServiceBusException on a manual renewSessionLock() call. Third, once another pod acquires the session, that pod can receive and complete any unacknowledged messages in the session, including the message the first pod is still actively processing. Azure Service Bus does not track that a message is “in-flight” for processing purposes when the session lock has transferred — only the lock token matters, and a new pod in the same consumer group can acquire a fresh lock token for an unacknowledged message after a session reassignment.

This differs from the SCS Azure Event Hubs Binder’s partition lease model. In the AEH Binder, when a partition lease expires, the new pod starts reading from the last committed checkpoint offset — it does not re-receive the specific in-flight message; it starts from a position in the event stream. In the ASB Binder session model, the new pod does receive the specific in-flight messages that were not acknowledged, because ASB tracks unacknowledged messages individually by lock token, not by stream position.

Binder Concurrent processing during reassignment How reassignment starts Duplicate charge mechanism
SCS AEH Binder Second pod starts from checkpoint offset — may overlap if checkpoint is behind Blob Storage lease expiry Second pod reprocesses events from checkpoint position (possibly already charged)
SCS ASB Binder (non-session) No rebalance; message redelivered only after lock expiry or abandon() Message lock expiry (60s default) Second delivery generates new UUID after first delivery committed ch_A
SCS ASB Binder (session-aware) Yes — second pod acquires session immediately; can receive M1 while Pod 1 is mid-Stripe Session lock expiry (30s default) Both pods generate UUID_A and UUID_B; both Stripe calls can commit if 3DS completes for both
SCS Kafka Binder No — consumer group protocol prevents two consumers owning the same partition Consumer group rebalance (heartbeat miss) New consumer replays from committed offset if checkpoint was behind

Diagnosing this failure in production

The session lock expiry pattern produces a distinctive charge pair in Stripe: two charges for the same customer on the same day for the same amount, with charge IDs created within seconds of each other. Unlike the AUTO mode retry-multiplication pattern (which produces charges spaced 1–3 seconds apart), the session lock expiry pattern produces two charges with a gap of approximately 30 seconds (the session lock duration) between them.

In the Azure Service Bus metrics, look for spikes in the ServerErrors metric on the billing queue namespace that correlate with 429 responses from the session lock renewal endpoint. The renewal failures that precipitate session lock expiry are logged as WARN entries in the SCS ASB Binder log containing Failed to renew session lock. Monitor these warnings and alert if more than 3 occur within a 60-second window on any billing pod.

The fix

The stable idempotency key from failure modes 1 and 2 applies here. When Pod 2 acquires the session and calls Stripe with UUID_B (different from Pod 1’s UUID_A), Stripe sees a new idempotency key and creates a new charge — the stable key does not help by itself if Pod 2 also generates a new UUID. The correct fix requires two changes.

First: use a stable idempotency key so that both pods generate the same Stripe key. If Pod 2 computes the same SHA-256(accountId + billingPeriod + amountCents + unitsConsumed) key as Pod 1, then when both pods call Stripe concurrently, Stripe’s idempotency layer serializes the two requests: the first one to arrive creates ch_A and the second one returns ch_A from cache (assuming both requests arrive within Stripe’s 30-minute concurrent idempotency window). If the two pods call Stripe sequentially (Pod 1 finishes before Pod 2 is even acquired), the BillingRecord guard on Pod 2 will find Pod 1’s record and skip the Stripe call entirely:

@Component
public class BillingConsumer implements Consumer<BillingEvent> {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    @Override
    public void accept(BillingEvent event) {

        // Guard: skip if already charged (handles sequential session reassignment)
        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            return;
        }

        // Stable key: same value on Pod 1 and Pod 2 for the same billing event
        // Stripe idempotency layer serializes concurrent calls with the same key
        String idempotencyKey = Hashing.sha256()
            .hashString(
                event.getAccountId() + "|" + event.getBillingPeriod()
                    + "|" + event.getAmountCents() + "|" + event.getUnitsConsumed(),
                StandardCharsets.UTF_8)
            .toString();

        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getStripeCustomerId())
            .setDescription("Session billing: " + event.getAccountId()
                + " period " + event.getBillingPeriod())
            .build();

        Charge charge = stripe.charges().create(params,
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            charge.getId(),
            idempotencyKey
        ));
    }
}

Second: increase the session lock duration to exceed the Stripe p99 call duration plus a safety margin. The default 30-second session lock duration is shorter than Stripe’s 3DS p99 for international cards (which can reach 35–45 seconds). Set the session lock duration to at least 120 seconds on billing queues where 3DS is used, and monitor session lock renewal success rate:

# Azure CLI: set session lock duration to 120 seconds for billing queue
az servicebus queue update \
  --resource-group billing-rg \
  --namespace-name billing-ns \
  --name billing \
  --lock-duration PT2M   # ISO 8601: 2 minutes

# Also configure the SCS ASB Binder to renew more aggressively
# (renewal interval = lock duration / 2 by default; explicit override below)
spring:
  cloud:
    azure:
      servicebus:
        processor:
          session:
            session-idle-timeout: PT30S    # release idle sessions faster
            # Note: lock renewal is managed by the azure-messaging-servicebus SDK
            # internally; set lock-duration on the queue, not in the binder config

Test: session reassignment with stable key does not commit duplicate Stripe charges

@SpringBootTest
@Testcontainers
class SessionLockExpiryTest {

    @Container
    static PostgreSQLContainer<?> postgres =
        new PostgreSQLContainer<>("postgres:16")
            .withDatabaseName("billing_test")
            .withUsername("billing")
            .withPassword("billing");

    @Autowired
    private BillingConsumer billingConsumer;

    @Autowired
    private BillingRecordRepository billingRecordRepo;

    private WireMockServer wireMock;

    @BeforeEach
    void setUp() {
        wireMock = new WireMockServer(WireMockConfiguration.wireMockConfig().port(9099));
        wireMock.start();
        wireMock.stubFor(WireMock.post(WireMock.urlEqualTo("/v1/charges"))
            .willReturn(WireMock.aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_A_session\",\"object\":\"charge\","
                        + "\"amount\":29900,\"status\":\"succeeded\"}")));
    }

    @Test
    void sessionReassignment_concurrentCallsWithSameStableKey_oneChargeCommitted() {
        // Simulate both Pod 1 and Pod 2 receiving M1 concurrently
        BillingEvent event = BillingEvent.builder()
            .accountId("acct_session_017")
            .billingPeriod("2026-10")
            .amountCents(29900L)
            .unitsConsumed(150L)
            .stripeCustomerId("cus_session_017")
            .build();

        // Both pods call the consumer concurrently (simulated sequentially here)
        // Pod 1: processes first, saves BillingRecord
        billingConsumer.accept(event);
        assertThat(wireMock.getAllServeEvents()).hasSize(1);
        assertThat(billingRecordRepo.existsByAccountIdAndBillingPeriod(
            "acct_session_017", "2026-10")).isTrue();

        // Pod 2: guard blocks — BillingRecord already exists, Stripe not called
        billingConsumer.accept(event);
        assertThat(wireMock.getAllServeEvents()).hasSize(1);  // still 1 Stripe call
        assertThat(billingRecordRepo.countByAccountIdAndBillingPeriod(
            "acct_session_017", "2026-10")).isEqualTo(1);    // still 1 record
    }

    @Test
    void sessionReassignment_concurrentCallsWithSameStableKey_stripeIdempotencyDeduplicates()
            throws Exception {
        // Simulate both pods calling Stripe simultaneously (before either saves BillingRecord)
        // This tests Stripe's concurrent idempotency deduplication
        BillingEvent event = BillingEvent.builder()
            .accountId("acct_session_018")
            .billingPeriod("2026-10")
            .amountCents(29900L)
            .unitsConsumed(150L)
            .stripeCustomerId("cus_session_018")
            .build();

        // Compute the stable key both pods would generate
        String stableKey = Hashing.sha256()
            .hashString("acct_session_018|2026-10|29900|150", StandardCharsets.UTF_8)
            .toString();

        // Verify both invocations would use the same key (same stable computation)
        CountDownLatch startLatch = new CountDownLatch(1);
        List<String> capturedKeys = Collections.synchronizedList(new ArrayList<>());

        // Concurrent execution: both threads start simultaneously
        ExecutorService executor = Executors.newFixedThreadPool(2);
        for (int pod = 0; pod < 2; pod++) {
            executor.submit(() -> {
                try {
                    startLatch.await();
                    billingConsumer.accept(event);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            });
        }
        startLatch.countDown();
        executor.shutdown();
        executor.awaitTermination(10, TimeUnit.SECONDS);

        // Stripe called once or twice (Stripe deduplicates concurrent calls with same key)
        // Either way: only one charge committed, one BillingRecord in DB
        assertThat(wireMock.getAllServeEvents().size()).isLessThanOrEqualTo(2);
        assertThat(billingRecordRepo.countByAccountIdAndBillingPeriod(
            "acct_session_018", "2026-10")).isEqualTo(1);
    }

    @AfterEach
    void tearDown() { wireMock.stop(); }
}

Eight-item diagnostic checklist for SCS Azure Service Bus Binder Stripe billing consumers

Run this checklist against any billing consumer using the Spring Cloud Stream Azure Service Bus Binder before deploying to production.

  1. Idempotency key source. Every call to stripe.charges().create(), stripe.paymentIntents().create(), or any Stripe write API in the consumer must use an idempotency key derived from stable payload fields — not from UUID.randomUUID(). In the SCS ASB Binder, a single billing event can generate up to max-attempts × MaxDeliveryCount consumer function invocations. Each invocation calling UUID.randomUUID() produces a distinct idempotency key and a distinct Stripe charge.
  2. AUTO mode + MaxDeliveryCount multiplication. If using checkpoint-mode=AUTO, calculate the effective maximum Stripe call count as max-attempts × MaxDeliveryCount. With the SCS default max-attempts=3 and ASB default MaxDeliveryCount=10, this is 30 Stripe calls per billing event. Either set max-attempts=1 and rely on ASB redeliveries (stable key + guard handles safety), or set MaxDeliveryCount=1 to prevent ASB redeliveries after SCS retry exhaustion. Document the chosen configuration explicitly alongside the queue definition.
  3. MANUAL mode Checkpointer ordering. In checkpoint-mode=MANUAL, verify that Checkpointer.success() is called AFTER billingRecordRepo.save(), not before. The SCS ASB Binder Checkpointer.success() call is an independent Azure Service Bus network call. It does not participate in the Spring transaction lifecycle. A DB write failure after Checkpointer.success() leaves the message ACKed with no BillingRecord guard — future replays will bypass the guard and commit a duplicate charge.
  4. Checkpointer transaction participation. Confirm the team understands that no SCS ASB Binder checkpoint call (Checkpointer.success(), Checkpointer.failure()) participates in a Spring @Transactional boundary. If the service method is annotated @Transactional, the Azure Service Bus disposition (complete/abandon) is NOT rolled back if the transaction rolls back. Design the message disposition order assuming these are always independent operations.
  5. Session lock duration vs. Stripe p99. If using session-aware consumers (session-aware=true), verify that the billing queue’s session lock duration exceeds the Stripe API p99 call latency for your charge type plus a 2× safety margin. The default session lock duration is 30 seconds. Stripe 3DS calls for international cards can take 30–45 seconds. Set the queue’s LockDuration to at least Stripe_p99 × 2 (120 seconds for 3DS). Monitor session lock renewal WARN logs from the SCS ASB Binder; more than 3 renewal failures per minute indicates the lock duration is too short or Azure Service Bus is under throttling pressure.
  6. Session lock renewal monitoring. The SCS ASB Binder’s session lock renewal failure is logged at WARN level and does not interrupt the consumer function or trigger the error channel. Without explicit monitoring, a session reassignment due to renewal failure is invisible until the duplicate charge appears in Stripe. Add a log-based alert on the SCS ASB Binder’s Failed to renew session lock warning. If this alert fires during a billing run, treat it as an incident requiring immediate Stripe charge audit for the affected session IDs.
  7. BillingRecord guard before Stripe call. Verify that the billing consumer checks for an existing BillingRecord by account ID and billing period BEFORE every Stripe call. This guard is the defense against session reassignment redeliveries that arrive after Stripe’s concurrent idempotency window closes, against DLQ re-enqueues (where the stable key may have expired in Stripe’s 24-hour cache), and against manual recovery scripts that re-enqueue billing messages without checking the billing records table first.
  8. DLQ re-enqueue runbook accuracy. Verify that any runbook for processing the billing queue’s DLQ includes a step that checks the billing_records table for an existing BillingRecord before re-enqueuing the DLQ message. The DeadLetterReason and DeadLetterErrorDescription headers on a DLQ message describe why the message was dead-lettered — not whether Stripe was called. A DataIntegrityViolationException recorded in the dead-letter headers means the DB write failed at some point; it does not mean Stripe was not called. Treat all DLQ messages as potentially already-charged until the billing records table confirms otherwise.

SCS Azure Service Bus Binder Stripe double-charge modes vs. SCS Kafka, RabbitMQ, RocketMQ, and AEH binders

Binder Retry multiplication risk Checkpoint ordering risk Session / partition reassignment risk
SCS Kafka Binder Low — DLT prevents silent retry multiplication; single retry loop N/A — Kafka offset commit is not reversible but can’t ACK before function returns in most modes Moderate — partition rebalance can redeliver from committed offset
SCS RabbitMQ Binder Low — basicReject dead-letters after retry; no MaxDeliveryCount multiplier Low — AUTO ack is post-return; MANUAL risks are similar to ASB Low — no session model; per-message lock prevents concurrent receipt
SCS RocketMQ Binder High — RECONSUME_LATER routes to retry topic; up to maxReconsumeTimes × SCS retries Low — RocketMQ ack is post-return; no MANUAL mode equivalent N/A — no session model; partition offset reset is the reassignment risk
SCS AEH Binder Low — BATCH retry is within one batch; no external redelivery multiplier Moderate — MANUAL Checkpointer.success() independent of Spring TX (same as ASB) High — partition lease expiry triggers redelivery from checkpoint position
SCS ASB Binder (non-session) High — max-attempts × MaxDeliveryCount (3 × 10 = 30 default) High — MANUAL Checkpointer.success() before DB guard leaves ACKed-but-unguarded gap Low — message lock (60s) is longer than most Stripe calls; reassignment requires explicit lock expiry
SCS ASB Binder (session-aware) High — same as non-session; plus session reassignment adds a third layer High — same as non-session High — session lock (30s default) shorter than Stripe 3DS p99; reassignment is immediate on expiry

The SCS Azure Service Bus Binder (especially in session-aware mode) has the highest combined risk profile of any SCS binder for Stripe idempotency failures, primarily because it has three independent failure layers that can each produce a duplicate charge, and the two lock durations (session lock at 30 seconds, message lock at 60 seconds) both have defaults shorter than Stripe’s 3DS p99 latency for international cards. The stable idempotency key + BillingRecord guard pattern closes all three failure modes regardless of which combination of locks, retry counts, and session configurations is used.

The universal fix: stable idempotency key + BillingRecord guard + correct Checkpointer ordering

All three SCS Azure Service Bus Binder failure modes described in this post are closed by the same implementation pattern, applied consistently:

@Component
public class BillingConsumer {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    @Bean
    public Consumer<Message<BillingEvent>> billing() {
        return message -> {
            BillingEvent event = message.getPayload();

            // Optional: inject Checkpointer for MANUAL mode
            // Checkpointer checkpointer = (Checkpointer) message.getHeaders()
            //     .get(ServiceBusMessageHeaders.CHECKPOINTER);

            // Rule 1: Guard before Stripe — check if already charged
            if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                    event.getAccountId(), event.getBillingPeriod())) {
                // Already charged: ACK if MANUAL mode, then return
                // checkpointer.success().block();
                return;
            }

            // Rule 2: Stable idempotency key — same value on every retry and redelivery
            String idempotencyKey = Hashing.sha256()
                .hashString(
                    event.getAccountId() + "|" + event.getBillingPeriod()
                        + "|" + event.getAmountCents() + "|" + event.getUnitsConsumed(),
                    StandardCharsets.UTF_8)
                .toString();

            ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build();

            Charge charge = stripe.charges().create(params,
                RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());

            // Rule 3: Save BillingRecord BEFORE calling Checkpointer.success() in MANUAL mode
            billingRecordRepo.save(new BillingRecord(
                event.getAccountId(),
                event.getBillingPeriod(),
                charge.getId(),
                idempotencyKey
            ));

            // Rule 4: ACK only after guard is durably written (MANUAL mode only)
            // checkpointer.success().block();
        };
    }
}

The four rules work together to close all three failure modes:

Stop running agent billing unguarded

Keybrake issues scoped vault keys for your agent’s Stripe calls — each key carries a per-billing-period spend cap, an allowlist of permitted charge amounts, and an audit trail of every POST /v1/charges call your agent makes. A runaway billing consumer hits the cap and stops. A duplicate charge attempt is logged as a policy violation before the second call reaches Stripe.