Spring Cloud Stream Azure Event Hubs Stripe Integration: How SCS AEH Binder BATCH Checkpoint Mode Retries the Full Batch After ch_A Through ch_49 Commit Generating UUID_B Committing ch_B × 49, How RECORD Mode Partition Lease Expiry During a Slow 3DS Call Delivers the In-Flight Event to a Second Pod Generating UUID_B Committing ch_B, and How TIME Mode Blob Storage Throttling Causes 200-Event Replay Generating UUID_B × 200 Committing ch_B × 200

The Spring Cloud Stream Azure Event Hubs Binder has three failure modes that produce duplicate Stripe charges, and they arise from the binder’s checkpoint mode abstraction behaving in ways that differ structurally from the SCS Kafka, RabbitMQ, RocketMQ, and Pulsar binders. The AEH Binder’s BATCH checkpoint mode retries the entire batch function — not just the failing event — when any event throws; its RECORD mode ties checkpoint writes to successful function return, which means a partition lease expiry while a slow Stripe 3DS call is in-flight delivers the same event to a second pod from the pre-charge offset; and its TIME mode commits checkpoints on a wall-clock interval entirely disconnected from Blob Storage availability, so throttling events leave a silent gap that a pod restart will replay from start to finish. Each of these behaviors can silently commit ch_B against a Stripe customer who was already charged ch_A.

This post covers all three end to end: the exact SCS Azure Event Hubs Binder configuration that enables each failure, the step-by-step timeline showing when ch_A commits and when UUID_B is generated, the specific misconception each one exploits about AEH Binder checkpoint semantics, and the idempotency fix that makes your billing consumer safe regardless of which checkpoint mode you use or how Azure Event Hubs partitioning delivers or redelivers the event. Code samples use Spring Cloud Stream 4.x, spring-cloud-azure-stream-binder-eventhubs 5.x, Spring Boot 3.x, WireMock, Testcontainers, and JUnit 5.

Background: how the SCS Azure Event Hubs Binder checkpoint model differs from the SCS Kafka and RabbitMQ binders

To understand why these three failure modes exist, you need to understand two structural differences between the SCS AEH Binder’s checkpoint model and the consumer acknowledgement models used by the SCS Kafka and RabbitMQ binders.

The AEH Binder externalizes consumer progress to Azure Blob Storage, making checkpoint writes a separate I/O operation with its own failure modes. SCS Kafka Binder tracks consumer offsets by committing them to the Kafka broker’s internal __consumer_offsets topic — a Kafka write that participates in the broker’s replication protocol. SCS RabbitMQ Binder tracks acknowledgements as AMQP frames that the broker persists to its own store. In both cases, consumer progress is stored within the same system that delivers the messages. The SCS AEH Binder stores consumer progress in Azure Blob Storage — a completely separate service with its own availability characteristics, IOPS limits, throttling behavior, and lease-expiry semantics. A Blob Storage partition key hotspot, a storage account IOPS limit, or a transient 503 from the Azure Storage service can decouple the consumer’s actual processing position from its last committed checkpoint position. The SCS AEH Binder does not stop processing when checkpoint writes fail; it logs a warning and continues advancing through the event stream, silently accumulating a gap between processed position and committed position.

The AEH Binder offers five checkpoint modes — BATCH, RECORD, TIME, COUNT, and MANUAL — each with different granularity and timing, and none of them is the per-event, in-line-with-processing acknowledgement that RabbitMQ’s basicAck provides. In SCS RabbitMQ Binder, when the consumer function returns normally, the binder immediately sends basicAck to the RabbitMQ broker as part of the same thread of control — before the binder’s message listener returns. The acknowledgement is synchronous with processing. In SCS AEH Binder, checkpoint writes are always asynchronous: they are submitted to the Blob Storage client, which batches and schedules them according to its own concurrency model. The checkpoint write can fail, be delayed, or race with a partition lease expiry entirely independently of whether the consumer function returned successfully.

With these two structural differences as context, the three failure modes follow directly from specific checkpoint mode configurations.

Failure mode 1: BATCH checkpoint mode — one event in a 50-event batch throws after ch_A through ch_49 commit; SCS retries the full batch function three times; UUID.randomUUID() generates UUID_B through UUID_B49 committing ch_B through ch_B49

The setup

A team runs a usage-based billing service for a developer tools platform. The message broker is Azure Event Hubs. The Spring Boot 3.x billing service uses the Spring Cloud Stream Azure Event Hubs Binder to consume metered usage events and convert them to Stripe charges. The team uses the AEH Binder’s default checkpoint mode and the SCS default retry settings:

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing
          group: billing-group
          consumer:
            max-attempts: 3          # SCS application-level retries (default)
            back-off-initial-interval: 1000
      eventhubs:
        bindings:
          billing-in-0:
            consumer:
              checkpoint:
                mode: BATCH          # checkpoint after entire batch (AEH Binder default)
    azure:
      eventhubs:
        connection-string: "${AZURE_EVENTHUBS_CONNECTION_STRING}"
        processor:
          checkpoint-store:
            account-name: "${AZURE_STORAGE_ACCOUNT}"
            account-key: "${AZURE_STORAGE_KEY}"
            container-name: billing-checkpoints

With checkpoint.mode=BATCH, the SCS AEH Binder delivers a batch of events to the consumer as a List<EventData> and writes the checkpoint (the partition offset of the last event in the batch) only after the consumer function returns normally for the entire batch. The consumer is written as a Consumer<List<BillingEvent>>:

@Component
public class BillingConsumer implements Consumer<List<BillingEvent>> {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    @Override
    public void accept(List<BillingEvent> events) {

        for (BillingEvent event : events) {

            // BUG: fresh UUID per invocation — no stable idempotency key
            String idempotencyKey = UUID.randomUUID().toString();

            ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .setDescription("Usage " + event.getAccountId()
                    + " period " + event.getBillingPeriod()
                    + " units " + event.getUnitsConsumed())
                .build();

            Charge charge = stripe.charges().create(params,
                RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .build());

            billingRecordRepo.save(new BillingRecord(
                event.getAccountId(),
                event.getBillingPeriod(),
                charge.getId(),
                idempotencyKey
            ));
        }
    }
}

The failure timeline

During an end-of-month billing run, the AEH Binder delivers a batch of 50 BillingEvent messages to the consumer. The batch covers accounts acct_001 through acct_050, each with USD 89 in usage charges for the billing period.

• Events 1–49 (t=0ms to t=24,500ms): Each event is processed sequentially within the for loop. For each event, the consumer generates a fresh UUID.randomUUID() and calls Stripe. Stripe commits ch_A_1 through ch_A_49, USD 89 each. billingRecordRepo.save() succeeds for all 49 accounts. So far: no exception, no checkpoint written yet (BATCH mode waits for the full batch).
• Event 50 (t=24,600ms): Consumer generates UUID_A_50 and calls Stripe. Stripe commits ch_A_50. Consumer calls billingRecordRepo.save() for acct_050. The billing_records table has a unique constraint on (account_id, billing_period). A prior partial billing run three weeks ago inserted a record for acct_050 in the current period. DataIntegrityViolationException is thrown.
• t=24,600ms: Exception propagates out of accept(). The SCS AEH Binder’s retry interceptor catches it. Because max-attempts=3, the binder schedules a retry. No checkpoint is written. The batch’s partition offset in Blob Storage still points to the position before event 1.
• t=25,600ms (first retry, after 1s back-off): The SCS binder re-invokes accept() with the same 50-event List. For each of events 1–49, the consumer calls UUID.randomUUID() again — generating UUID_B_1 through UUID_B_49. Stripe receives 49 new POST /v1/charges requests with the new idempotency keys. Because UUID_B_1 is different from UUID_A_1, Stripe creates 49 new charges: ch_B_1 through ch_B_49. At event 50, the same exception fires. First retry ends.
• t=26,600ms (second retry): Same as above. UUID_C_1 through UUID_C_49 → ch_C_1 through ch_C_49. Third try for event 50 also fails.
• t=27,600ms (third retry): UUID_D_1 through UUID_D_49 → ch_D_1 through ch_D_49. Event 50 fails again. After max-attempts=3 retries, the binder routes the message to the error channel (or drops it if no error channel is configured). Checkpoint remains uncommitted.

Outcome: 49 accounts each charged 4 times (original + 3 retries). acct_001 through acct_049: USD 89 × 4 = USD 356 each, totalling USD 17,444 in duplicate charges against USD 4,361 owed. acct_050 was charged once (Stripe succeeded on the first attempt) but has no BillingRecord — a different class of problem.

Because the checkpoint was never written, the partition offset in Blob Storage still points before event 1. If the pod restarts at any point during or after this sequence, the entire 50-event batch is redelivered again, potentially adding four more duplicate charges per account.

The misconception

“BATCH checkpoint mode in SCS Azure Event Hubs Binder means the checkpoint is written at the end of the batch after all events are processed. If one event fails, only that event is retried — events 1 through 49 were processed successfully and their per-event offsets are committed into the Azure Blob Storage checkpoint store as part of the batch’s incremental checkpoint writes. Retrying the batch function re-invokes the consumer only from the failing event, not from event 1.”

This is wrong in two ways. First, the SCS AEH Binder’s BATCH mode writes exactly one checkpoint per batch: the offset of the last event in the batch. It does not write incremental per-event checkpoints as the for loop advances. Between the delivery of the batch and the function’s normal return, the Blob Storage checkpoint store has no record that any events in the batch were processed. Second, when the SCS application-level retry re-invokes the consumer function (because max-attempts > 1), it re-invokes the function with the same input that caused the original failure. For a Consumer<List<BillingEvent>>, the same input is the same 50-event List. The retry does not strip out the events that succeeded on the first attempt.

This is a meaningful difference from SCS Kafka Binder behavior. The SCS Kafka Binder’s default retry template also re-invokes the consumer function on the same input, but in Kafka’s case the “input” for a Consumer<BillingEvent> is a single BillingEvent — there is no in-binder batching of the retry scope. In SCS AEH Binder, the batch size is controlled by the Azure Event Hubs EventProcessorClient’s prefetch and max-batch-size settings, and the entire batch is the unit of retry.

The SCS AEH Binder checkpoint mode descriptions, for reference:

Diagnosing this failure in production

The BATCH mode duplicate charge pattern has a distinctive Stripe dashboard signature: multiple charges for the same account and billing period, with charge IDs clustered in groups based on batch size. In a 50-event batch with max-attempts=3, you see 4 charges per account (the original plus 3 retries), all within a window of max-attempts × back-off-initial-interval — 3 seconds for the default 1-second back-off.

In the application logs, you see the DataIntegrityViolationException for event 50 repeated three times, with “Retrying” log lines in between. The 49 Stripe calls for events 1–49 on each retry appear as normal successes in the logs because they do not throw — they only create the duplicate charges silently.

The fastest diagnostic SQL once you suspect a billing run produced batch-mode duplicates:

-- Find accounts with multiple charges per billing period,
-- grouped by the time window consistent with BATCH retry spacing
SELECT account_id, billing_period,
       COUNT(*) AS charge_count,
       MIN(created_at) AS first_charge_at,
       MAX(created_at) AS last_charge_at,
       EXTRACT(EPOCH FROM (MAX(created_at) - MIN(created_at))) AS window_seconds,
       ARRAY_AGG(stripe_charge_id ORDER BY created_at) AS charges
FROM billing_records
WHERE billing_period = '2026-10'
GROUP BY account_id, billing_period
HAVING COUNT(*) > 1
ORDER BY charge_count DESC, first_charge_at;

If window_seconds is consistently small (under 10 seconds for default back-off settings) and charge_count matches max-attempts + 1 for most affected accounts, you are looking at a BATCH checkpoint mode retry blast.

The fix

Generate the idempotency key from stable payload fields before the Stripe call, and add a guard that checks for an existing BillingRecord per account per billing period. Both are necessary: the stable key deduplicates concurrent Stripe calls within the 30-minute Stripe idempotency cache window, and the guard protects against retries that arrive after the cache expires or against replays from pod restarts where the checkpoint was never written.

@Component
public class BillingConsumer implements Consumer<List<BillingEvent>> {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    @Override
    public void accept(List<BillingEvent> events) {

        for (BillingEvent event : events) {

            // Stable key: deterministic from payload — same on every retry
            String idempotencyKey = sha256(
                event.getAccountId()    + "|" +
                event.getBillingPeriod() + "|" +
                event.getAmountCents()   + "|" +
                event.getUnitsConsumed()
            );

            // Guard: skip Stripe if this billing period was already committed
            if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                    event.getAccountId(), event.getBillingPeriod())) {
                continue; // idempotent: already processed, skip without re-charging
            }

            ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .setDescription("Usage " + event.getAccountId()
                    + " period " + event.getBillingPeriod()
                    + " units " + event.getUnitsConsumed())
                .build();

            Charge charge = stripe.charges().create(params,
                RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .build());

            billingRecordRepo.save(new BillingRecord(
                event.getAccountId(),
                event.getBillingPeriod(),
                charge.getId(),
                idempotencyKey
            ));
        }
    }

    private static String sha256(String input) {
        try {
            MessageDigest digest = MessageDigest.getInstance("SHA-256");
            byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash);
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
    }
}

With the stable key and guard, a BATCH mode retry for events 1–49 computes the same SHA-256 key as the original attempt. Stripe’s idempotency cache returns the cached ch_A_N response for each of those 49 events, and then the guard’s existsByAccountIdAndBillingPeriod check returns true before the Stripe call even happens — so Stripe is called exactly once per account per billing period regardless of how many times the batch is retried.

Test: Stripe called exactly once per event despite three BATCH mode retries

@SpringBootTest
@Testcontainers
class BatchCheckpointRetryTest {

    @Container
    static PostgreSQLContainer<?> postgres =
        new PostgreSQLContainer<>("postgres:16")
            .withDatabaseName("billing_test")
            .withUsername("billing")
            .withPassword("billing");

    @Autowired
    private BillingConsumer billingConsumer;

    @Autowired
    private BillingRecordRepository billingRecordRepo;

    private WireMockServer wireMock;

    @BeforeEach
    void setUp() {
        wireMock = new WireMockServer(WireMockConfiguration.wireMockConfig().port(9099));
        wireMock.start();
        // Return unique charge IDs but accept any idempotency key
        wireMock.stubFor(WireMock.post(WireMock.urlEqualTo("/v1/charges"))
            .willReturn(WireMock.aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_A\",\"object\":\"charge\","
                    + "\"amount\":8900,\"status\":\"succeeded\"}")));
    }

    @Test
    void batchRetry_stripeCalledOncePerEventAcrossThreeRetries() {
        // Build a 5-event batch (use 5 instead of 50 for test speed)
        List<BillingEvent> batch = IntStream.range(1, 6)
            .mapToObj(i -> BillingEvent.builder()
                .accountId("acct_00" + i)
                .billingPeriod("2026-10")
                .amountCents(8900L)
                .unitsConsumed(100L)
                .stripeCustomerId("cus_test_" + i)
                .build())
            .toList();

        // First invocation: all 5 events processed; Stripe commits ch_A × 5
        billingConsumer.accept(batch);
        assertThat(wireMock.getAllServeEvents()).hasSize(5);

        // Simulate BATCH retry #1 (binder re-invokes with same batch)
        billingConsumer.accept(batch);
        // Simulate BATCH retry #2
        billingConsumer.accept(batch);
        // Simulate BATCH retry #3
        billingConsumer.accept(batch);

        // Stripe still called exactly 5 times total — guard blocked all retries
        assertThat(wireMock.getAllServeEvents()).hasSize(5);

        // Exactly 1 BillingRecord per account
        for (int i = 1; i <= 5; i++) {
            assertThat(billingRecordRepo.countByAccountIdAndBillingPeriod(
                "acct_00" + i, "2026-10")).isEqualTo(1);
        }
    }

    @AfterEach
    void tearDown() { wireMock.stop(); }
}

Failure mode 2: RECORD checkpoint mode — Stripe 3DS authentication takes 18 seconds; partition lease expires at t=60s; second EventProcessorClient pod redelivers E1 from the pre-charge offset; UUID.randomUUID() generates UUID_B committing ch_B

The setup

The team switches to checkpoint.mode=RECORD after reading about the BATCH mode retry problem. In RECORD mode, the binder writes a checkpoint to Blob Storage after each individual event’s consumer function returns normally. The intent is that a partition rebalance or pod restart will resume from the last successfully processed event, not from the start of a batch.

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing
          group: billing-group
          consumer:
            max-attempts: 1          # no application-level retries
      eventhubs:
        bindings:
          billing-in-0:
            consumer:
              checkpoint:
                mode: RECORD         # checkpoint after each individual event

The billing consumer is now a Consumer<BillingEvent> processing one event at a time:

@Component
public class BillingConsumer implements Consumer<BillingEvent> {

    private final StripeClient stripe;
    private final BillingRecordRepository billingRecordRepo;

    @Override
    public void accept(BillingEvent event) {

        // BUG: fresh UUID per invocation
        String idempotencyKey = UUID.randomUUID().toString();

        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getStripeCustomerId())
            .setDescription("Usage " + event.getAccountId()
                + " period " + event.getBillingPeriod()
                + " units " + event.getUnitsConsumed())
            .build();

        Charge charge = stripe.charges().create(params,
            RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build());

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            charge.getId(),
            idempotencyKey
        ));
    }
}

A subset of the team’s B2B customers require 3D Secure (3DS2) authentication for charges above EUR 150. When Stripe’s POST /v1/charges triggers a 3DS flow, the API call can take 15–25 seconds to complete as the Stripe server waits for the card issuer’s authentication response. The team’s billing service is deployed as a 3-pod Kubernetes Deployment with rolling updates enabled.

The AEH Binder partition lease model

Before describing the failure, it helps to understand how the SCS AEH Binder manages partition ownership. The binder wraps the Azure Event Hubs SDK’s EventProcessorClient, which uses blob leases in Azure Blob Storage to claim and renew partition ownership. Each partition has a corresponding blob in the checkpoint container. When an EventProcessorClient instance claims a partition, it acquires a 60-second blob lease on that partition’s blob. It must renew the lease before it expires. The renewal is performed by a background thread on a configurable interval (default: 30 seconds).

If the renewal thread cannot reach Blob Storage, or if the processing thread is blocked and does not allow the renewal to complete within the 60-second lease duration, the lease expires. When the lease expires, another EventProcessorClient instance — running in one of the other 3 pods — can acquire the lease during its load-balancing loop. The new owner reads the last committed checkpoint from Blob Storage and begins consuming from that offset.

In RECORD mode, the last committed checkpoint for a partition is the offset of the most recently processed event for which the consumer function returned normally and the binder successfully wrote the checkpoint blob. An event that is currently being processed — whose function has not yet returned — has not been checkpointed.

The failure timeline

Event E1 carries a BillingEvent for account acct_EU_001, EUR 249/month, which triggers 3DS.

• t=0ms: Pod 1 owns partition 3. AEH Binder delivers E1 to Pod 1’s consumer. Pod 1 invokes accept(BillingEvent). Last committed checkpoint for partition 3 in Blob Storage: offset of E0 (the event before E1).
• t=5ms: Pod 1 generates UUID_A = "3f7b2a1d-...". Calls POST /v1/charges with Idempotency-Key: UUID_A. Stripe enters 3DS authentication flow. The Stripe HTTP call is blocking on the calling thread.
• t=30,000ms: Pod 1’s renewal background thread attempts to call BlobClient.getLease().renew() to extend the 60-second lease on partition 3’s blob. The Blob Storage service returns a StorageErrorException with BlobUsedForLocking — a transient storage conflict. The renewal fails. (This can also happen under storage account IOPS limits or network transients.)
• t=60,000ms: The blob lease for partition 3 expires. Pod 1’s EventProcessorClient still has the consumer blocked on the Stripe HTTP call — the processing thread does not know the lease has expired.
• t=60,100ms: Pod 2’s EventProcessorClient runs its load-balancing loop, detects that partition 3’s lease has expired, and acquires it. Pod 2 reads the last committed checkpoint from Blob Storage: offset of E0 (E1 was never checkpointed because E1’s accept() on Pod 1 has not yet returned). Pod 2 begins consuming from E0’s offset + 1 = E1’s offset.
• t=60,150ms: Pod 2 receives E1. Pod 2 generates UUID_B = "9c4e1f82-...". Calls POST /v1/charges with Idempotency-Key: UUID_B. Stripe receives a second charge request for the same customer. Because UUID_B ≠ UUID_A, this is treated as a new charge.
• t=60,500ms: Pod 2’s Stripe call succeeds immediately (no 3DS on second attempt — some card issuers authenticate 3DS once per session). Stripe commits ch_B. Pod 2 calls billingRecordRepo.save(), which succeeds. Pod 2’s binder writes checkpoint for E1’s offset. EUR 249 committed.
• t=73,000ms: Pod 1’s Stripe 3DS call finally completes. Stripe returns the authenticated charge. Pod 1 calls billingRecordRepo.save(). DataIntegrityViolationException thrown — the row already exists (written by Pod 2 at t=60,500ms). Pod 1’s consumer throws. Pod 1’s binder tries to write the checkpoint for E1, but BlobClient.setMetadata() returns HTTP 412 Precondition Failed because Pod 2 now holds the lease. The checkpoint write fails silently.

At this point: acct_EU_001 has been charged EUR 249 twice — once by Pod 1 (ch_A, the 3DS-authenticated charge whose ID was lost when Pod 1’s DB write failed) and once by Pod 2 (ch_B, with a clean BillingRecord). The BillingRecord in the database reflects only ch_B. ch_A exists in Stripe’s charge history with no corresponding local record.

The misconception

“RECORD mode in SCS Azure Event Hubs Binder checkpoints each event atomically with its processing — the checkpoint write and the consumer function return happen as a single unit. A partition rebalance therefore cannot redeliver an event that is currently being processed, because the Azure Event Hubs SDK tracks in-progress event deliveries in the lease metadata and excludes them from the rebalancing logic. The new partition owner starts from the last committed checkpoint, which is always the event after the one currently being processed.”

None of these three claims are correct. First, the checkpoint write and the consumer function return are not atomic. The binder writes the checkpoint asynchronously to Blob Storage after the function returns — there is no transaction linking the two. Second, the Azure Event Hubs SDK does not track in-progress event deliveries. It tracks partition ownership via blob leases. Once the lease expires, the partition is available for claim, regardless of whether the current owner’s processing thread is in the middle of a Stripe call. Third, the new partition owner reads the last committed checkpoint, not the last delivered offset. If E1 was delivered but not yet checkpointed (because the consumer function has not yet returned), the new owner starts from E1 again.

This is a structural limitation of the lease-based partition ownership model. There is no way for the SCS AEH Binder to know that a specific event is “in flight” in a way that survives a lease expiry. The only durable record of progress is the committed checkpoint in Blob Storage. If the checkpoint has not been written, the event will be redelivered.

Diagnosing this failure

The 3DS rebalance double-charge has a different signature from the BATCH mode retry blast. You see exactly 2 charges per affected account (not max-attempts + 1), with the second charge arriving approximately 60 seconds after the first (the lease expiry window). In Stripe’s dashboard, the first charge (ch_A) may show as a 3DS-authenticated charge with no corresponding local BillingRecord, while the second (ch_B) has a clean local record.

In Kubernetes logs, the signal is: a DataIntegrityViolationException on Pod 1 at approximately t=73s for an event that Pod 2 processed successfully at t=60.5s, combined with a Pod 2 log showing acquisition of partition 3’s lease at t=60.1s. The AEH Binder logs the partition ownership change at INFO level: EventProcessor claimed ownership of partition 3.

The fix

The stable key + guard from failure mode 1 applies here. When Pod 2 processes E1 and computes the same SHA-256 idempotency key as Pod 1’s in-flight call, Stripe deduplicates the concurrent requests using its own idempotency cache: if Pod 1’s call was already received and is being processed, Stripe holds Pod 2’s call until Pod 1’s completes, then returns the same ch_A response to both callers. If Pod 1’s call has already completed (within the 30-minute Stripe cache window), Stripe immediately returns the cached ch_A response to Pod 2, and the guard’s existsByAccountIdAndBillingPeriod check finds the existing BillingRecord and returns early before calling Stripe.

Additionally, for 3DS-heavy billing workloads, consider setting a longer connection timeout on the Stripe HTTP client to keep the processing thread alive for the full 3DS window, and configure the AEH Binder’s lease renewal interval to be shorter than the expected Stripe call duration:

# Extend lease renewal frequency to reduce expiry risk during slow Stripe calls
spring:
  cloud:
    azure:
      eventhubs:
        processor:
          # Renew partition lease every 15 seconds (default: 30)
          partition-ownership-expiration-interval: PT15S

This reduces but does not eliminate the window between last renewal and lease expiry. The stable key + guard is the only defense that closes the failure mode completely.

Test: concurrent pod redelivery does not commit a second Stripe charge

@SpringBootTest
@Testcontainers
class RecordModeLeaseExpiryTest {

    @Container
    static PostgreSQLContainer<?> postgres =
        new PostgreSQLContainer<>("postgres:16")
            .withDatabaseName("billing_test")
            .withUsername("billing")
            .withPassword("billing");

    @Autowired
    private BillingConsumer billingConsumer;

    @Autowired
    private BillingRecordRepository billingRecordRepo;

    private WireMockServer wireMock;

    @BeforeEach
    void setUp() {
        wireMock = new WireMockServer(WireMockConfiguration.wireMockConfig().port(9099));
        wireMock.start();
        wireMock.stubFor(WireMock.post(WireMock.urlEqualTo("/v1/charges"))
            .willReturn(WireMock.aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_A\",\"object\":\"charge\","
                    + "\"amount\":24900,\"status\":\"succeeded\"}")));
    }

    @Test
    void concurrentRedelivery_stripeCalledOnceWhenSecondPodDeliversSameEvent()
            throws InterruptedException {

        BillingEvent event = BillingEvent.builder()
            .accountId("acct_EU_001")
            .billingPeriod("2026-10")
            .amountCents(24900L)
            .unitsConsumed(1L)
            .stripeCustomerId("cus_eu_test")
            .build();

        // Simulate Pod 2 processing first (lease winner, commits BillingRecord)
        billingConsumer.accept(event);

        // Simulate Pod 1 redelivery (lease loser, arrives after Pod 2 committed)
        // With stable key + guard: Stripe is not called, function returns silently
        billingConsumer.accept(event);

        // Stripe called exactly once
        List<LoggedRequest> requests = wireMock.findAll(
            WireMock.postRequestedFor(WireMock.urlEqualTo("/v1/charges")));
        assertThat(requests).hasSize(1);

        // Single BillingRecord with ch_A
        assertThat(billingRecordRepo.countByAccountIdAndBillingPeriod(
            "acct_EU_001", "2026-10")).isEqualTo(1);
        assertThat(billingRecordRepo
            .findByAccountIdAndBillingPeriod("acct_EU_001", "2026-10")
            .getStripeChargeId()).isEqualTo("ch_A");
    }

    @AfterEach
    void tearDown() { wireMock.stop(); }
}

Failure mode 3: TIME checkpoint mode — Azure Blob Storage throttling prevents checkpoint writes for 45 seconds; consumer processes 200 events and commits ch_A_1 through ch_A_200; pod OOM-killed and restarted; pod redelivers 200 events from last committed checkpoint; UUID.randomUUID() generates UUID_B_1 through UUID_B_200 committing ch_B_1 through ch_B_200

The setup

After investigating the RECORD mode lease expiry issue, the team decides to switch to TIME mode, reasoning that periodic checkpointing is more robust than per-event checkpoint writes because the checkpoint I/O is amortized across many events. The configuration:

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing
          group: billing-group
          consumer:
            max-attempts: 1
      eventhubs:
        bindings:
          billing-in-0:
            consumer:
              checkpoint:
                mode: TIME
                time: 10000          # checkpoint every 10 seconds

In TIME mode, the SCS AEH Binder schedules a periodic checkpoint write every checkpoint.time milliseconds. Each scheduled write flushes the current partition offset to Blob Storage. Unlike RECORD mode, the checkpoint is not tied to any individual event’s processing outcome. The binder simply records “we have processed up to offset X as of now” on the schedule.

The billing service is deployed across 32 Event Hub partitions, with all partitions actively processing billing events during a large end-of-day batch job. The storage account used for checkpointing is shared with three other services in the same Azure subscription. The storage account IOPS limit (20,000 IOPS for standard tier) is approached during peak batch processing.

The failure timeline

The failure unfolds across three phases:

Phase 1: Blob Storage throttling (t=0 to t=45s)

• t=0ms: The billing job starts. The SCS AEH Binder begins scheduling TIME mode checkpoint writes every 10 seconds. At t=0, the last committed checkpoint for partition 7 is at offset 6,800,000 (the end of the previous billing run).
• t=10,000ms: First scheduled checkpoint write attempts to write offset 6,800,050 (50 events processed in 10 seconds) to the partition 7 blob. Blob Storage returns HTTP 503 Service Unavailable with ServerBusy. The SCS AEH Binder logs: WARN c.a.m.e.EventProcessorClient - Failed to update checkpoint for partition 7: StorageRequestFailedException: The server is busy. The binder does NOT stop processing. The consumer continues receiving and processing events from partition 7.
• t=10,000ms to t=45,000ms: Each subsequent 10-second checkpoint write attempt for partition 7 also returns 503. Four more checkpoint writes fail. During this 45-second window, the consumer processes 200 billing events (events at offsets 6,800,000 through 6,800,199). For each event, UUID.randomUUID() generates a unique key, Stripe commits ch_A_1 through ch_A_200, and billingRecordRepo.save() saves the BillingRecord. Processing is normal — only the checkpoint writes are failing.
• t=45,000ms: Blob Storage IOPS pressure eases. The next scheduled checkpoint write succeeds — but it writes offset 6,800,000 (the last attempted offset that failed), not 6,800,199 (the current processing position). The SCS AEH Binder’s TIME mode buffers the “checkpoint to write” based on when the timer fires, not on the current offset at the time of the actual write. The committed checkpoint lags the actual processing position by 199 events.

Phase 2: Pod OOM-kill (t=55s)

• t=55,000ms: The billing job’s memory usage spikes — the batch is generating large Stripe response objects that are not being garbage-collected quickly enough. Kubernetes OOM-kills the pod. The pod terminates without checkpointing the remaining 199 events.

Phase 3: Pod restart and replay (t=65s to t=162s)

• t=65,000ms: Kubernetes restarts the pod. The SCS AEH Binder’s EventProcessorClient initializes and claims partition 7’s lease from Blob Storage. The last committed checkpoint is at offset 6,800,000 — the value written at t=45s during the brief throttling recovery. The pod begins consuming from offset 6,800,001.
• t=65,100ms to t=162,000ms: The pod processes events at offsets 6,800,001 through 6,800,199 — the 200 events that were processed and Stripe-charged during the throttling window. For each, UUID.randomUUID() generates a new key: UUID_B_1 through UUID_B_200. Stripe has no record of these keys. 200 new charges are committed: ch_B_1 through ch_B_200. billingRecordRepo.save() throws DataIntegrityViolationException for each (the BillingRecord from the first processing run already exists). With max-attempts=1, the binder routes each failure to the error channel. 200 events end up in the error channel with no corresponding duplicate BillingRecord — but with 200 duplicate Stripe charges that are invisible in the local DB.

Outcome: 200 accounts double-charged. The local database shows no evidence of the duplicates (each BillingRecord has a single row per account per period, written during Phase 1). The duplicates are only visible in Stripe’s charge history, where each affected account has two charges under different charge IDs and different idempotency keys.

The misconception

“When the SCS Azure Event Hubs Binder’s TIME mode checkpoint write fails due to Blob Storage throttling, the binder pauses processing and queues the failed events for retry. The consumer does not advance beyond the last successfully committed checkpoint position during a throttling event. When the throttling resolves and the checkpoint write succeeds, the committed position reflects the actual processing progress. A pod restart therefore resumes from the true processing position, not from the stale pre-throttling position.”

The SCS AEH Binder in TIME mode does not pause processing on checkpoint write failure. The checkpoint write is a background operation that runs on a fixed schedule. If the write fails, the binder logs the failure and the background timer fires again at the next interval. Meanwhile, the consumer continues processing events from Azure Event Hubs — the event stream delivery is not gated on checkpoint success. This is by design: pausing event consumption every time a Blob Storage write fails would cascade storage throttling into complete consumer stalls during high-throughput batch processing. The trade-off is that checkpoint write failures silently accumulate a gap between processed position and committed position.

The TIME mode checkpoint gap is particularly dangerous because it is invisible without actively monitoring Blob Storage write failures. The application logs show WARN-level messages from the AEH SDK, but these are easy to miss in the noise of a billing batch run. The billing_records table shows all 200 events processed correctly. The Stripe dashboard shows 200 additional charges. The mismatch is only visible by comparing the Stripe charge count to the billing record count per billing period.

The TIME mode checkpoint lag vs. other checkpoint modes

The maximum checkpoint lag for each mode under a 45-second Blob Storage outage:

The important conclusion from this table: no SCS AEH Binder checkpoint mode protects against Blob Storage throttling during a pod restart without a consumer-side idempotency implementation. All checkpoint modes ultimately depend on Blob Storage availability for durable progress tracking. The checkpoint mode controls when checkpoint writes are attempted, not whether they succeed. Under storage throttling, all modes accumulate lag. A pod restart during or after a throttling event will replay all events in the lag window regardless of checkpoint mode.

The fix

The stable key + guard from failure modes 1 and 2 applies identically here. When the pod restarts and reprocesses the 200 events from the throttling window, the guard’s existsByAccountIdAndBillingPeriod check finds the existing BillingRecord (written during Phase 1) and returns early for all 200 events. No Stripe calls are made for events that were already processed. The DataIntegrityViolationException from the DB save is also eliminated because the function returns before reaching the save call.

Additionally, for billing services on Azure, configure a dedicated storage account for AEH checkpointing, separate from the storage accounts used by other services in the subscription. This prevents checkpoint write failures caused by IOPS contention with unrelated workloads:

# application.yml — dedicated storage account for AEH checkpoints
spring:
  cloud:
    azure:
      eventhubs:
        processor:
          checkpoint-store:
            account-name: "${BILLING_CHECKPOINT_STORAGE_ACCOUNT}"
            account-key: "${BILLING_CHECKPOINT_STORAGE_KEY}"
            container-name: billing-eventhubs-checkpoints
            # Use a Premium Blob Storage account for higher IOPS if billing
            # throughput exceeds standard tier limits during batch runs

And monitor checkpoint write failures explicitly via the AEH SDK’s metrics or by parsing the WARN log pattern from the EventProcessorClient:

# alertmanager / Loki rule for checkpoint write failures in billing pods
- alert: BillingAEHCheckpointWriteFailure
  expr: |
    count_over_time({app="billing-service"} |=
      "Failed to update checkpoint" [5m]) > 3
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: AEH checkpoint writes failing — billing replay risk
    description: >
      SCS Azure Event Hubs Binder is failing to write checkpoints to Blob
      Storage. Events processed since the last successful checkpoint will be
      replayed on pod restart. Without stable idempotency keys + BillingRecord
      guard, this produces duplicate Stripe charges equal to the replay count.

Test: TIME mode replay after checkpoint gap does not commit duplicate Stripe charges

@SpringBootTest
@Testcontainers
class TimeModeCheckpointGapReplayTest {

    @Container
    static PostgreSQLContainer<?> postgres =
        new PostgreSQLContainer<>("postgres:16")
            .withDatabaseName("billing_test")
            .withUsername("billing")
            .withPassword("billing");

    @Autowired
    private BillingConsumer billingConsumer;

    @Autowired
    private BillingRecordRepository billingRecordRepo;

    private WireMockServer wireMock;

    @BeforeEach
    void setUp() {
        wireMock = new WireMockServer(WireMockConfiguration.wireMockConfig().port(9099));
        wireMock.start();
        wireMock.stubFor(WireMock.post(WireMock.urlEqualTo("/v1/charges"))
            .willReturn(WireMock.aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withTransformerParameter("responseTemplatingEnabled", true)
                .withBody("{\"id\":\"ch_A{{randomValue length=8 type='NUMERIC'}}\","
                        + "\"object\":\"charge\",\"amount\":8900,\"status\":\"succeeded\"}")));
    }

    @Test
    void timeModeReplay_alreadyChargedEventsSkippedOnReplay() {
        // Simulate 10 events processed before pod OOM-kill
        // (representing events during the throttling window)
        List<BillingEvent> throttlingWindowEvents = IntStream.range(0, 10)
            .mapToObj(i -> BillingEvent.builder()
                .accountId("acct_time_" + i)
                .billingPeriod("2026-10")
                .amountCents(8900L)
                .unitsConsumed(50L)
                .stripeCustomerId("cus_time_" + i)
                .build())
            .toList();

        // Phase 1: original processing (all 10 events charged and recorded)
        throttlingWindowEvents.forEach(billingConsumer::accept);
        long stripeCallsAfterFirstRun = wireMock.getAllServeEvents().size();
        assertThat(stripeCallsAfterFirstRun).isEqualTo(10);

        // Phase 3: pod restart replay (same events redelivered from stale checkpoint)
        throttlingWindowEvents.forEach(billingConsumer::accept);

        // Stripe still called exactly 10 times — guard blocked all 10 replays
        assertThat(wireMock.getAllServeEvents()).hasSize(10);

        // 10 BillingRecords, one per account — no duplicates
        for (int i = 0; i < 10; i++) {
            assertThat(billingRecordRepo.countByAccountIdAndBillingPeriod(
                "acct_time_" + i, "2026-10")).isEqualTo(1);
        }
    }

    @AfterEach
    void tearDown() { wireMock.stop(); }
}

Eight-item diagnostic checklist for SCS Azure Event Hubs Binder Stripe billing consumers

Run this checklist against any billing consumer using the Spring Cloud Stream Azure Event Hubs Binder before deploying to production.

1. Idempotency key source. Every call to stripe.charges().create(), stripe.paymentIntents().create(), or any Stripe write API in the consumer must use an idempotency key derived from stable payload fields (account ID, billing period, amount, units consumed), not from UUID.randomUUID(). UUID.randomUUID() generates a different value on every invocation; in SCS AEH Binder, every BATCH mode retry, RECORD mode partition rebalance redelivery, and TIME mode replay from a checkpoint gap is a new invocation of the consumer function.
2. Checkpoint mode and batch scope. If using checkpoint.mode=BATCH, confirm whether the retry scope is the individual event or the full batch. With max-attempts > 1 and a Consumer<List<BillingEvent>>, the retry re-invokes the entire batch. Reduce to max-attempts=1 and handle retry at the Stripe / DB level, or switch to checkpoint.mode=RECORD with a Consumer<BillingEvent> to limit the retry scope to individual events.
3. Lease expiry and 3DS call duration. If any Stripe calls in the consumer can take longer than the AEH partition lease duration (60 seconds by default), configure a shorter partition-ownership-expiration-interval in the Azure Event Hubs processor settings, and verify that the stable key + guard handles the resulting concurrent-pod redelivery case. The stable key ensures Stripe’s in-flight deduplication handles simultaneous calls; the guard handles sequential calls where the first pod’s result was committed.
4. Blob Storage IOPS and throttling monitoring. Monitor the storage account used for AEH checkpoints for IOPS saturation. Standard tier storage accounts are limited to 20,000 IOPS total across all containers. A billing batch job running 32 partitions with RECORD mode checkpointing (one write per processed event) can easily saturate this limit. Use a dedicated Premium Blob Storage account for billing checkpoints during large batch runs, and alert on StorageRequestFailedException from the SCS AEH Binder.
5. BillingRecord guard before Stripe call. Confirm that the billing consumer checks for an existing BillingRecord by account ID and billing period before every Stripe call. This guard is the defense against replay events arriving after Stripe’s 30-minute idempotency cache has expired and against TIME mode checkpoint gaps that go back more than 30 minutes.
6. Checkpoint mode and restart behavior. Understand the replay window for your checkpoint mode under a worst-case Blob Storage outage of the duration you might experience. For TIME mode with checkpoint.time=10000 and a 45-second storage outage, the replay window is approximately 45 seconds of events. Ensure the guard’s DB lookup is fast enough to process all replay events within the billing job’s SLA window.
7. Shared vs. dedicated storage account. Verify that the AEH checkpoint container’s storage account is not shared with other services in the subscription that could generate IOPS spikes during billing batch runs. A shared storage account creates correlated failure modes: a non-billing service hitting the IOPS limit can cause billing checkpoint write failures and billing replay charges without any billing-specific anomaly to alert on.
8. Error channel handling for replay duplicates. If using max-attempts=1 and relying on the error channel for failed events, ensure the error channel handler also uses the stable idempotency key check before taking any action. An error channel handler that re-publishes the failed event to the billing topic without checking the stable key can trigger a second round of duplicate charges.

SCS Azure Event Hubs Binder Stripe double-charge modes vs. SCS Kafka and SCS RabbitMQ binders

Summary: three SCS Azure Event Hubs Binder Stripe failure modes at a glance