Spring Cloud Stream Google PubSub Binder Stripe Integration: How nackOnError=true SCS max-attempts=3 Exhaustion Nacks to GCP After ch_A Through ch_C Commit Making Indefinite Redelivery Generate UUID_D Committing ch_D, How max-ack-extension-period=0 Lets ackDeadline=10s Expire During Stripe 3DS Delivering the Message to a Second Pod Generating UUID_B Committing ch_B, and How deadLetterPolicy.maxDeliveryAttempts=5 × SCS max-attempts=3 Produces 15 Stripe Charges Before DLT Re-Ingestion Generates UUID_P Committing ch_P

The Spring Cloud Stream Google PubSub Binder introduces three layers of GCP-specific abstraction over the raw com.google.cloud:google-cloud-pubsub Java client — a nackOnError=true default that returns messages to GCP’s at-least-once delivery loop indefinitely, a max-ack-extension-period=0 default that disables the ack deadline extension that the raw Subscriber client enables by default, and a dead-letter policy delivery counter that counts GCP-level nacks from SCS retry exhaustion rather than individual consumer function invocations — and each layer has a failure mode that produces duplicate Stripe charges in ways that are invisible to developers coming from Kafka, RabbitMQ, or Azure Service Bus.

This post covers all three failure modes end to end: the exact SCS GCP PubSub Binder configuration that enables each one, the step-by-step timeline showing when ch_A commits and when UUID_B is generated, the specific misconception each one exploits about the GCP PubSub Binder’s behavior, and the idempotency fix that makes your billing consumer safe regardless of nack-driven redelivery, ack deadline expiry, or dead-letter multiplier effects. Code samples use Spring Cloud Stream 4.x, spring-cloud-gcp-pubsub-stream-binder 5.x (Spring Cloud GCP), Spring Boot 3.x, WireMock, Testcontainers, and JUnit 5.

Background: how the SCS Google PubSub Binder differs from other SCS binders at the acknowledgement and retry layers

To understand why these three failure modes exist at the SCS GCP PubSub Binder layer rather than at the raw google-cloud-pubsub client layer, you need to understand three structural differences the GCP PubSub Binder introduces on top of the underlying Subscriber client.

The SCS GCP PubSub Binder defaults nackOnError=true, meaning any consumer function exception returns the message to GCP PubSub for redelivery. In Google Cloud Pub/Sub, a subscriber has two settlement options for each delivered message: acknowledge the message (call AckReplyConsumer.ack(), which removes it from the subscription permanently) or negatively acknowledge it (call AckReplyConsumer.nack(), which returns it to the subscription for redelivery after a backoff determined by the subscription’s retry policy). The SCS GCP PubSub Binder wraps this choice: when the consumer function completes normally, the binder calls ack(); when the consumer function throws and all SCS max-attempts retries are exhausted, the binder calls nack(). This is the GCP PubSub equivalent of the SCS RabbitMQ Binder’s requeueRejected=true default — failed messages re-enter the delivery loop. The critical difference from RabbitMQ is that GCP PubSub has no partition-level ordering and delivers to any subscriber in the subscription, not necessarily the original subscriber pod that nacked the message. With no dead-letter policy configured, GCP redelivers indefinitely up to messageRetentionDuration (default 7 days). Every redelivery triggers a fresh SCS retry cycle, and every retry in that cycle calls UUID.randomUUID().

The SCS GCP PubSub Binder sets max-ack-extension-period=0 by default, which disables the ack deadline extension that the raw Google Cloud Pub/Sub Subscriber Java client enables out of the box. Google Cloud Pub/Sub guarantees each delivered message an acknowledgement deadline: the subscriber must call ack() or nack() within this window (configurable at subscription creation, minimum 10 seconds, maximum 600 seconds). The raw com.google.cloud.pubsub.v1.Subscriber client automatically sends ModifyAckDeadline requests to extend this deadline while the message receiver is processing, up to a configurable maximum (setMaxAckExtensionPeriod(Duration.ofMinutes(60)) is the raw client default). The SCS GCP PubSub Binder configures the underlying Subscriber with max-ack-extension-period from spring.cloud.gcp.pubsub.subscriber.max-ack-extension-period. In many Spring Cloud GCP configurations this defaults to 0, meaning no ack deadline extension. If your GCP subscription has ackDeadlineSeconds=10 (the subscription creation default) and your billing consumer takes 13 seconds due to Stripe 3DS authentication, GCP PubSub redelivers the message at t=10.001s to another subscriber while the first consumer is still processing. Both pods independently generate idempotency keys via UUID.randomUUID() and commit separate Stripe charges.

GCP PubSub’s deadLetterPolicy.maxDeliveryAttempts counts GCP-level nacks, not individual SCS consumer function invocations. When you configure a dead-letter policy on a GCP subscription, the Pub/Sub infrastructure counts each time a subscriber calls nack() (or lets the ack deadline expire) as one delivery attempt. After maxDeliveryAttempts nacks, GCP routes the message to the configured dead-letter topic. The SCS GCP PubSub Binder calls nack() once after all max-attempts SCS retries are exhausted — one nack() per GCP delivery, regardless of how many times SCS internally re-invoked the consumer function. With max-attempts=3 and maxDeliveryAttempts=5, GCP sees 5 nacks and routes to DLT; in reality 5 × 3 = 15 consumer function executions ran, each calling UUID.randomUUID() and making a Stripe charge attempt. The DLT message carries x-goog-pubsub-delivery-attempt: 5 — but engineers reading this as “the billing consumer tried 5 times” will underestimate the actual Stripe call count by a factor of 3.

With these three structural differences as context, the three failure modes follow directly from specific configurations and operational patterns.

Failure mode 1: nackOnError=true (default) + SCS max-attempts=3 — binder nacks after ch_A through ch_C commit; GCP redelivers indefinitely; each redelivery generates UUID_D → ch_D

The setup

A team runs a metered billing service for a developer-tools API platform. The message broker is Google Cloud Pub/Sub. The Spring Boot 3.x billing service uses the SCS GCP PubSub Binder to consume usage events from the billing-events topic (subscription billing-processor) and convert them to Stripe charges. The team uses the SCS GCP PubSub Binder defaults:

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing-events
          group: billing-processor
          consumer:
            max-attempts: 3
            back-off-initial-interval: 500
            back-off-multiplier: 2.0
    gcp:
      pubsub:
        subscriber:
          parallel-pull-count: 1
          # max-ack-extension-period: 0  # default in Spring Cloud GCP 5.x
          # nack-on-error: true          # default in SCS GCP PubSub Binder

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // BUG: fresh random UUID on every consumer invocation (SCS retry or GCP redelivery)
        String idempotencyKey = UUID.randomUUID().toString();

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        // DB write — can fail under connection pool exhaustion
        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            idempotencyKey,
            charge.getId()
        ));
    };
}

The timeline

The billing service is under elevated load during month-end batch processing. The PostgreSQL connection pool is at 97% utilization. A billing event M1 arrives for account acct_99, billing period 2024-09, amountCents=4900:

GCP delivery 1, SCS attempt 1 (t=0ms): SCS delivers M1 to the billing() function. UUID.randomUUID() → idem_A = "a1b2c3d4-e5f6-4789-abcd-ef0123456789". Stripe call begins. At t=490ms Stripe commits ch_A (HTTP 200, charge id ch_stripe_111). At t=495ms billingRecordRepo.save() throws CannotGetJdbcConnectionException (pool exhausted). Consumer function throws. ch_A is committed in Stripe.

GCP delivery 1, SCS attempt 2 (t=1000ms, after 500ms back-off): SCS RetryTemplate re-invokes the consumer with the same M1. UUID.randomUUID() → idem_B = "b2c3d4e5-f6a7-4890-bcde-f01234567890". Stripe commits ch_B at t=1490ms. billingRecordRepo.save() throws again. ch_A and ch_B now committed.

GCP delivery 1, SCS attempt 3 (t=3000ms): UUID.randomUUID() → idem_C = "c3d4e5f6-a7b8-4901-cdef-012345678901". Stripe commits ch_C at t=3490ms. DB throws again. SCS has exhausted max-attempts=3. Exception propagates to the SCS GCP PubSub Binder’s error handler. Binder calls ackReplyConsumer.nack(). M1 returns to the GCP PubSub subscription backlog.

GCP delivery 2 (t=~32s, after GCP retry backoff): GCP PubSub redelivers M1 to any subscriber in the billing-processor subscription — possibly the same pod, possibly a different pod. The subscription has no dead-letter policy, so GCP will keep redelivering indefinitely. SCS receives M1 again. UUID.randomUUID() → idem_D = "d4e5f6a7-b8c9-4012-def0-123456789012". The DB connection pool is still saturated. Stripe commits ch_D. DB throws. SCS retries twice more: idem_E → ch_E, idem_F → ch_F. After 3 SCS attempts, binder calls nack() again.

GCP deliveries 3–N (continuing): Every GCP redelivery triggers 3 more SCS retries, each generating a fresh UUID and a fresh Stripe charge. The Stripe calls succeed consistently — only the DB write fails. By the time the PostgreSQL connection pool recovers 8 minutes later, M1 has been through approximately 16 GCP deliveries × 3 SCS retries = 48 consumer function executions. On the final GCP delivery after DB recovery, the consumer generates idem_P, Stripe commits ch_P, and the DB write succeeds. The monitoring alert for “high DB connection wait” clears. The team closes the incident.

Account acct_99 has been charged 49 times: ch_A through ch_P for a $49.00 billing event. Total: $2,401.00. No error was logged for any of the Stripe calls — they all returned HTTP 200. The only logged errors were CannotGetJdbcConnectionException on the DB writes that followed.

The misconception

“The SCS GCP PubSub Binder retries the billing consumer until it succeeds. max-attempts=3 means each delivery gets 3 chances. If all 3 fail, the binder nacks the message and GCP redelivers it. This is the retry loop — GCP keeps trying until the consumer can process the message successfully. When the DB recovered, the message was finally processed. The consumer generated one Stripe charge (the successful one), and all the failed attempts before that were failures — Stripe was never charged for the failed attempts because the DB write failed first.”

This is wrong on the ordering of operations. In this consumer, Stripe is called before the DB write. The Stripe call commits ch_A at t=490ms; the DB write fails at t=495ms. The Stripe charge is irrevocable. The DB write failure does not undo the Stripe charge. The consumer throws after Stripe has committed the charge, not before. Every SCS retry attempt (whether within a single GCP delivery or across multiple GCP redeliveries) generates a fresh idempotency key via UUID.randomUUID(), calls Stripe with that key, and Stripe commits a new charge for each unique key.

This failure mode has a GCP PubSub-specific amplification factor compared to other SCS binders. With the SCS RabbitMQ Binder and requeueRejected=true (default), a failed message returns to the RabbitMQ queue and the team eventually notices the message cycling. With the SCS Kafka Binder, failed messages are published to the .DLT topic after max-attempts — no further automatic redelivery. With the SCS GCP PubSub Binder and no dead-letter policy, the message bounces indefinitely in the GCP subscription. The bouncing is invisible at the application layer — there is no DLT depth alert, no message cycling indicator, no visible queue depth spike for a single message. The only observable signal is the stream of CannotGetJdbcConnectionException log lines, which the team attributes to the DB connectivity issue, not to the accumulating Stripe charges.

The fix

Derive the idempotency key from the billing event’s immutable content fields. With a stable key, every SCS retry and every GCP redelivery sends the same string to Stripe. Stripe’s 24-hour idempotency cache returns ch_A for every subsequent call with the same key, regardless of how many times the consumer has been invoked:

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // Stable key: identical string for the same billing event on every attempt
        String idempotencyKey = sha256Hex(
            event.getAccountId() + "|" +
            event.getBillingPeriod() + "|" +
            event.getAmountCents()
        );

        // Content-based guard: fast-path return if already processed
        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            return;
        }

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            idempotencyKey,
            charge.getId()
        ));
    };
}

With this fix, the 48 consumer invocations across 16 GCP redeliveries all compute SHA-256("acct_99|2024-09|4900") = "7e3a...". Stripe’s idempotency cache returns ch_A on every call after the first. The 47 subsequent Stripe calls are all cache hits — no new charges are committed. The DB write succeeds on the first attempt where the pool has capacity, and the billing record is stored once. The GCP redelivery loop that would otherwise run for 7 days is harmless.

Also add a dead-letter policy to bound the redelivery loop and trigger an alert when a billing event genuinely cannot be processed:

# Terraform: configure DLT to catch truly unprocessable messages
resource "google_pubsub_subscription" "billing_processor" {
  name  = "billing-processor"
  topic = google_pubsub_topic.billing_events.id

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.billing_dlt.id
    max_delivery_attempts = 10
  }

  retry_policy {
    minimum_backoff = "30s"
    maximum_backoff = "600s"
  }

  ack_deadline_seconds = 120  # > Stripe 3DS p99 (see Mode 2)
}

Failure mode 2: max-ack-extension-period=0 + ackDeadline=10s expires during Stripe 3DS — GCP delivers M1 to a second subscriber pod generating UUID_B committing ch_B

The setup

A team migrates from the raw com.google.cloud.pubsub.v1.Subscriber Java client to the SCS GCP PubSub Binder. In their previous raw-client implementation, the Subscriber was constructed with default settings, which includes setMaxAckExtensionPeriod(Duration.ofMinutes(60)) — the Subscriber client automatically extended the ack deadline every ~33 seconds while the message receiver callback was processing. The team never had to think about ack deadlines.

After migrating to the SCS GCP PubSub Binder, the subscription retains its original ackDeadlineSeconds=10 (the default when a subscription is created via the GCP Console or Terraform without specifying a value). The team does not configure spring.cloud.gcp.pubsub.subscriber.max-ack-extension-period, which defaults to 0 in Spring Cloud GCP — meaning the SCS GCP PubSub Binder’s underlying Subscriber is configured with setMaxAckExtensionPeriod(Duration.ZERO), disabling all ack deadline extension.

# application.yml — MISSING a critical configuration property
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing-events
          group: billing-processor
          consumer:
            max-attempts: 1
    gcp:
      pubsub:
        subscriber:
          # max-ack-extension-period: 0  # DEFAULT — not set, but 0 disables extension
          flow-control:
            max-outstanding-element-count: 10

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // BUG: fresh UUID — different key on concurrent delivery to second pod
        String idempotencyKey = UUID.randomUUID().toString();

        // Stripe Payment Intent with 3DS for EU SCA compliance
        PaymentIntent intent = stripeClient.paymentIntents().create(
            PaymentIntentCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("eur")
                .setCustomer(event.getStripeCustomerId())
                .setPaymentMethod(event.getPaymentMethodId())
                .setConfirm(true)
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        // Waits for 3DS completion (blocking poll)
        PaymentIntent confirmed = waitFor3dsCompletion(intent, Duration.ofSeconds(90));

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(),
            event.getBillingPeriod(),
            idempotencyKey,
            confirmed.getId()
        ));
    };
}

The timeline

M1 is a billing event for a German subscriber using a 3DS-enrolled Maestro card. The billing service runs 3 pods in production.

t=0ms: GCP PubSub delivers M1 to Pod-A. The SCS GCP PubSub Binder’s underlying Subscriber client receives M1. Because max-ack-extension-period=0, the Subscriber will NOT send any ModifyAckDeadline requests. The ack deadline countdown begins: Pod-A has 10 seconds to call ack() or nack() before GCP PubSub redelivers M1.

t=100ms: Pod-A’s billing() function executes. UUID.randomUUID() → idem_A = "f9e8d7c6-b5a4-4321-9876-fedcba098765". Stripe’s Payment Intent API is called. Stripe creates the PaymentIntent and sends a 3DS challenge to the cardholder’s bank. The card is a Maestro card issued by Deutsche Bank with a customer authentication server response time of p95=11s. Pod-A enters waitFor3dsCompletion().

t=10,001ms: GCP PubSub server checks M1’s ack deadline: 10 seconds have elapsed and no ack(), nack(), or ModifyAckDeadline has been received from any subscriber. GCP PubSub marks M1 as available for redelivery. GCP delivers M1 to Pod-B (round-robin among available subscribers).

t=10,050ms: Pod-B’s billing() function executes. UUID.randomUUID() → idem_B = "1a2b3c4d-5e6f-4789-abcd-012345678901". Stripe’s Payment Intent API is called with idem_B. Since idem_B ≠ idem_A, Stripe creates a new, independent PaymentIntent. The cardholder’s bank receives a second authentication challenge for a second Stripe PaymentIntent.

t=11,200ms: Pod-B’s 3DS completes (non-Maestro path, or bank cached response: 1.15s). Stripe commits the charge on Pod-B’s PaymentIntent: ch_B. Pod-B calls billingRecordRepo.save() — succeeds. Pod-B calls ack(). Pod-B’s processing is complete. From Pod-B’s perspective, billing succeeded normally.

t=12,800ms: The cardholder completes the 3DS challenge on their phone for Pod-A’s PaymentIntent. waitFor3dsCompletion() returns. Stripe commits the charge on Pod-A’s PaymentIntent: ch_A. Pod-A calls billingRecordRepo.save() — throws DataIntegrityViolationException (Pod-B already wrote the BillingRecord row with the unique constraint on (accountId, billingPeriod)). Pod-A’s call fails. Pod-A calls nack(). Pod-A retries (max-attempts=1, so no retries). The exception is logged.

Account acct_99 has been charged twice: ch_A and ch_B. The German cardholder received two 3DS challenges and two bank debit notifications. The team’s error log shows one DataIntegrityViolationException on Pod-A — but the team reads this as “a duplicate delivery was correctly rejected by the unique constraint,” not as “two separate Stripe charges were committed.”

The misconception

“The SCS GCP PubSub Binder wraps the raw com.google.cloud.pubsub.v1.Subscriber Java client. The raw Subscriber client automatically extends the ack deadline while the message receiver is processing — it calls ModifyAckDeadline every 33 seconds by default (one-third of the 100-second maximum extension period). When we migrated from the raw client to the SCS Binder, we assumed that behavior was preserved. The ackDeadlineSeconds=10 on our subscription doesn’t matter because the Subscriber keeps extending it. Our 3DS calls take 13 seconds and have always worked fine — they just seem to run a bit longer than 10 seconds without issue. The DataIntegrityViolationException in the logs is the unique constraint working correctly: a duplicate delivery was attempted and rejected.”

The raw Subscriber Java client does extend the ack deadline by default via its background AckDeadlineRenewer. The SCS GCP PubSub Binder configures the Subscriber via spring.cloud.gcp.pubsub.subscriber.max-ack-extension-period. When this property is 0 (the Spring Cloud GCP default), the Subscriber’s setMaxAckExtensionPeriod(Duration.ZERO) is set, which disables the AckDeadlineRenewer entirely. The migration from raw Subscriber client to SCS Binder silently dropped the deadline extension that made the 10-second deadline tolerable for 13-second 3DS flows. The DataIntegrityViolationException is the unique constraint blocking a second DB write — but the second Stripe charge was already committed before the DB write attempt. The unique constraint prevents a third charge if both pods retry, but it cannot undo the second charge that is already committed in Stripe.

The GCP PubSub-specific asymmetry: Azure Service Bus’s ServiceBusProcessorClient auto-renews the message lock by default and there is a single explicit property to configure the renewal duration (maxAutoLockRenewDuration). GCP PubSub’s equivalent behavior is split across two places: the subscription’s ackDeadlineSeconds (set in GCP infrastructure, not Spring config) and the Subscriber’s maxAckExtensionPeriod (set in Spring config, zero by default in the SCS GCP Binder). Developers who understand Azure Service Bus lock renewal can overlook the GCP Binder’s disabled-by-default ack extension.

The fix

Configure ack deadline extension explicitly and set the subscription’s ackDeadlineSeconds to at least twice the Stripe 3DS p99 latency:

# application.yml — configure ack deadline extension
spring:
  cloud:
    gcp:
      pubsub:
        subscriber:
          max-ack-extension-period: 600   # extend ack deadline for up to 10 minutes
          # This causes the underlying Subscriber client to send ModifyAckDeadline
          # requests every ~200s while the consumer function is executing,
          # preventing GCP from redelivering M1 to a second pod.

# Terraform — set subscription ackDeadlineSeconds > Stripe 3DS p99
resource "google_pubsub_subscription" "billing_processor" {
  name                 = "billing-processor"
  topic                = google_pubsub_topic.billing_events.id
  ack_deadline_seconds = 120   # > Stripe 3DS p99=90s, with 33% safety margin
}

And additionally, use a stable idempotency key. Even with ack deadline extension correctly configured, network partitions and pod restarts can cause concurrent delivery. A stable key is the only defense that works under all re-delivery scenarios:

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // Stable key: derived from immutable billing event fields
        String idempotencyKey = sha256Hex(
            event.getAccountId() + "|" +
            event.getBillingPeriod() + "|" +
            event.getAmountCents()
        );

        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            return;
        }

        PaymentIntent intent = stripeClient.paymentIntents().create(
            PaymentIntentCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("eur")
                .setCustomer(event.getStripeCustomerId())
                .setPaymentMethod(event.getPaymentMethodId())
                .setConfirm(true)
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        PaymentIntent confirmed = waitFor3dsCompletion(intent, Duration.ofSeconds(90));

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(), event.getBillingPeriod(),
            idempotencyKey, confirmed.getId()
        ));
    };
}

With a stable key, if Pod-A and Pod-B both process M1 concurrently (same idem_A for both), their concurrent Stripe calls carry identical idempotency keys. Stripe serializes concurrent requests with the same key: one call commits the charge, the other receives the cached response. No second charge is committed. The DB write that follows on Pod-B (the slower one) hits the DataIntegrityViolationException from the unique constraint and throws — but that exception is correctly handled as “already processed” since the DB already has the row from Pod-A’s successful write and the Stripe charge is committed exactly once.

Failure mode 3: deadLetterPolicy.maxDeliveryAttempts=5 × SCS max-attempts=3 — 15 Stripe charges committed before DLT routing; engineer reads x-goog-pubsub-delivery-attempt: 5 and re-ingests generating UUID_P committing ch_P

The setup

A team configures a dead-letter policy on their GCP PubSub subscription after a previous incident where a billing message cycled indefinitely (similar to Mode 1). The intent is to limit retries: after 5 delivery attempts, route the message to the dead-letter topic for manual review. The team configures both GCP infrastructure (via Terraform) and the SCS Binder:

# Terraform
resource "google_pubsub_subscription" "billing_processor" {
  name  = "billing-processor"
  topic = google_pubsub_topic.billing_events.id

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.billing_dlt.id
    max_delivery_attempts = 5   # Route to DLT after 5 GCP nacks
  }
}

# application.yml
spring:
  cloud:
    stream:
      bindings:
        billing-in-0:
          destination: billing-events
          group: billing-processor
          consumer:
            max-attempts: 3           # SCS-level retries per GCP delivery
            back-off-initial-interval: 500
            back-off-multiplier: 2.0

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // BUG: still using UUID.randomUUID() despite stable key recommendations
        String idempotencyKey = UUID.randomUUID().toString();

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(), event.getBillingPeriod(),
            idempotencyKey, charge.getId()
        ));
    };
}

The timeline

A billing event M1 for account acct_77, billing period 2024-11, amountCents=9900 arrives while PostgreSQL is experiencing a partial outage (writes to one shard are failing with PSQLException: FATAL: terminating connection due to administrator command).

GCP delivery 1 (t=0): SCS delivers M1 to billing().

• SCS attempt 1: UUID.randomUUID() → idem_A. Stripe commits ch_A. DB throws PSQLException.
• SCS attempt 2: idem_B. Stripe commits ch_B. DB throws.
• SCS attempt 3: idem_C. Stripe commits ch_C. DB throws. SCS exhausts max-attempts.
• Binder calls nack(). GCP delivery count: 1.

GCP delivery 2 (t=~35s):

• SCS attempt 1: idem_D. Stripe commits ch_D. DB throws.
• SCS attempt 2: idem_E. Stripe commits ch_E. DB throws.
• SCS attempt 3: idem_F. Stripe commits ch_F. DB throws.
• Binder calls nack(). GCP delivery count: 2.

GCP deliveries 3–5 (t=~95s, ~215s, ~455s):

• Each delivery: 3 SCS attempts → 3 Stripe charges (ch_G/H/I, ch_J/K/L, ch_M/N/O). DB throws each time. Binder calls nack().
• After delivery 5’s nack: GCP delivery count = 5 = maxDeliveryAttempts. GCP routes M1 to the dead-letter topic.

DLT routing (t=~8min): M1 is forwarded to the billing-dlt topic with GCP PubSub-added attributes: x-goog-pubsub-delivery-attempt: 5 and CloudPubSubDeadLetterSourceSubscription: billing-processor. The billing team receives a PagerDuty alert: DLT depth > 0.

Engineer review (t=~25min): The on-call engineer pulls M1 from the DLT subscription using the GCP PubSub Console. The engineer reads the message attributes: x-goog-pubsub-delivery-attempt: 5. The on-call runbook states: “The dead-letter attribute records how many times the billing consumer attempted to process this message. If the attempt count is 5 and the message is in the DLT, the consumer failed to process it 5 times. Review the consumer logs for the original failure reason, fix the underlying issue, then re-publish the message to billing-events to retry.” The PostgreSQL shard has recovered. The engineer re-publishes M1 to billing-events.

GCP delivery 6 (re-published): SCS receives the re-published M1. UUID.randomUUID() → idem_P. Stripe commits ch_P. PostgreSQL write succeeds. Consumer calls ack(). The incident is marked resolved. The billing record shows idem_P and ch_P.

Account acct_77 has been charged 16 times: ch_A through ch_O (15 charges from the 5 × 3 retry multiplication) plus ch_P (the re-ingested charge). Total: $1,584.00 for a $99.00 billing event. Stripe’s dashboard shows 16 charges, all successful, all with different idempotency keys.

The misconception

“The x-goog-pubsub-delivery-attempt: 5 attribute tells me the billing consumer tried to process this message exactly 5 times. Our billing consumer is an all-or-nothing operation: it either charges Stripe and writes to the DB, or it fails entirely. If it failed 5 times, the charge was never committed to Stripe — something upstream prevented the billing from completing. The message is safe to re-ingest because after 5 failed attempts, no money moved. I should find the root cause, fix it, and re-process the message.”

There are two compounding errors here. The first: x-goog-pubsub-delivery-attempt counts GCP-level delivery increments — each call to nack() from the SCS binder increments this counter by 1. The SCS binder calls nack() once per GCP delivery, after all SCS internal retries are exhausted. With max-attempts=3, each GCP delivery results in 3 consumer function invocations before the single nack(). A delivery_attempt of 5 means 5 SCS-retry-exhaustion cycles, each consisting of 3 consumer function invocations: 15 total executions. The second error: the consumer calls Stripe before writing to the DB. The DB failure is after the Stripe success. A consumer function “failing” in the SCS error model means the consumer function threw an exception — it does not mean Stripe was never called. Stripe was called, and returned 200, before the DB write that failed.

The GCP PubSub-specific angle that distinguishes this from equivalent binders: the Kafka Binder publishes to the DLT from within the SCS error handler after exhausting max-attempts. The Kafka DLT message carries kafka_exception-message and kafka_dlt-original-topic headers that record the exception at the SCS retry layer. There is no GCP-native equivalent of the Kafka DLT exception headers. The only GCP attribute is x-goog-pubsub-delivery-attempt, which tracks GCP-layer nacks. Engineers from Kafka backgrounds expect DLT headers to describe why the message failed at the consumer-function level. GCP’s DLT attribute describes why GCP stopped retrying. These are different views of the same event, and the mismatch causes the overcounting error.

The fix

Use a stable idempotency key. With a stable key, all 15 consumer function invocations across 5 GCP deliveries compute the same idempotency key string from SHA-256("acct_77|2024-11|9900"). Stripe’s cache returns ch_A for every subsequent call. When the engineer re-ingests from the DLT, the re-ingest also computes the same stable key → Stripe returns cached ch_A again. The DB write on re-ingest succeeds (PostgreSQL is recovered). One charge total.

Also update the on-call runbook to reflect the actual retry multiplication behavior:

@Bean
public Consumer<BillingEvent> billing() {
    return event -> {
        // Stable key — identical across all retries and GCP redeliveries
        String idempotencyKey = sha256Hex(
            event.getAccountId() + "|" +
            event.getBillingPeriod() + "|" +
            event.getAmountCents()
        );

        // Guard: idempotent early return if already committed
        if (billingRecordRepo.existsByAccountIdAndBillingPeriod(
                event.getAccountId(), event.getBillingPeriod())) {
            log.info("Billing already processed for account={} period={} — skipping",
                     event.getAccountId(), event.getBillingPeriod());
            return;
        }

        StripeCharge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setAmount(event.getAmountCents())
                .setCurrency("usd")
                .setCustomer(event.getStripeCustomerId())
                .build(),
            RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
        );

        billingRecordRepo.save(new BillingRecord(
            event.getAccountId(), event.getBillingPeriod(),
            idempotencyKey, charge.getId()
        ));
    };
}

With the stable key in place, a DLT message in the GCP Pub/Sub dead-letter topic represents a message where the billing consumer ran maxDeliveryAttempts × max-attempts times and still failed at the DB write every time (while Stripe committed the same charge on the first attempt and cached it for all subsequent attempts). When you re-ingest from the DLT, the DB write succeeds on the first attempt with the same stable key → no new Stripe charge → BillingRecord written with the original ch_A charge ID. The DLT becomes a safe recovery mechanism rather than a double-charge trigger.

Cross-layer comparison: raw Google Cloud Pub/Sub client, SCS GCP PubSub Binder, Spring Integration Google PubSub, SCS Kafka

Diagnostic checklist: 8 questions before deploying a GCP Pub/Sub-backed billing consumer

1. Is your Stripe idempotency key derived from the billing event’s immutable content fields (account ID, billing period, amount cents), or from UUID.randomUUID()? If the latter, every SCS retry, GCP redelivery, and DLT re-ingestion will generate a new key and a new Stripe charge. This is Mode 1, Mode 2, and Mode 3 simultaneously if your consumer has all three configurations in play.
2. Is spring.cloud.gcp.pubsub.subscriber.max-ack-extension-period explicitly set to a value greater than the maximum expected processing time of your billing consumer? If this property is absent or zero, the SCS GCP PubSub Binder does not extend the ack deadline. For billing flows that include Stripe 3DS (p99 up to 120s for some EU issuers), set this to at least 300 seconds. Verify by checking the Spring Cloud GCP effective configuration at startup.
3. Is your GCP subscription ackDeadlineSeconds set to at least twice your Stripe operation p99 latency? Even with max-ack-extension-period configured, the initial ack deadline determines the window before the first extension is needed. A 10-second initial deadline requires an extension request within the first few seconds. Setting the subscription deadline to 120 seconds provides a full safety margin for Stripe 3DS without requiring extension infrastructure to fire on the first request.
4. Do you have a dead-letter policy configured on your GCP subscription? Without one, a billing message that fails every SCS retry cycle bounces indefinitely for up to 7 days (the default messageRetentionDuration), generating a fresh Stripe charge attempt on every GCP redelivery. Configure maxDeliveryAttempts to bound the bounce and trigger a DLT alert.
5. Does your on-call runbook correctly describe the x-goog-pubsub-delivery-attempt multiplier? If the runbook says “delivery_attempt=N means the billing consumer tried N times,” update it to read “delivery_attempt=N means GCP saw N nacks from SCS retry exhaustion; actual consumer invocations = N × max-attempts.” Update the runbook section on DLT re-ingestion to note that re-ingestion is only safe if the consumer uses a stable idempotency key.
6. Is your pre-flight guard based on immutable business keys (accountId, billingPeriod) with a DDL-level unique constraint, not just an application-level existsByX() check? Under concurrent GCP delivery (Mode 2), two pods can pass the application-level check simultaneously before either completes the DB write. The DDL constraint ensures one insert wins and one throws DataIntegrityViolationException. With a stable Stripe idempotency key, the concurrent Stripe calls both resolve to the same charge — only the DB write races.
7. Is nackOnError set to true (default) intentionally? nackOnError=false causes the SCS GCP PubSub Binder to call ack() even when the consumer function throws — the message is dropped rather than retried. This eliminates the Mode 1 bounce entirely, at the cost of potentially dropping billing events that fail due to transient infrastructure issues. If you choose nackOnError=false, implement an outbox pattern so billing events are not lost when the consumer function throws before completing.
8. Have you tested the behavior of your billing consumer when the same GCP message is delivered concurrently to two pods (simulated by publishing M1 and manually pulling it twice from the same subscription before either consumer calls ack)? The correct behavior with stable keys is: both Stripe calls succeed with the same charge ID; one DB write succeeds; one DB write throws DataIntegrityViolationException; the pod that threw calls nack() but the other pod’s ack() has already committed the message; GCP PubSub deduplicates the ack. Verify this in a Testcontainers integration test before deploying.

Tests: JUnit 5 + WireMock + Testcontainers + GCP PubSub emulator

The following tests use the Google Cloud Pub/Sub emulator (available as a Testcontainers module via com.google.cloud:google-cloud-pubsub test support, or the gcr.io/google.com/cloudsdktool/cloud-sdk:emulators image), WireMock for Stripe, and JUnit 5. Each test isolates one failure mode and verifies both the broken behavior (without stable key) and the fixed behavior (with stable key).

@SpringBootTest
@Testcontainers
class GcpPubSubStripeIdempotencyTest {

    @Container
    static GenericContainer<?> pubsubEmulator =
        new GenericContainer<>("gcr.io/google.com/cloudsdktool/cloud-sdk:latest")
            .withCommand("gcloud beta emulators pubsub start --host-port=0.0.0.0:8085")
            .withExposedPorts(8085);

    @Container
    static WireMockContainer wireMock =
        new WireMockContainer("wiremock/wiremock:3.x");

    @Autowired BillingRecordRepository billingRecordRepo;
    @Autowired PubSubTemplate pubSubTemplate;
    @Autowired PubSubAdmin pubSubAdmin;

    static final String BILLING_TOPIC = "billing-events";
    static final String BILLING_SUBSCRIPTION = "billing-processor";
    static final String BILLING_DLT = "billing-dlt";

    // ── Mode 1: nackOnError=true + max-attempts=3 → multiple Stripe charges ──

    @Test
    @DisplayName("Mode 1: GCP redelivery after max-attempts nack → UUID_D → ch_D (without stable key)")
    void nackRedelivery_withoutStableKey_chargesOnEachRedelivery() throws Exception {
        // Arrange: Stripe accepts any key
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200)
                .withBody(chargeJson("ch_NEW"))));

        // Simulate DB failure on first 6 attempts, success on 7th
        AtomicInteger saveAttempt = new AtomicInteger(0);
        when(billingRecordRepo.save(any())).thenAnswer(inv -> {
            if (saveAttempt.incrementAndGet() < 7) {
                throw new CannotGetJdbcConnectionException("pool exhausted");
            }
            return inv.getArgument(0);
        });

        BillingEvent event = new BillingEvent("acct_99", "2024-09", 4900L);
        pubSubTemplate.publish(BILLING_TOPIC, event);

        // Wait for 7th attempt to succeed (2 GCP deliveries × 3 SCS + 1 GCP delivery × 1)
        await().atMost(Duration.ofMinutes(2))
               .until(() -> billingRecordRepo.count() == 1);

        // Assert: 7 Stripe calls with 7 different keys
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSizeGreaterThanOrEqualTo(7);
        Set<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(keys).hasSizeGreaterThanOrEqualTo(7); // different key each invocation
    }

    @Test
    @DisplayName("Mode 1: GCP redelivery after max-attempts nack → stable key → one charge (with fix)")
    void nackRedelivery_withStableKey_deduplicates() throws Exception {
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200).withBody(chargeJson("ch_A"))));

        AtomicInteger saveAttempt = new AtomicInteger(0);
        when(billingRecordRepo.save(any())).thenAnswer(inv -> {
            if (saveAttempt.incrementAndGet() < 7) {
                throw new CannotGetJdbcConnectionException("pool exhausted");
            }
            return inv.getArgument(0);
        });

        BillingEvent event = new BillingEvent("acct_99", "2024-09", 4900L);
        pubSubTemplate.publish(BILLING_TOPIC, event);

        await().atMost(Duration.ofMinutes(2))
               .until(() -> billingRecordRepo.count() == 1);

        // Assert: all Stripe calls used the same stable key
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        Set<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(keys).hasSize(1); // one stable key → Stripe cache returns ch_A each time
    }

    // ── Mode 2: ackDeadline expiry during 3DS → dual delivery ─────────────────

    @Test
    @DisplayName("Mode 2: concurrent delivery after ackDeadline expiry → UUID_B → ch_B (without stable key)")
    void ackDeadlineExpiry_withoutStableKey_dualCharges() throws Exception {
        // Arrange: Stripe takes 12s (longer than ackDeadline=10s) for first call
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .inScenario("3ds-flow")
            .whenScenarioStateIs(STARTED)
            .willReturn(aResponse()
                .withFixedDelay(12_000)    // 12s simulates Stripe 3DS
                .withStatus(200)
                .withBody(chargeJson("ch_A")))
            .willSetStateTo("second-delivery"));

        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .inScenario("3ds-flow")
            .whenScenarioStateIs("second-delivery")
            .willReturn(aResponse().withStatus(200).withBody(chargeJson("ch_B"))));

        BillingEvent event = new BillingEvent("acct_88", "2024-09", 9900L);
        pubSubTemplate.publish(BILLING_TOPIC, event);

        // Allow time for both deliveries and ackDeadline expiry
        await().atMost(Duration.ofSeconds(30))
               .until(() -> wireMock.findAll(
                   postRequestedFor(urlEqualTo("/v1/charges"))).size() >= 2);

        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSize(2);
        Set<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(keys).hasSize(2); // 2 different keys = 2 charges
    }

    // ── Mode 3: DLT deadLetterPolicy × SCS max-attempts ───────────────────────

    @Test
    @DisplayName("Mode 3: maxDeliveryAttempts=5 × max-attempts=3 → 15 Stripe calls; re-ingest → 16th")
    void dltReingestion_withoutStableKey_chargesMultipleTimes() throws Exception {
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200).withBody(chargeJson("ch_NEW"))));

        // Simulate DB always failing (outage scenario)
        when(billingRecordRepo.save(any()))
            .thenThrow(new CannotGetJdbcConnectionException("outage"));

        BillingEvent event = new BillingEvent("acct_77", "2024-11", 9900L);
        pubSubTemplate.publish(BILLING_TOPIC, event);

        // Wait for message to appear in DLT (5 GCP deliveries × 3 SCS retries)
        await().atMost(Duration.ofMinutes(5))
               .until(() -> pubSubAdmin.getSubscriptionMessageCount(
                   "billing-dlt-subscription") > 0);

        // Assert: 15 Stripe calls were made (5 × 3)
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSize(15);

        // Simulate DB recovery and engineer re-ingesting from DLT
        reset(billingRecordRepo);
        when(billingRecordRepo.save(any())).thenReturn(mock(BillingRecord.class));

        PubsubMessage dltMessage = pubSubTemplate.pull(
            "billing-dlt-subscription", 1, false).get(0);
        pubSubTemplate.publish(BILLING_TOPIC, dltMessage.getData());

        await().atMost(Duration.ofSeconds(10))
               .until(() -> billingRecordRepo.count() == 1);

        // Assert: 16th Stripe call with a new key (re-ingest without stable key)
        assertThat(wireMock.findAll(postRequestedFor(urlEqualTo("/v1/charges")))).hasSize(16);
    }

    @Test
    @DisplayName("Mode 3: stable key → all 16 calls return same ch_A → one charge (with fix)")
    void dltReingestion_withStableKey_onlyOneCharge() throws Exception {
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse().withStatus(200).withBody(chargeJson("ch_A"))));

        AtomicInteger saveAttempt = new AtomicInteger(0);
        when(billingRecordRepo.save(any())).thenAnswer(inv -> {
            if (saveAttempt.incrementAndGet() < 16) {
                throw new CannotGetJdbcConnectionException("outage");
            }
            return inv.getArgument(0);
        });

        BillingEvent event = new BillingEvent("acct_77", "2024-11", 9900L);
        pubSubTemplate.publish(BILLING_TOPIC, event);

        await().atMost(Duration.ofMinutes(5))
               .until(() -> billingRecordRepo.count() == 1);

        // Assert: all 16 calls used the same stable key → Stripe returned ch_A each time
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            postRequestedFor(urlEqualTo("/v1/charges")));
        Set<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(keys).hasSize(1); // one stable key → ch_A returned from cache × 15
    }

    private String chargeJson(String id) {
        return "{\"id\":\"" + id + "\",\"object\":\"charge\",\"status\":\"succeeded\"}";
    }
}