Spring Cloud Stream RabbitMQ Stripe Integration: How the SCS RabbitMQ Binder Dead-Letter Queue x-death Header Misleads Engineers Into Re-Ingesting Already-Charged Messages Generating UUID_B Committing ch_B, How rabbitmqctl close_connection During Maintenance Requeues the Unacked In-Flight Billing Message After ch_A Commits, and How basicRecover(true) from an Admin Endpoint Requeues All In-Flight Channel Messages Including Post-Stripe-Commit Records
This post is part of the ongoing series on Stripe idempotency failure modes in JVM frameworks. Earlier entries covered Spring Cloud Stream (SCS maxAttempts=3 retry lambda re-invocation, Kafka Binder BATCH ackMode crash-mid-batch redelivery, and RabbitMQ Binder republishToDlq reprocessing with a fresh UUID), Spring Cloud Stream Binder (header-mode=none silently dropping the x-idempotency-key header, RabbitMQ Binder transacted=true basicNack redelivery after Stripe commits ch_A, and Kafka Binder autoAddPartitions partition-creation rebalance replaying uncommitted offsets), Spring Cloud Stream Kafka (enableDlq=true DLT re-ingestion misread as pre-Stripe failure, max.poll.interval.ms consumer group eviction redelivery, and resetOffsets=true full topic replay), and several others covering Spring Kafka, Spring AMQP, Spring Retry, Spring Batch, and Spring Integration failure modes. This post documents three additional failure modes that are specific to the SCS RabbitMQ Binder and arise from RabbitMQ broker-level operational procedures — dead-letter queue management, AMQP connection lifecycle operations, and channel-level recovery — rather than from consumer function code or binder configuration settings.
The three failure modes are: a developer who enables the SCS RabbitMQ Binder dead-letter queue automation via spring.cloud.stream.rabbit.bindings.billing-in-0.consumer.autoBindDlq=true, which causes the binder to declare a dead-letter exchange (billing.dlq), configure the billing queue with x-dead-letter-exchange: billing.dlq, and automatically route exhausted-retry messages to the dead-letter queue with an x-death header array; when an on-call engineer reads the x-death[0].reason field on a message in billing.dlq and sees rejected, the engineer interprets this as an explicit consumer rejection without processing (a dead message that never reached the billing logic), concludes that Stripe was never called, and re-ingests the message to the billing exchange via the RabbitMQ Management Console Move Messages feature — the re-ingested consumer generates UUID.randomUUID() and Stripe commits ch_B alongside the already-committed ch_A; a developer who deploys the BillingService with the SCS RabbitMQ Binder using default acknowledgeMode=AUTO, and whose production RabbitMQ connections are rotated periodically by an SRE using rabbitmqctl close_connection <conn-name> "TLS cert rotation", whereby RabbitMQ immediately closes the AMQP connection and requeues all messages that were in the unacknowledged state on that connection’s channels at the moment of closure, including the billing message for which Stripe had already committed ch_A at t=500ms but billingRecordRepo.save() had not yet returned, so that the SCS binder’s SimpleMessageListenerContainer reconnects and the requeued message is redelivered with redelivered=true, and the consumer generates UUID.randomUUID() and Stripe commits ch_B; and a developer who builds a custom Spring Boot Actuator endpoint documented in the team runbook as a “stuck message recovery” tool, which calls channel.basicRecover(true) on the shared AMQP channel, not realizing that basicRecover(requeue=true) is a channel-wide operation that instructs RabbitMQ to resend all messages currently in the unacknowledged state on the channel — not just the message that appears to be stuck — so that any billing message for which Stripe committed a charge but basicAck had not yet been sent is immediately redelivered, and the consumer generates UUID.randomUUID() and Stripe commits ch_B for every such message.
What makes these three RabbitMQ operational failure modes particularly dangerous is the degree to which each one is taken in good faith. Dead-letter queue re-ingestion is the documented recovery path for failed messages. Connection close is a standard maintenance procedure for certificate rotation, connection pool management, and broker-side connection cleanup. The basicRecover endpoint appears to be a sensible operational lever for clearing consumer backlogs. None of the three operations generates a warning, a Stripe error response, or an application log entry that would alert the operator that a charge has already been committed. The first sign of a problem in all three modes is typically an angry customer email, a Stripe dispute notification, or a billing team Slack alert about unexpected charge.succeeded webhook events.
Stack for examples: Spring Boot 3.x, Spring Cloud Stream 4.x (spring-cloud-starter-stream-rabbit), Spring AMQP 3.x (spring-rabbit), RabbitMQ 3.12+, PostgreSQL 16, Stripe Java SDK v24+, JUnit 5, Testcontainers 1.20+, WireMock 3.x.
Background: three categories of SCS RabbitMQ Binder Stripe failure modes
The failure modes in this series fall into three categories when examined against the SCS RabbitMQ Binder. The first category is consumer function code failures: the Consumer<T> lambda generates an unstable idempotency key (UUID.randomUUID()) and SCS retry or binder redelivery causes a fresh invocation. The original SCS post covered maxAttempts=3 retry and republishToDlq in this category. The second category is binder configuration failures: binder-layer settings alter how messages arrive at the consumer — transaction scope, header extraction, routing — documented in the SCS Binder post (covering transacted=true basicNack redelivery and header-mode=none key extraction). The third category — the subject of this post — is RabbitMQ operational failures: broker-side procedures performed by SREs, on-call engineers, or automated tooling that interact with the AMQP protocol in ways that produce message redelivery without the consumer receiving any signal that the billing request has already been processed.
The RabbitMQ-operational category is the hardest to defend against purely by code review because these operations are performed by people (or automated jobs) who may have no visibility into the current billing processing state. An SRE closing a connection for TLS rotation has no way to know that connection hosts an AMQP channel with an unacknowledged billing message at t=500ms post-Stripe-commit. An on-call engineer reading a RabbitMQ Management Console DLQ message sees metadata headers, not a list of external API calls that were made on behalf of that message. The operator performing basicRecover to clear a stuck consumer may be following a runbook written before the billing service was integrated with Stripe. The correct defense against all three failure modes is identical to every other mode in this series: a stable content-hash idempotency key (SHA-256(subscriptionId + billingPeriod + amountCents)) computed once before any external call, stored durably in a BillingRecord row, combined with a guard that reads the BillingRecord before calling Stripe. These two patterns together make the consumer function naturally idempotent regardless of why the redelivery occurred.
Mode 1: SCS RabbitMQ Binder autoBindDlq=true dead-letter queue — x-death header reason=rejected misread as pre-Stripe consumer rejection — re-ingestion generates UUID_B — ch_B
How the SCS RabbitMQ Binder dead-letter queue feature works
The SCS RabbitMQ Binder provides a dead-letter queue automation feature controlled by the spring.cloud.stream.rabbit.bindings.<binding-name>.consumer.autoBindDlq property. When set to true, the binder declares a dead-letter exchange (by default named <destination>.dlq — so billing.dlq for a destination named billing) and configures the main billing queue with two arguments: x-dead-letter-exchange: billing.dlq and x-dead-letter-routing-key: billing. Messages that are rejected by the consumer (via basicReject(false) or basicNack(false)) or that expire due to TTL are automatically routed to billing.dlq by the RabbitMQ broker.
When the SCS RabbitMQ Binder consumer exhausts its maxAttempts retry sequence without the message being processed successfully, the binder calls basicReject(deliveryTag, false) on the AMQP channel with requeue=false. This is the AMQP protocol signal that triggers RabbitMQ dead-letter routing: the broker moves the message from the billing queue to the billing.dlq queue and appends an x-death entry to the message’s headers. The x-death header is an AMQP array of maps; each entry records one dead-letter event with the following fields: exchange (the original exchange), queue (the queue the message was dead-lettered from), routing-keys (array of routing keys), reason (why it was dead-lettered: rejected, expired, maxlen, or delivery-limit), count (number of times this entry has been incremented), and time (timestamp of the dead-letter event).
The reason field value for SCS-driven dead-lettering is always rejected, because the SCS binder issues a basicReject(false) call. This is the same reason value that appears when a consumer explicitly calls basicReject to signal that a message is malformed and should not be retried. The AMQP protocol does not distinguish between “consumer rejected because the message was malformed” and “consumer rejected because it exhausted its retry limit after partially processing the message (including a successful Stripe call)”. Both result in x-death[0].reason=rejected in the dead-letter queue.
The failure scenario
The BillingService is configured with autoBindDlq=true, maxAttempts=3, and requeueRejected=false. The consumer function processes a BillingRequestedEvent for subscription sub_Abc123, billing period 2026-10, amount 9900 cents. The consumer generates UUID.randomUUID() as the idempotency key, calls stripe.charges.create() with that key, and Stripe commits ch_A at t=500ms. At t=620ms, billingRecordRepo.save() throws a DataIntegrityViolationException because the billing_records table already has a row for (sub_Abc123, 2026-10) from a partially successful prior session. SCS catches the exception and retries: attempt 2 generates UUID_B (ch_B committed) and also fails at billingRecordRepo.save(); attempt 3 generates UUID_C (ch_C committed) and also fails. After three failed attempts, the SCS binder calls basicReject(deliveryTag, false) and RabbitMQ routes the message to billing.dlq with the following x-death header (as seen in the RabbitMQ Management Console’s message inspector or via rabbitmqadmin get queue=billing.dlq ackmode=ack_requeue_false):
x-death: [
{
"count": 1,
"exchange": "billing",
"queue": "billing",
"reason": "rejected",
"routing-keys": ["billing"],
"time": "2026-10-24 04:12:38+00:00"
}
]
x-exception-message: could not execute statement [ERROR: duplicate key value violates unique
constraint "billing_records_sub_period_key"]
x-exception-stacktrace: org.springframework.dao.DataIntegrityViolationException: ...
at com.example.billing.BillingFunctions.lambda$billingConsumer$0(BillingFunctions.java:52)
x-original-deliveryTag: 142An on-call engineer receives a PagerDuty alert about messages accumulating in billing.dlq. The engineer opens the RabbitMQ Management Console, navigates to the billing.dlq queue, and uses the “Get messages” feature to inspect the first message. The engineer reads the x-death header and sees reason: rejected. The engineer also sees x-exception-stacktrace containing a DataIntegrityViolationException at the top frame with a stack trace pointing to billingRecordRepo.save().
Developer misconception: “The
x-death[0].reasonvalue isrejected. In RabbitMQ, a message is dead-lettered with reasonrejectedwhen the consumer explicitly callsbasicRejectto refuse the message. This means the consumer rejected the message without processing it — it decided this message was not handleable. Thex-exception-stacktraceshows aDataIntegrityViolationExceptionfrom the database layer, which confirms the consumer tried to save to the DB but failed before reaching the Stripe call. I should move this message back to thebillingexchange so the consumer can try again with a clean state.”
The engineer uses the RabbitMQ Management Console’s “Move messages” feature (backed by the Shovel plugin) to re-route the DLQ message back to the billing exchange. The BillingService consumer processes the re-ingested message, generates a new UUID.randomUUID() (because the stable key pattern is not implemented), and calls stripe.charges.create() with the new key. Stripe has never seen this UUID before and commits ch_D (the fourth distinct charge for this billing request — ch_A, ch_B, ch_C having been committed on the three maxAttempts invocations before dead-lettering). The customer has now been charged four times for the same billing period.
Why the misconception is wrong
The AMQP reason=rejected value is set whenever basicReject(false) or basicNack(false) is called on the AMQP channel with requeue=false, regardless of the processing state of the message at the time of rejection. The SCS RabbitMQ Binder calls basicReject(false) after all maxAttempts retries are exhausted — but by the time the binder issues this call, it has already invoked the consumer lambda three times. All three invocations called Stripe. All three Stripe calls succeeded (or at least committed a charge) before the DataIntegrityViolationException was thrown. The reason=rejected header entry records that the binder rejected the message from the queue, not that the consumer rejected the message’s content before doing any work.
The AMQP protocol has no concept of “consumer rejected because it was malformed” versus “consumer rejected because it failed after partial processing”. Both produce reason=rejected. The distinction that matters operationally — whether external effects (Stripe charges, emails, webhooks) have already been applied — is invisible to the RabbitMQ protocol and to the Management Console. An engineer reading only the x-death header cannot determine whether Stripe was called. The only authoritative source is the application-level billing_records table and the Stripe charge list for the subscription.
The x-exception-stacktrace header shows only the last exception that caused the binder to exhaust retries, and only the stack frames captured at the time of that exception. On attempt 1, the exception was thrown from billingRecordRepo.save() at line 52 — which is a line that appears after the Stripe call in the method body. The Stripe call at line 44 completed successfully and did not throw an exception on attempt 1, so there is no Stripe frame visible in the stack trace. A developer reading only the top exception frame cannot distinguish “exception was thrown before any Stripe interaction” from “exception was thrown after Stripe committed a charge”. This is the same misconception documented for the SCS Kafka Binder DLT re-ingestion failure mode, but with a different RabbitMQ-specific trigger: the x-death.reason=rejected field adds an additional layer of false confidence that the consumer explicitly rejected the message without processing it.
The buggy consumer configuration
# BillingService application.yml — BUGGY.
# autoBindDlq=true routes exhausted-retry messages to billing.dlq.
# The x-death header reason=rejected field is populated by the SCS binder
# calling basicReject(false) AFTER all maxAttempts are exhausted — which means
# AFTER Stripe was called (and committed ch_A, ch_B, ch_C) on each attempt.
# An engineer reading reason=rejected in the DLQ will incorrectly conclude
# the consumer rejected the message before processing it.
spring:
cloud:
stream:
bindings:
billing-in-0:
destination: billing
group: billing-service
consumer:
maxAttempts: 3
backOffInitialInterval: 1000
backOffMaxInterval: 10000
backOffMultiplier: 2.0
rabbit:
bindings:
billing-in-0:
consumer:
autoBindDlq: true
# requeueRejected=false is correct — prevents infinite redelivery loop.
# But autoBindDlq=true without a stable idempotency key means:
# 1. Stripe is called on each of the 3 maxAttempts before dead-lettering.
# 2. The DLQ message x-death.reason=rejected misleads the on-call engineer.
# 3. Manual re-ingestion calls Stripe a 4th time.
requeueRejected: false// BillingService BillingFunctions.java — BUGGY.
// UUID.randomUUID() inside the Consumer lambda generates a new key per invocation.
// SCS retries (maxAttempts=3) generate UUID_A, UUID_B, UUID_C — three Stripe charges.
// Manual DLQ re-ingestion generates UUID_D — a fourth Stripe charge.
@Configuration
public class BillingFunctions {
@Bean
public Consumer<BillingRequestedEvent> billingConsumer(
StripeClient stripe,
BillingRecordRepository billingRecordRepo) {
return event -> {
// BUG: UUID.randomUUID() generates a new key on every Consumer invocation:
// attempt 1 (UUID_A → ch_A), attempt 2 (UUID_B → ch_B),
// attempt 3 (UUID_C → ch_C), DLQ re-ingestion (UUID_D → ch_D).
String idempotencyKey = UUID.randomUUID().toString();
ChargeCreateParams params = ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("usd")
.setCustomer(event.getCustomerId())
.build();
// Stripe commits ch_A at t=500ms on attempt 1.
// Stripe commits ch_B at t=500ms on attempt 2.
// Stripe commits ch_C at t=500ms on attempt 3.
// These calls succeed — no Stripe exception thrown.
Charge charge = stripe.charges().create(params,
RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());
// DataIntegrityViolationException thrown here on all 3 attempts
// (billing_records row already exists from a prior partial session).
// The exception is visible in x-exception-stacktrace.
// The Stripe calls above are NOT visible in the stack trace.
billingRecordRepo.save(new BillingRecord(
event.getSubscriptionId(),
event.getBillingPeriod(),
charge.getId(),
idempotencyKey));
};
}
}The fixed consumer: stable key + guard before Stripe
// BillingFunctions.java — FIXED.
// SHA-256(subscriptionId + billingPeriod + amountCents) produces the same
// idempotency key on every invocation for the same billing request — including
// all maxAttempts retries and any DLQ re-ingestion.
// BillingRecord guard checks for an existing charge before calling Stripe.
// DLQ re-ingestion is then safe: the guard returns early on re-ingestion.
@Configuration
public class BillingFunctions {
@Bean
public Consumer<BillingRequestedEvent> billingConsumer(
StripeClient stripe,
BillingRecordRepository billingRecordRepo) {
return event -> {
// FIX: Stable key derived from billing intent fields.
// Same value on attempt 1, 2, 3, and on DLQ re-ingestion.
String idempotencyKey = stableKey(
event.getSubscriptionId(),
event.getBillingPeriod(),
event.getAmountCents());
// FIX: Guard before Stripe — read-before-write.
// On DLQ re-ingestion: BillingRecord exists (from attempt 1's DB
// write? No — attempt 1 also failed the DB write). But: if Stripe
// returns the SAME charge for the stable key (idempotency cache hit),
// and we re-derive the key from the event, the guard here would not
// find a BillingRecord (because the DB write failed on all 3 attempts).
// Therefore the guard must ALSO check Stripe's idempotency: if
// billingRecordRepo is empty for this key, check Stripe for a prior
// charge using the stable idempotency key BEFORE issuing a new one.
//
// Better pattern: save the BillingRecord BEFORE calling Stripe,
// using the stable key as a unique natural key. If save fails with
// a unique-constraint violation, the BillingRecord already exists —
// read it back, return the existing charge ID. This avoids the
// Stripe-then-DB ordering problem entirely.
Optional<BillingRecord> existing =
billingRecordRepo.findByIdempotencyKey(idempotencyKey);
if (existing.isPresent()) {
return; // Already charged; DLQ re-ingestion or redelivery.
}
ChargeCreateParams params = ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("usd")
.setCustomer(event.getCustomerId())
.build();
Charge charge = stripe.charges().create(params,
RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());
billingRecordRepo.save(new BillingRecord(
event.getSubscriptionId(),
event.getBillingPeriod(),
charge.getId(),
idempotencyKey));
};
}
private static String stableKey(
String subscriptionId, String billingPeriod, long amountCents) {
try {
MessageDigest sha256 = MessageDigest.getInstance("SHA-256");
String input = subscriptionId + "|" + billingPeriod + "|" + amountCents;
byte[] hash = sha256.digest(input.getBytes(StandardCharsets.UTF_8));
return HexFormat.of().formatHex(hash);
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException(e);
}
}
}Supplementary fix: annotate DLQ messages with processing metadata
Even with a stable key and a guard, it is good operational practice to include a custom message header that records whether Stripe was called before the message was dead-lettered. The SCS RabbitMQ Binder allows a DlqMessageDeliveryMode or a custom MessagePostProcessor on the DLQ binding to enrich the dead-letter message with additional headers before it reaches the DLQ. A header named x-stripe-charge-id set to the Stripe charge ID (or empty if Stripe was never called) would allow an on-call engineer to definitively determine processing state from the DLQ message alone, without needing to query the billing_records table or the Stripe API.
# BillingService application.yml — FIXED (binder config).
# deadLetterQueueName and deadLetterExchange are optional but useful for
# giving the DLQ a distinct, inspectable name.
# republishToDlq: true is used instead of autoBindDlq when you need to
# add custom headers to the dead-lettered message via DlqPublisher.
spring:
cloud:
stream:
bindings:
billing-in-0:
destination: billing
group: billing-service
consumer:
maxAttempts: 3
backOffInitialInterval: 1000
backOffMaxInterval: 10000
backOffMultiplier: 2.0
rabbit:
bindings:
billing-in-0:
consumer:
autoBindDlq: true
requeueRejected: false
# dlqDeadLetterExchange and dlqDeadLetterRoutingKey allow
# chaining DLQs for observability pipelines.
dlqTtl: 86400000 # DLQ messages expire after 24h to prevent unbounded growth.Test: DLQ re-ingestion is idempotent with a stable key
@SpringBootTest
@Testcontainers
class DlqReIngestionIdempotencyTest {
@Container
static RabbitMQContainer rabbitMQ = new RabbitMQContainer("rabbitmq:3.12-management");
@RegisterExtension
static WireMockExtension wm = WireMockExtension.newInstance()
.options(wireMockConfig().dynamicPort())
.build();
@Autowired
private StreamBridge streamBridge;
@Autowired
private BillingRecordRepository billingRecordRepo;
@Test
void dlqReIngestionDoesNotProduceDuplicateCharge() throws Exception {
// Stripe stub: always returns ch_A for any idempotency key.
// (With a stable key, the same key is sent on all attempts.)
wm.stubFor(post(urlPathEqualTo("/v1/charges"))
.willReturn(aResponse()
.withStatus(200)
.withBody("""
{"id":"ch_A","object":"charge","amount":9900,
"currency":"usd","status":"succeeded"}
""")));
BillingRequestedEvent event = new BillingRequestedEvent(
"sub_Abc123", "2026-10", 9900L, "cus_X");
// First delivery — BillingRecord saved.
streamBridge.send("billing-out-0", event);
await().atMost(Duration.ofSeconds(10))
.until(() -> billingRecordRepo.existsBySubscriptionIdAndBillingPeriod(
"sub_Abc123", "2026-10"));
// Simulate DLQ re-ingestion: same event published again by engineer.
streamBridge.send("billing-out-0", event);
Thread.sleep(2000);
// Stripe was called exactly once — stable key → guard blocks re-ingestion.
wm.verify(exactly(1), postRequestedFor(urlPathEqualTo("/v1/charges")));
assertThat(billingRecordRepo.countBySubscriptionIdAndBillingPeriod(
"sub_Abc123", "2026-10")).isEqualTo(1);
}
}Mode 2: rabbitmqctl close_connection during TLS certificate rotation requeues unacked billing message after ch_A commits — UUID_B — ch_B
How AMQP connection closure affects unacknowledged messages
AMQP 0-9-1 (the protocol used by RabbitMQ) defines strict semantics for message acknowledgment: a message delivered to a consumer via basicDeliver remains in the unacknowledged state until the consumer sends basicAck, basicNack, or basicReject on the same channel. While a message is in the unacknowledged state, RabbitMQ considers it “in flight” — it has been removed from the queue’s ready state but has not yet been confirmed as processed. If the AMQP channel or connection is closed before the consumer sends an acknowledgment, RabbitMQ applies the following rule: all messages that were delivered on that channel but not yet acknowledged are immediately requeued to the source queue in their original position (at the head of the queue, before any other ready messages). This behavior is mandated by the AMQP specification and cannot be configured away for standard queues.
RabbitMQ’s rabbitmqctl close_connection command (and its equivalent HTTP API call DELETE /api/connections/<conn-name>) closes an AMQP connection immediately, without coordinating with the consumer application. The consumer application receives a TCP RST or FIN and the Spring AMQP SimpleMessageListenerContainer detects the connection failure via its ShutdownSignalException handler. The SCS RabbitMQ Binder, which wraps SimpleMessageListenerContainer, automatically attempts to reconnect using its configured recoveryInterval (default 5 seconds). Once the reconnection succeeds, the binder registers a new consumer on the billing queue. RabbitMQ delivers the requeued messages (those that were unacknowledged when the connection was closed) to the new consumer via basicDeliver with the redelivered flag set to true.
The redelivery happens within milliseconds to seconds of the connection close, depending on the broker’s requeue backlog. From the application’s perspective, it looks identical to any other message delivery — the Consumer<T> lambda receives a BillingRequestedEvent with the same payload as the original delivery. The only AMQP-protocol difference is the redelivered=true flag on the basicDeliver frame, which the SCS binder exposes as a AmqpHeaders.REDELIVERED message header. Most consumer functions do not check this header.
The failure scenario
The BillingService is deployed on Kubernetes with the SCS RabbitMQ Binder using default acknowledgeMode=AUTO (which in Spring AMQP means AUTO acknowledge mode: the channel sends basicAck after the consumer method returns normally, or basicNack if the method throws an exception). The service processes billing messages from the billing queue using a single consumer with concurrency=1 and prefetch=250.
At t=0ms, the consumer receives a BillingRequestedEvent for subscription sub_Abc123. The consumer generates UUID.randomUUID() as the idempotency key and calls stripe.charges.create(). At t=500ms, Stripe commits ch_A and returns a successful response. At t=510ms, the consumer calls billingRecordRepo.save() to persist the billing record. While the PostgreSQL write is in progress (t=510ms to t=640ms), an SRE runs the following command as part of a scheduled TLS certificate rotation:
# SRE runs this on the RabbitMQ node during scheduled TLS rotation.
# The connection name includes the IP and port of the billing service pod.
# The SRE identifies the connection by listing: rabbitmqctl list_connections name
rabbitmqctl close_connection \
"172.16.1.42:54322 -> 172.16.1.10:5672" \
"TLS cert rotation - closing old connections"RabbitMQ closes the AMQP connection at t=590ms — 90ms after Stripe committed ch_A but 50ms before billingRecordRepo.save() completes. The in-flight billing message (delivered at t=0ms and still unacknowledged) is immediately requeued to the billing queue. The SCS binder’s SimpleMessageListenerContainer catches the ShutdownSignalException, logs "Restarting SimpleMessageListenerContainer", and begins a reconnect sequence. At t=5590ms (recoveryInterval=5s), the binder reconnects successfully and registers a new consumer on the billing queue. RabbitMQ delivers the requeued message to the new consumer with redelivered=true.
The consumer receives the redelivered message and — because it does not check AmqpHeaders.REDELIVERED or any application-level guard — generates a new UUID.randomUUID(), calls stripe.charges.create(), and Stripe commits ch_B. The billingRecordRepo.save() at t=510ms (from the original consumer invocation) may or may not have succeeded before the connection close at t=590ms. If it succeeded, the second invocation’s billingRecordRepo.save() throws DataIntegrityViolationException after Stripe commits ch_B — and the process repeats if requeueRejected=true is set. If the original save did not succeed before the connection close, the second invocation saves successfully, but with a different charge ID (ch_B instead of ch_A), and the customer is charged once but with an incorrect charge ID stored in the billing record.
Developer misconception: “The SRE closed an AMQP connection during TLS rotation, which is a network-level event. The consumer was in the middle of a database write when the connection closed — the billing message was not yet acknowledged, so RabbitMQ correctly redelivers it. The
redelivered=trueflag means RabbitMQ is giving the consumer a second chance because the first delivery was interrupted before the consumer could process the message. Since the database write was interrupted by the connection close, the billing record was not saved, which means Stripe was also not called successfully before the interruption.”
The misconception contains a factual error in the last sentence. The Stripe call at t=500ms completed successfully and returned a committed charge ID before the connection close at t=590ms. The AMQP basicAck that marks the message as “done” from RabbitMQ’s perspective is sent by the SCS binder after the consumer method returns normally — which is after billingRecordRepo.save() completes. The connection close at t=590ms interrupted the database write (which was in progress), but the Stripe call had already completed and was not interrupted. The key insight is that the AMQP acknowledgment lifecycle (basicDeliver → processing → basicAck) and the Stripe API lifecycle (HTTP request → charge committed → HTTP response) are independent. RabbitMQ has no visibility into whether Stripe was called during the consumer’s processing.
The buggy consumer configuration and connection recovery
# BillingService application.yml — BUGGY.
# Default acknowledgeMode=AUTO means basicAck is sent AFTER the consumer
# method returns. If the AMQP connection is closed between the Stripe call
# (t=500ms) and the billingRecordRepo.save() completion (t=640ms), the message
# is requeued and redelivered. The consumer generates UUID_B on redelivery.
spring:
cloud:
stream:
bindings:
billing-in-0:
destination: billing
group: billing-service
consumer:
maxAttempts: 3
rabbit:
bindings:
billing-in-0:
consumer:
# Default acknowledgeMode is AUTO.
# basicAck is sent after consumer method returns normally.
# basicNack is sent after consumer method throws.
# Connection close before basicAck means message is requeued.
acknowledgeMode: AUTO
prefetch: 250
recoveryInterval: 5000 # Default: reconnect after 5 seconds.// BillingFunctions.java — BUGGY.
// Consumer does not check AmqpHeaders.REDELIVERED.
// Consumer generates UUID.randomUUID() on every invocation including redeliveries.
@Configuration
public class BillingFunctions {
@Bean
public Consumer<Message<BillingRequestedEvent>> billingConsumer(
StripeClient stripe,
BillingRecordRepository billingRecordRepo) {
return message -> {
BillingRequestedEvent event = message.getPayload();
// BUG: Does not check AmqpHeaders.REDELIVERED.
// Boolean redelivered = message.getHeaders()
// .get(AmqpHeaders.REDELIVERED, Boolean.class);
// Even if it did check, a redelivered=true check without a stable
// key and guard is insufficient: if redelivered=true, should we skip?
// We don't know if Stripe was called on the first delivery.
// BUG: New UUID on every invocation including connection-recovery redelivery.
String idempotencyKey = UUID.randomUUID().toString();
Charge charge = stripe.charges().create(
ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("usd")
.setCustomer(event.getCustomerId())
.build(),
RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());
billingRecordRepo.save(new BillingRecord(
event.getSubscriptionId(),
event.getBillingPeriod(),
charge.getId(),
idempotencyKey));
};
}
}The fixed consumer: stable key + guard makes connection recovery safe
// BillingFunctions.java — FIXED.
// Stable key + BillingRecord guard makes close_connection redelivery safe.
// On redelivery: guard finds BillingRecord (if original DB write succeeded)
// or Stripe returns ch_A from idempotency cache (if stable key is sent).
// In either case, no duplicate charge.
@Configuration
public class BillingFunctions {
@Bean
public Consumer<BillingRequestedEvent> billingConsumer(
StripeClient stripe,
BillingRecordRepository billingRecordRepo) {
return event -> {
// FIX: Stable key — same value on original delivery and redelivery.
String idempotencyKey = stableKey(
event.getSubscriptionId(),
event.getBillingPeriod(),
event.getAmountCents());
// FIX: Guard before Stripe.
// On connection-recovery redelivery:
// - If original billingRecordRepo.save() succeeded before connection
// close: guard finds the existing BillingRecord — return early.
// - If original billingRecordRepo.save() was interrupted by connection
// close: no BillingRecord exists; Stripe is called with the SAME
// stable key; Stripe returns ch_A from idempotency cache (same charge).
if (billingRecordRepo.existsByIdempotencyKey(idempotencyKey)) {
return;
}
Charge charge = stripe.charges().create(
ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("usd")
.setCustomer(event.getCustomerId())
.build(),
RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());
billingRecordRepo.save(new BillingRecord(
event.getSubscriptionId(),
event.getBillingPeriod(),
charge.getId(),
idempotencyKey));
};
}
private static String stableKey(
String subscriptionId, String billingPeriod, long amountCents) {
try {
MessageDigest sha256 = MessageDigest.getInstance("SHA-256");
String input = subscriptionId + "|" + billingPeriod + "|" + amountCents;
byte[] hash = sha256.digest(input.getBytes(StandardCharsets.UTF_8));
return HexFormat.of().formatHex(hash);
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException(e);
}
}
}Operational note: monitoring for connection close events
The SCS RabbitMQ Binder logs a WARN entry when it detects a ShutdownSignalException on the SimpleMessageListenerContainer: "SimpleMessageListenerContainer: Restarting SimpleMessageListenerContainer". This log line is evidence that an in-flight message may have been requeued. If your billing service logs this entry during an incident, and you have an unstable idempotency key strategy in place, you should immediately query both the Stripe charge list (stripe charges list --customer=<cus_id> --created[gte]=<window-start>) and the billing_records table for the subscription in question before the next billing cycle.
With a stable key, the ShutdownSignalException log entry is a no-op from a billing-correctness perspective: the redelivered consumer sends the same idempotency key to Stripe, Stripe returns the cached charge response, and the BillingRecord guard prevents any duplicate DB write. The connection recovery is transparent to the billing system. This makes the stable key pattern not just a correctness fix but also an operational simplification: SREs can close connections for maintenance without coordinating with the billing team about the current processing state.
Test: connection close redelivery is idempotent with stable key and Stripe idempotency
@SpringBootTest
@Testcontainers
class ConnectionCloseRedeliveryTest {
@Container
static RabbitMQContainer rabbitMQ = new RabbitMQContainer("rabbitmq:3.12-management");
@RegisterExtension
static WireMockExtension wm = WireMockExtension.newInstance()
.options(wireMockConfig().dynamicPort())
.build();
@Autowired
private StreamBridge streamBridge;
@Autowired
private BillingRecordRepository billingRecordRepo;
@Test
void connectionCloseRedeliveryDoesNotProduceDuplicateCharge() throws Exception {
// Stripe stub: returns ch_A for the stable idempotency key.
// On redelivery, the same stable key is sent — Stripe returns ch_A again.
String stableKey = sha256("sub_Abc123|2026-10|9900");
wm.stubFor(post(urlPathEqualTo("/v1/charges"))
.withHeader("Idempotency-Key", equalTo(stableKey))
.willReturn(aResponse()
.withStatus(200)
.withBody("""
{"id":"ch_A","object":"charge","amount":9900,
"currency":"usd","status":"succeeded"}
""")));
BillingRequestedEvent event = new BillingRequestedEvent(
"sub_Abc123", "2026-10", 9900L, "cus_X");
// First delivery — BillingRecord saved with ch_A.
streamBridge.send("billing-out-0", event);
await().atMost(Duration.ofSeconds(10))
.until(() -> billingRecordRepo.existsBySubscriptionIdAndBillingPeriod(
"sub_Abc123", "2026-10"));
// Simulate connection-close redelivery (redelivered=true):
// same event published again, as if requeued by broker.
streamBridge.send("billing-out-0", event);
Thread.sleep(2000);
// Stripe was called exactly once — guard found the existing BillingRecord.
wm.verify(exactly(1), postRequestedFor(urlPathEqualTo("/v1/charges")));
assertThat(billingRecordRepo.countBySubscriptionIdAndBillingPeriod(
"sub_Abc123", "2026-10")).isEqualTo(1);
}
private static String sha256(String input) throws Exception {
MessageDigest d = MessageDigest.getInstance("SHA-256");
byte[] hash = d.digest(input.getBytes(StandardCharsets.UTF_8));
return HexFormat.of().formatHex(hash);
}
}Mode 3: channel.basicRecover(true) called from a Spring Actuator admin endpoint requeues all in-flight unacknowledged billing messages including those for which Stripe committed ch_A — UUID_B × N — ch_B × N
What basicRecover(requeue=true) does in the AMQP protocol
The AMQP 0-9-1 basic.recover method instructs the broker to redeliver all messages that were delivered on the channel but not yet acknowledged. When called with requeue=true, the broker re-queues each unacknowledged message and delivers it again, potentially to a different consumer if multiple consumers are registered on the queue. When called with requeue=false, the broker redelivers each message to the same consumer that originally received it. The basic.recover method is specified as a channel-wide operation: it affects every unacknowledged message on the channel, not a specific message identified by delivery tag.
In Spring AMQP, Channel.basicRecover(requeue) can be called directly on the underlying com.rabbitmq.client.Channel object. The Spring AMQP RabbitTemplate does not expose basicRecover directly — it is accessed via rabbitTemplate.execute(channel -> channel.basicRecover(true)) or by injecting the channel via a ChannelAwareMessageListener. The AMQP protocol specification notes that basic.recover is deprecated in AMQP 0-9-1 in favor of using basicNack with requeue=true for individual message rejection, but it remains in the RabbitMQ Java client library and is fully functional on RabbitMQ 3.x and 4.x.
A common misunderstanding about basicRecover(true) is that it operates on “stuck” messages specifically — messages that the consumer has been holding for a long time without acknowledging. In reality, basicRecover has no concept of “stuck”: it redelivers all unacknowledged messages on the channel regardless of when they were delivered or how long they have been in the unacknowledged state. A message that was delivered 10 milliseconds ago is treated identically to a message that has been unacknowledged for 10 minutes.
The failure scenario
A senior engineer at the company builds a custom Spring Boot Actuator endpoint for the BillingService to assist with incident response. The endpoint is intended to address scenarios where the billing consumer appears to be hung: it has received a message (visible in RabbitMQ Management Console as an unacknowledged delivery) but has not acknowledged it after a suspicious length of time. The runbook entry for this endpoint reads: “If billing consumers appear stuck (unacked message count > 0 for >5 minutes), call POST /actuator/billing/recover to force RabbitMQ to redeliver stuck messages to healthy consumers.”
// BillingRecoveryEndpoint.java — the BUGGY admin endpoint.
// Calls channel.basicRecover(true) on the shared AMQP channel.
// This redelivers ALL unacknowledged messages on the channel,
// not just messages that appear stuck.
@Component
@Endpoint(id = "billing-recover")
public class BillingRecoveryEndpoint {
private final CachingConnectionFactory connectionFactory;
private final RabbitTemplate rabbitTemplate;
@WriteOperation
public Map<String, Object> recover() {
try {
// BUG: basicRecover(true) redelivers ALL unacknowledged messages
// on the channel — including messages currently being processed
// (e.g., in the middle of a Stripe HTTP call at t=500ms post-commit).
// There is no way to target only "stuck" messages.
rabbitTemplate.execute(channel -> {
channel.basicRecover(true);
return null;
});
return Map.of("status", "recovery triggered", "requeue", true);
} catch (IOException e) {
return Map.of("status", "recovery failed", "error", e.getMessage());
}
}
}The production scenario: the BillingService is processing a peak billing run at the start of a monthly cycle. The consumer has prefetch=10 and concurrency=1. At t=0ms, the consumer is delivered 10 billing messages simultaneously (filling the prefetch window). The consumer begins processing message 1: generates UUID_A, calls Stripe, which commits ch_A at t=500ms. At t=510ms, the consumer is writing to the database. Messages 2–10 are held in the prefetch buffer, all in the unacknowledged state.
At t=550ms, a first-responder engineer observes the RabbitMQ Management Console showing 10 unacknowledged messages on the billing queue consumer. The engineer has seen this count for the past 30 seconds (the 10 messages were delivered at the start of the billing run, and the consumer is slowly working through them). Misinterpreting the persistent unacked count as a “stuck” state (not understanding that the consumer is actively processing one message at a time with prefetch=10), the engineer calls POST /actuator/billing/recover.
The basicRecover(true) call causes RabbitMQ to requeue all 10 unacknowledged messages: message 1 (for which ch_A was already committed at t=500ms), messages 2–9 (which have not yet been delivered to the consumer function but are in the prefetch buffer), and message 10. The consumer, now in the middle of processing message 1 at t=550ms, receives an AMQP basic.recover-ok response. The current consumer invocation for message 1 may complete normally or may throw, depending on timing. Either way, message 1 is now back in the billing queue’s ready state and will be redelivered to the consumer. The consumer processes message 1 again, generates UUID_B (because the stable key pattern is not implemented), and Stripe commits ch_B. If messages 2–10 were also partially processed before the basicRecover call, the same pattern applies to each.
Developer misconception: “
basicRecover(requeue=true)requeues messages that the consumer has been holding without acknowledging — it’s a way to reset a stuck consumer so that healthy consumers can pick up messages it can’t handle. The consumer is stuck on message 1 (it has been in the unacknowledged state for 30 seconds), so callingbasicRecoverwill requeue message 1 for another consumer to handle. Messages 2–10 are waiting in the prefetch buffer and haven’t started processing yet, so requeuing them is also safe.”
The misconception has three errors. First, the consumer was not stuck: it was actively processing message 1 (making a Stripe call that takes up to 2 seconds). The 30-second unacked count the engineer observed was the normal prefetch window during a billing run, not evidence of a hung consumer. A genuinely hung consumer would be identifiable by other signals: no CPU usage, no Stripe API metrics, no database connection activity. Second, message 1 was not “unprocessed” at the time of recovery: Stripe committed ch_A at t=500ms, 50ms before the basicRecover call. Third, basicRecover(true) does not distinguish between messages that the consumer has started processing and messages that are waiting in the prefetch buffer: all 10 unacknowledged messages are requeued regardless of their processing state.
The scope of basicRecover and how to observe it
The effect of basicRecover(true) on a channel with prefetch=N is proportional to the number of messages in the prefetch window at the time of the call. In a peak billing run with prefetch=250 (the SCS RabbitMQ Binder default) and a consumer that is midway through the prefetch window, a single basicRecover(true) call can requeue up to 250 billing messages simultaneously. Each requeued message is redelivered to the consumer. If the consumer uses UUID.randomUUID() as the idempotency key, each redelivered message generates a new UUID and a new Stripe charge. The practical upper bound on damage from a single basicRecover(true) call with default prefetch is 250 duplicate charges committed in the seconds following the call.
RabbitMQ logs a basic.recover event at the DEBUG level in the channel lifecycle log (rabbitmq_channel_interceptor_trace). The event is not logged at WARN or ERROR level and does not appear in the default RabbitMQ Management Console activity feed. The only indicator in the application logs is the SCS binder logging "Received message with redelivered=true" (if the binder is configured with logging.level.org.springframework.amqp=DEBUG) or the Spring AMQP container logging a ListenerExecutionFailedException if the redelivered message fails at the database guard. In a production environment without DEBUG-level AMQP logging, a basicRecover(true) call and its redelivery effects may be completely invisible in application logs until Stripe charge.succeeded webhooks begin arriving for the duplicate charges.
The fix: remove basicRecover from operational tooling; stable key makes any recovery safe
// BillingRecoveryEndpoint.java — FIXED.
// basicRecover is removed from the operational tooling.
// If a consumer appears genuinely stuck, the correct action is to:
// 1. Check consumer metrics (CPU, DB connections, Stripe API latency).
// 2. Identify the stuck message delivery tag.
// 3. Use rabbitTemplate.execute() to basicNack a specific delivery tag
// (NOT basicRecover which is channel-wide).
// 4. Or, simply restart the pod — SCS binder will reconnect and requeue
// the unacked message from the new connection.
//
// With a stable key + BillingRecord guard, even a pod restart that
// redelivers all 250 prefetched messages is safe: the guard blocks
// any message for which ch_A was already committed.
@Component
@Endpoint(id = "billing-recover")
public class BillingRecoveryEndpoint {
private final BillingRecordRepository billingRecordRepo;
@ReadOperation
public Map<String, Object> status() {
// Expose billing health metrics instead of recovery controls.
long pendingCount = billingRecordRepo.countByStatus("PENDING");
long failedCount = billingRecordRepo.countByStatus("FAILED");
return Map.of(
"pendingBillingRecords", pendingCount,
"failedBillingRecords", failedCount
);
}
// basicRecover is intentionally NOT exposed.
// Recovery from stuck consumers is done via pod restart, which triggers
// SCS binder reconnection and RabbitMQ unacked-message requeue.
// The BillingRecord guard makes pod-restart redelivery safe.
}If the team does need a programmatic way to nack a specific message from an operational endpoint, the correct primitive is basicNack(deliveryTag, false, true) targeting a specific delivery tag, not basicRecover(true). The delivery tag of a specific message can be obtained from the Spring AMQP Channel via the ChannelAwareMessageListener interface or by storing delivery tags in a concurrent map keyed by the billing request ID. However, even a targeted basicNack(tag, false, true) for a message that is at t=500ms post-Stripe-commit will generate a redelivery and a potential double charge if the consumer lacks a stable key and guard.
Test: basicRecover redelivery is idempotent with stable key and guard
@SpringBootTest
@Testcontainers
class BasicRecoverRedeliveryTest {
@Container
static RabbitMQContainer rabbitMQ = new RabbitMQContainer("rabbitmq:3.12-management");
@RegisterExtension
static WireMockExtension wm = WireMockExtension.newInstance()
.options(wireMockConfig().dynamicPort())
.build();
@Autowired
private StreamBridge streamBridge;
@Autowired
private BillingRecordRepository billingRecordRepo;
@Test
void basicRecoverRedeliveryDoesNotProduceDuplicateCharge() throws Exception {
// Stripe stub: returns ch_A on first call for this stable key.
// On redelivery, the same stable key is sent — Stripe returns ch_A from cache.
wm.stubFor(post(urlPathEqualTo("/v1/charges"))
.willReturn(aResponse()
.withStatus(200)
.withBody("""
{"id":"ch_A","object":"charge","amount":9900,
"currency":"usd","status":"succeeded"}
""")));
BillingRequestedEvent event = new BillingRequestedEvent(
"sub_Abc123", "2026-10", 9900L, "cus_X");
// First delivery — BillingRecord saved.
streamBridge.send("billing-out-0", event);
await().atMost(Duration.ofSeconds(10))
.until(() -> billingRecordRepo.existsBySubscriptionIdAndBillingPeriod(
"sub_Abc123", "2026-10"));
// Simulate basicRecover redelivery: N additional deliveries of the same event.
// In production this comes from basicRecover(true) on the AMQP channel.
int REDELIVERY_COUNT = 5;
for (int i = 0; i < REDELIVERY_COUNT; i++) {
streamBridge.send("billing-out-0", event);
}
Thread.sleep(3000);
// Stripe was called exactly once — stable key + guard blocked all redeliveries.
wm.verify(exactly(1), postRequestedFor(urlPathEqualTo("/v1/charges")));
assertThat(billingRecordRepo.countBySubscriptionIdAndBillingPeriod(
"sub_Abc123", "2026-10")).isEqualTo(1);
}
@Test
void multipleMessagesRecoveredSimultaneouslyAreAllIdempotent() throws Exception {
// Stripe stub: returns distinct charge IDs per stable idempotency key.
// For 5 distinct subscriptions, 5 distinct stable keys → 5 distinct ch_X values.
for (int i = 1; i <= 5; i++) {
final int idx = i;
wm.stubFor(post(urlPathEqualTo("/v1/charges"))
.withHeader("Idempotency-Key",
matching(sha256Pattern("sub_Sub" + idx + "|2026-10|9900")))
.willReturn(aResponse()
.withStatus(200)
.withBody(String.format(
"""
{"id":"ch_%d","object":"charge","amount":9900,
"currency":"usd","status":"succeeded"}
""", idx))));
}
// Publish 5 distinct billing events (simulating prefetch=5 window).
List<BillingRequestedEvent> events = new ArrayList<>();
for (int i = 1; i <= 5; i++) {
BillingRequestedEvent e = new BillingRequestedEvent(
"sub_Sub" + i, "2026-10", 9900L, "cus_" + i);
events.add(e);
streamBridge.send("billing-out-0", e);
}
// Wait for all 5 BillingRecords to be saved.
await().atMost(Duration.ofSeconds(20))
.until(() -> billingRecordRepo.countByBillingPeriod("2026-10") == 5);
// Simulate basicRecover: re-publish all 5 events.
for (BillingRequestedEvent e : events) {
streamBridge.send("billing-out-0", e);
}
Thread.sleep(3000);
// Stripe was called exactly 5 times total (once per subscription).
wm.verify(exactly(5), postRequestedFor(urlPathEqualTo("/v1/charges")));
assertThat(billingRecordRepo.countByBillingPeriod("2026-10")).isEqualTo(5);
}
}Diagnostic checklist for SCS RabbitMQ Binder + Stripe billing
Use this checklist when investigating a suspected double-charge in a Spring Cloud Stream RabbitMQ-backed billing system. This checklist covers the three RabbitMQ operational failure modes in this post plus the binder-configuration failure modes from the SCS Binder post and the original SCS post.
- Check
billing_recordsfor duplicate rows:SELECT subscription_id, billing_period, COUNT(*) FROM billing_records GROUP BY 1, 2 HAVING COUNT(*) > 1. Multiple rows confirm a duplicate charge regardless of which failure mode caused it. Compare theidempotency_keycolumn values: if they differ, the consumer is not using a stable key strategy. - Check Stripe for charges with differing idempotency keys:
stripe charges list --customer=<cus_id> --created[gte]=<window-start>. Compareidempotency_keyvalues on duplicate charges. Differing keys confirm aUUID.randomUUID()strategy. Same keys on duplicate charges indicate a Stripe idempotency cache miss or a billing-record-level bug, not a Stripe double-commit. - Check
billing.dlqmessage count andx-deathheaders: If messages are present inbilling.dlq, inspect thex-death[0].reasonfield. A value ofrejecteddoes NOT mean the consumer rejected the message without processing it — it means the SCS binder issuedbasicReject(false)after exhaustingmaxAttempts. Check thex-exception-stacktraceheader to determine whether the exception in the trace occurred before or after the Stripe call in the consumer function body. Do not re-ingest DLQ messages without a stable idempotency key and BillingRecord guard in place. - Check RabbitMQ connection lifecycle logs for
close_connectionevents near the double-charge window: On the RabbitMQ management node, check/var/log/rabbitmq/rabbit@<host>.logfor lines matchingclosing AMQP connectionorconnection closed by clientnear the timestamp of the suspected double-charge. A connection close at t=X within 30–60 seconds of the suspected charge (after Stripe commits but beforebasicAckis sent) is strong evidence for Mode 2. - Check application logs for
SimpleMessageListenerContainer: Restartingentries: This log line in the billing service logs indicates an AMQP connection failure and reconnect event. Each occurrence is a potential source of unacknowledged message redelivery. Correlate the timestamp with Stripe charge events to identify redeliveries that occurred post-commit. - Check for
basicRecovercalls in the codebase and runbooks: Search the billing service codebase forchannel.basicRecover,rabbitTemplate.execute, and any Actuator endpoints that expose recovery operations. Check the team runbook for any step that calls a recovery endpoint orrabbitmqctlcommands that could trigger unacknowledged message requeue. AnybasicRecover(true)call in a production operations context is a latent Mode 3 bug if the consumer does not have a stable key and BillingRecord guard. - Verify the idempotency key strategy in the consumer function: Search the consumer lambda for
UUID.randomUUID(). Any call toUUID.randomUUID()inside aConsumer<T>orConsumer<Message<T>>lambda that calls Stripe is a latent double-charge bug regardless of which failure mode triggers the redelivery. - Verify the BillingRecord guard is in place and is checked before Stripe: Check whether the consumer function calls
billingRecordRepo.existsByIdempotencyKey()(or equivalent) before calling Stripe. The guard must use the stable idempotency key as its lookup field — a guard that queries bysubscriptionId + billingPeriodwithout the idempotency key check will correctly identify an existing charge but cannot be used to short-circuit the Stripe call on redelivery when the original DB write failed.
Summary
The three SCS RabbitMQ operational failure modes documented in this post share the same root cause as every other failure mode in this series: the consumer function generates a non-repeatable idempotency key (UUID.randomUUID()) so that any redelivery of the same message — regardless of reason — produces a new Stripe charge. What distinguishes the RabbitMQ operational modes is that the redelivery trigger is external to the application: an on-call engineer re-ingesting a DLQ message, an SRE closing an AMQP connection during a TLS rotation, or a first-responder calling a basicRecover endpoint from the runbook. None of these operators can know from the RabbitMQ interface alone whether Stripe was called for the message they are about to redeliver.
The DLQ re-ingestion failure mode is driven by the AMQP x-death[0].reason=rejected header misleading the engineer into concluding that the SCS binder rejected the message before the consumer processed it — when in fact reason=rejected is set by basicReject(false) after all maxAttempts retries are exhausted, each of which called Stripe. The close_connection failure mode is driven by the standard AMQP behavior that closes a connection immediately without coordinating with in-flight consumers, causing unacked messages to be requeued at the broker level in a way the SCS application layer cannot prevent. The basicRecover(true) failure mode is driven by a misunderstanding of the AMQP protocol: basicRecover is a channel-wide operation that requeues all unacknowledged messages simultaneously, not a targeted operation for stuck messages.
The universal fix is unchanged from every other post in this series: a stable content-hash idempotency key (SHA-256(subscriptionId + billingPeriod + amountCents)) that produces the same value for the same billing intent regardless of which Consumer invocation generates it, combined with a BillingRecord guard that checks the database for an existing charge before calling Stripe. These two patterns together make the consumer function naturally idempotent against every RabbitMQ delivery path: autoBindDlq re-ingestion, connection-close requeue, basicRecover channel flush, transacted=true basicNack redelivery, republishToDlq reprocessing, and maxAttempts retry. No RabbitMQ operational procedure — however well-intentioned — can produce a duplicate Stripe charge when both safeguards are in place.
Keybrake — Stripe API key governance for billing services
Keybrake issues per-billing-period restricted Stripe API keys with hard spend caps. A double-charge from DLQ re-ingestion, connection close requeue, or basicRecover channel flush commits at most one billing period’s cap — not an uncapped set of charges across every message in the prefetch window.