Spring Integration IBM MQ Stripe Integration: How JmsInboundChannelAdapter SESSION_TRANSACTED Commits the IBM MQ Transaction on JmsTemplate.receive() Return Before ExecutorChannel Dispatches to the ServiceActivator Silently Losing the BillingRecord on DB Exception, How ChainedTransactionManager Commits the JmsTransactionManager Phase Before DataSourceTransactionManager Permanently Consuming M1 Without a BillingRecord on DB Commit Failure, and How IBM MQ BACKOUT_THRESHOLD Moves M1 to BACKOUT_QUEUE_NAME Where a Separate Consumer Generates UUID_B Committing ch_B Alongside ch_A from the Retry Path
Spring Integration’s IBM MQ integration introduces three transaction and message-lifecycle behaviors that differ structurally from both Spring Integration’s push-mode DefaultMessageListenerContainer adapter and the ActiveMQ, RabbitMQ, and Solace adapters covered earlier in this series: a polling JmsInboundChannelAdapter whose JmsTemplate-based session management commits the IBM MQ local transaction on channel.send() return rather than on handler completion, a ChainedTransactionManager composition that commits IBM MQ’s JMS local transaction before the database transaction in a way that cannot be rolled back on DB commit failure, and IBM MQ’s broker-side BACKOUT_THRESHOLD mechanism that moves messages to a BACKOUT_QUEUE independently of what the application’s retry logic believes — each producing duplicate Stripe charges when the billing handler uses UUID.randomUUID() per invocation.
This post covers all three failure modes end to end: the exact Spring Integration + IBM MQ configuration that enables each one, the step-by-step timeline showing precisely when ch_A commits and when UUID_B is generated, the specific misconception each mode exploits, and the idempotency fix that makes the billing ServiceActivator safe regardless of channel type, transaction manager composition, or IBM MQ broker-side redelivery behavior. Code samples target ibm-mq-spring-boot-starter 3.x, com.ibm.mq.jakarta.client 9.x, Spring Integration 6.x, Spring Boot 3.x, WireMock, Testcontainers with the IBM MQ Docker image, and JUnit 5.
Background: three IBM MQ-specific behaviors that differ from RabbitMQ, ActiveMQ, and Solace at the transaction and message-lifecycle layers
To understand why these three failure modes are specific to Spring Integration IBM MQ rather than to other Spring Integration JMS adapters, you need to understand three structural differences introduced by the combination of IBM MQ’s JMS client, Spring Integration’s polling adapter, and IBM MQ’s broker-side backout mechanism.
JmsInboundChannelAdapter in polling mode uses JmsTemplate internally, and JmsTemplate manages the JMS session transaction as a single unit scoped to the execute(SessionCallback) call. The JmsInboundChannelAdapter operates in two modes: push mode (backed by DefaultMessageListenerContainer) and polling mode (backed by JmsTemplate). In polling mode, the adapter calls JmsTemplate.receiveAndConvert() or JmsTemplate.execute(new ReceiveCallback()) on each polling interval. JmsTemplate.execute(SessionCallback) opens a JMS session, passes it to the callback, and — if sessionTransacted=true — commits the session when the callback returns. The dispatch to the output channel happens inside the callback, before the commit. For DirectChannel, channel.send() is synchronous and blocks until the entire downstream handler chain returns, meaning the ServiceActivator completes before JmsTemplate’s session commits. For ExecutorChannel, channel.send() enqueues the task and returns immediately, so JmsTemplate commits the IBM MQ transaction before any ServiceActivator thread has started. This is a fundamentally different commit scope from DefaultMessageListenerContainer in push mode, where the session commit is tied to the onMessage() callback completion regardless of channel type.
ChainedTransactionManager commits its constituent transaction managers in registration order, not in a 2PC manner. ChainedTransactionManager (from spring-data-commons) is a common pattern for coordinating JMS and database transactions without a full JTA XA implementation. It creates a composite transaction that, on commit, calls each constituent manager’s commit() in the order they were registered, and on rollback calls each manager’s rollback() in reverse order. This means that if JmsTransactionManager is registered first (which is the common pattern to ensure JMS acks are rolled back on DB failure), JMS commits happen before DB commits. In the vast majority of cases — normal execution, runtime exceptions — this is fine: if the ServiceActivator throws, the exception triggers a rollback before any commit occurs, and both managers roll back in reverse order. The failure case is a commit-phase failure: the ServiceActivator completes without exception, JmsTransactionManager.commit() executes successfully (IBM MQ message permanently consumed), and then DataSourceTransactionManager.commit() fails (DB connection reset, timeout, or optimistic lock at commit time). At that point, the IBM MQ message is gone and cannot be recovered; the BillingRecord was never persisted. This is a well-known limitation of ChainedTransactionManager — its Javadoc explicitly warns that “if the second resource fails to commit, the first will not be rolled back” — but teams migrating from single-resource transactions often miss the implication for IBM MQ billing consumers.
IBM MQ’s BACKOUT_THRESHOLD is a broker-side property that moves messages to a designated BACKOUT_QUEUE based on MQMD.BackoutCount, independently of the application’s retry logic or Spring Integration’s error handling. When a message is rolled back (via JMS session.rollback() or DefaultMessageListenerContainer exception handling with sessionTransacted=true), IBM MQ increments the MQMD.BackoutCount field in the message descriptor. If BackoutCount >= BOTHRESH (BackoutThreshold, configured as a queue attribute), IBM MQ will not redeliver the message on the next session.receive() call; instead, at the point of the next attempted delivery, it moves the message to the queue named in BOQNAME (BackoutQueueName). Critically, this BOTHRESH check and the move to BOQNAME happen at the IBM MQ broker layer, not at the Spring Integration layer. If a team has set up a separate Spring Integration pipeline consuming from the BOQNAME queue as a “recovery handler,” that handler may run even when the main-queue consumer ultimately succeeded on an earlier attempt but the message was redelivered multiple times before success. The BOQNAME handler, if it calls Stripe with UUID.randomUUID(), generates a second Stripe charge alongside the one already committed by the main-queue pipeline.
With these three structural differences understood, each failure mode follows directly from a specific configuration choice.
Failure mode 1: JmsInboundChannelAdapter polling + SESSION_TRANSACTED + ExecutorChannel — JmsTemplate commits IBM MQ transaction on channel.send() return before ServiceActivator runs; DB exception silently drops BillingRecord; reconciliation republish → UUID_B → ch_B
The setup
A team building a billing consumer on IBM MQ chooses JmsInboundChannelAdapter in polling mode because their IBM MQ deployment uses a specific TLS mutual-authentication configuration that works reliably with JmsTemplate’s connection management but has historically caused reconnection issues with DefaultMessageListenerContainer’s background listener threads. The adapter is initially wired to a DirectChannel, which processes billing events one at a time on the polling thread. To improve throughput, the team switches the output channel to an ExecutorChannel backed by a thread pool.
The billing handler generates its Stripe idempotency key with UUID.randomUUID() at the start of each invocation:
@ServiceActivator(inputChannel = "billingChannel")
public void handleBillingEvent(Message<BillingEvent> message) {
BillingEvent event = message.getPayload();
String idempotencyKey = UUID.randomUUID().toString(); // new key on every call
ChargeCreateParams params = ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("gbp")
.setCustomer(event.getCustomerId())
.build();
Charge charge = Charge.create(params,
RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());
billingRecordRepo.save(new BillingRecord(event, idempotencyKey, charge.getId()));
}
The Spring Integration configuration switches from DirectChannel to ExecutorChannel and uses a transacted JmsInboundChannelAdapter:
@Bean
public JmsInboundChannelAdapter billingAdapter(
@Qualifier("billingChannel") MessageChannel outputChannel,
ConnectionFactory connectionFactory) {
JmsTemplate jmsTemplate = new JmsTemplate(connectionFactory);
jmsTemplate.setSessionTransacted(true); // IBM MQ local JMS transaction
jmsTemplate.setDefaultDestinationName("BILLING.EVENTS.QUEUE");
JmsInboundChannelAdapter adapter =
new JmsInboundChannelAdapter(jmsTemplate);
adapter.setOutputChannel(outputChannel);
adapter.setAutoStartup(true);
return adapter;
}
@Bean
public MessageChannel billingChannel() {
// Switched from DirectChannel for throughput
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(8);
executor.setMaxPoolSize(16);
executor.setQueueCapacity(100);
executor.initialize();
return new ExecutorChannel(executor);
}
The IBM MQ queue BILLING.EVENTS.QUEUE is configured with BOTHRESH(5) and BOQNAME(BILLING.EVENTS.BACKOUT). The reconciliation job runs every 72 hours and compares Stripe charges against BillingRecord rows, republishing any orphaned charge.
The failure timeline
A billing event M1 arrives for customer cust_GB_019. A transient HikariCP pool exhaustion during a load spike causes billingRecordRepo.save() to throw CannotGetJdbcConnectionException at t=520ms, after the Stripe API has already committed ch_A.
- t=0.000s — The polling thread wakes.
JmsInboundChannelAdaptercallsJmsTemplate.execute(new ReceiveCallback()).JmsTemplateopens aSessionwithtransacted=trueagainst the IBM MQ queue manager. IBM MQ delivers M1 to the JMSSession. - t=0.001s —
JmsTemplate’sReceiveCallbackwraps M1 in a Spring IntegrationMessage<BillingEvent>and callsbillingChannel.send(siMessage). BecausebillingChannelis anExecutorChannel,send()enqueues a task in the executor thread pool and returns immediately. The polling thread is unblocked. TheReceiveCallback.doInJms(Session)method returns. - t=0.002s —
JmsTemplate.execute()detects thatsessionTransacted=true. It callssession.commit()on the IBM MQ JMS session. IBM MQ marks M1 as permanently consumed under syncpoint. M1 is no longer on the queue. IBM MQ’s BackoutCount for M1 is never incremented because the transaction committed successfully. - t=0.003s — An executor thread picks up the task.
handleBillingEvent()begins.UUID.randomUUID()generatesUUID_A. The StripeCharge.create()API call begins. - t=0.500s — Stripe processes the charge and responds
200 OKwithcharge.id = ch_A. - t=0.520s —
billingRecordRepo.save(new BillingRecord(event, UUID_A, ch_A))throwsCannotGetJdbcConnectionException(HikariCP pool exhausted under concurrent load). The exception propagates out ofhandleBillingEvent(). - t=0.521s — Spring Integration’s executor error handling routes the exception to
errorChannel. The error handler logs the exception. From IBM MQ’s perspective: M1 was committed-consumed at t=0.002s. M1 will not be redelivered. The IBM MQBackoutCountfor M1 remains 0; theBOTHRESHthreshold is never triggered. - t=72h+ — The reconciliation job finds Stripe charge
ch_Awithout a matchingBillingRecord. It reconstructs the original billing event and publishes M2 (new message, same business payload) toBILLING.EVENTS.QUEUE. - t=72h+0.500s —
handleBillingEvent(M2)runs.UUID.randomUUID()generatesUUID_B. Stripe’s 24-hour idempotency cache has expired. Stripe creates a new charge and returnsch_B.BillingRecord(UUID_B, ch_B)is saved successfully.
The customer is charged twice: ch_A at t=0.500s and ch_B at t=72h+0.500s. No error is visible at the IBM MQ layer — the message was consumed cleanly. The IBM MQ BOTHRESH mechanism is never triggered, so the operations team has no dead-letter evidence that anything went wrong. The double charge appears as a reconciliation “successful recovery” in the billing logs.
Developer misconception
“
SESSION_TRANSACTED=trueon theJmsTemplatemeans the IBM MQ transaction does not commit until myServiceActivatorhas finished processing the message.JmsTemplatemust hold the session open across thechannel.send()dispatch so that any exception in theServiceActivatorcan trigger asession.rollback(). I’ve usedDefaultMessageListenerContainerwithsessionTransacted=truefor years and it always rolls back on exception — I assumed the polling adapter worked the same way.”
This misconception conflates two different architectures. DefaultMessageListenerContainer in push mode calls onMessage() synchronously inside the JMS session scope — the session does not commit until onMessage() returns (or the DMLC exception handler decides to rollback). The JMS session is kept open for the entire duration of message processing. JmsInboundChannelAdapter in polling mode uses JmsTemplate.execute(SessionCallback), which scopes the session to a single execute() call. The adapter calls channel.send() inside that session scope, but JmsTemplate has no knowledge of what the output channel does downstream — it only knows when channel.send() returns. For DirectChannel, those happen to be the same point in time. For ExecutorChannel, channel.send() returns as soon as the task is queued, decoupling the session scope from handler execution. JmsTemplate’s design was never intended to hold sessions open across asynchronous downstream work; the adaptation is the application developer’s responsibility.
The fix
Two independent fixes, both required:
Fix 1: Switch to push mode (DefaultMessageListenerContainer backed adapter) when using ExecutorChannel. DMLC’s AbstractMessageListenerContainer processes messages inside the JMS session callback, so the session commit is tied to the listener completion. With sessionTransacted=true and a concurrent DMLC listener, each listener thread has its own session, and the session commits only after onMessage() returns on that thread — which, for a DirectChannel wired to a concurrent-consumer DMLC, means after the ServiceActivator completes:
@Bean
public JmsMessageDrivenEndpoint billingEndpoint(
@Qualifier("billingChannel") MessageChannel outputChannel,
ConnectionFactory connectionFactory) {
DefaultMessageListenerContainer container =
new DefaultMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setDestinationName("BILLING.EVENTS.QUEUE");
container.setSessionTransacted(true); // IBM MQ local JMS transaction
container.setConcurrentConsumers(4);
container.setMaxConcurrentConsumers(16);
ChannelPublishingJmsMessageListener listener =
new ChannelPublishingJmsMessageListener();
listener.setRequestChannel(outputChannel);
return new JmsMessageDrivenEndpoint(container, listener);
}
With DMLC, each concurrent consumer thread holds its own Session across the full onMessage() execution. If handleBillingEvent() throws, DMLC catches the exception and calls session.rollback() — IBM MQ increments BackoutCount for M1 and redelivers it.
Fix 2: Use a stable content-derived idempotency key. DMLC + sessionTransacted=true handles the transactional case, but the reconciliation job and any BACKOUT_QUEUE handler also need to be safe. A stable key derived from the business payload ensures every redelivery — from DMLC rollback, from reconciliation republish, or from the BACKOUT_QUEUE consumer — produces the same Stripe idempotency key:
@ServiceActivator(inputChannel = "billingChannel")
public void handleBillingEvent(Message<BillingEvent> message) {
BillingEvent event = message.getPayload();
String stableKey = "billing:" + sha256(
event.getAccountId() + ":" + event.getBillingPeriod() + ":" + event.getAmountCents()
);
if (billingRecordRepo.existsByIdempotencyKey(stableKey)) {
return; // Stripe already charged; BillingRecord already saved
}
ChargeCreateParams params = ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("gbp")
.setCustomer(event.getCustomerId())
.build();
Charge charge = Charge.create(params,
RequestOptions.builder().setIdempotencyKey(stableKey).build());
billingRecordRepo.save(new BillingRecord(event, stableKey, charge.getId()));
}
private String sha256(String input) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8));
return HexFormat.of().formatHex(hash);
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException(e);
}
}
With both fixes applied: DMLC ensures that a DB exception rolls back the JMS session and IBM MQ redelivers M1 (incrementing BackoutCount); the stable key ensures that every redelivery — including the reconciliation republish M2 — uses the same Stripe idempotency key; Stripe returns ch_A from its cache within 24 hours, and the existsByIdempotencyKey guard short-circuits after the BillingRecord is saved.
Failure mode 2: ChainedTransactionManager + JmsTransactionManager-first ordering — IBM MQ message committed before DB transaction commits; DB commit-phase failure leaves no BillingRecord; message permanently consumed; reconciliation republish → UUID_B → ch_B
The setup
A platform team is asked to add transactional guarantees to a billing consumer that uses DefaultMessageListenerContainer in push mode. They want to ensure that a DB exception during billingRecordRepo.save() triggers both a database rollback and an IBM MQ JMS session rollback (so M1 is redelivered). Rather than using a full JTA XA implementation (which requires IBM MQ’s XA client and a JTA provider), they use ChainedTransactionManager from spring-data-commons — a well-documented pattern for “best-effort 1.5 phase commit”.
The team registers JmsTransactionManager first (on the reasoning that JMS transactions should be the “outer” resource, so DB failures cause JMS rollback) and DataSourceTransactionManager second:
@Bean
public PlatformTransactionManager transactionManager(
ConnectionFactory connectionFactory,
DataSource dataSource) {
JmsTransactionManager jmsTm = new JmsTransactionManager(connectionFactory);
DataSourceTransactionManager dbTm = new DataSourceTransactionManager(dataSource);
// JmsTm registered first → commits first → rolls back last
return new ChainedTransactionManager(jmsTm, dbTm);
}
@Bean
public DefaultMessageListenerContainer billingContainer(
ConnectionFactory connectionFactory,
PlatformTransactionManager transactionManager) {
DefaultMessageListenerContainer container =
new DefaultMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setDestinationName("BILLING.EVENTS.QUEUE");
container.setSessionTransacted(false); // managed by ChainedTransactionManager
container.setTransactionManager(transactionManager);
container.setConcurrentConsumers(4);
return container;
}
@ServiceActivator(inputChannel = "billingChannel")
@Transactional
public void handleBillingEvent(Message<BillingEvent> message) {
BillingEvent event = message.getPayload();
String idempotencyKey = UUID.randomUUID().toString();
ChargeCreateParams params = ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("gbp")
.setCustomer(event.getCustomerId())
.build();
Charge charge = Charge.create(params,
RequestOptions.builder().setIdempotencyKey(idempotencyKey).build());
billingRecordRepo.save(new BillingRecord(event, idempotencyKey, charge.getId()));
}
Under normal conditions this works: when billingRecordRepo.save() throws a runtime exception, the @Transactional proxy triggers a rollback on the ChainedTransactionManager. The chained manager rolls back in reverse order — DataSourceTransactionManager first, then JmsTransactionManager — so IBM MQ increments BackoutCount for M1 and redelivers it.
The failure timeline
Customer cust_GB_033 triggers a billing event. The ServiceActivator executes without runtime exception — the Stripe charge succeeds and the DB write completes within the transaction. The @Transactional proxy initiates a commit. At commit time, a DB connection to the PostgreSQL primary is reset mid-commit (a brief network partition between the application pod and the database pod during a Kubernetes node drain): DataSourceTransactionManager.commit() throws TransactionSystemException wrapping PSQLException: connection reset.
- t=0.000s — DMLC delivers M1 to the Spring Integration pipeline. The
ChainedTransactionManagerbegins a composite transaction:JmsTransactionManageropens a transacted IBM MQ JMS session;DataSourceTransactionManageropens a DB connection and begins a transaction. - t=0.001s —
handleBillingEvent()runs on the DMLC listener thread.UUID.randomUUID()generatesUUID_A. The StripeCharge.create()call begins. - t=0.430s — Stripe responds
200 OK.charge.getId()returnsch_A. - t=0.431s —
billingRecordRepo.save(new BillingRecord(event, UUID_A, ch_A))executes successfully. The row is written to the DB connection’s transaction buffer (not yet committed). - t=0.432s —
handleBillingEvent()returns without exception. The@Transactionalproxy callsChainedTransactionManager.commit(). - t=0.433s —
ChainedTransactionManagercallsJmsTransactionManager.commit()first (registration order). The IBM MQ JMS session callssession.commit(). IBM MQ marks M1 as permanently consumed under syncpoint. M1 is removed from the queue. This is irreversible. - t=0.434s —
ChainedTransactionManagercallsDataSourceTransactionManager.commit(). The PostgreSQL connection attempts to send theCOMMITcommand. Mid-send, the TCP connection is reset by the network partition.connection.commit()throwsPSQLException: An I/O error occurred while sending to the backend.DataSourceTransactionManagerwraps this inTransactionSystemExceptionand throws. - t=0.435s —
ChainedTransactionManager.commit()receives the exception from the DB phase. It attempts to roll back the remaining managers. However,JmsTransactionManagerhas already committed — there is nothing to roll back. IBM MQ has permanently consumed M1. TheChainedTransactionManagerpropagates theTransactionSystemException. - t=0.436s — The exception propagates to the DMLC listener infrastructure. DMLC logs
TransactionSystemException. The billing event forcust_GB_033is not retried: from IBM MQ’s perspective, M1 was committed-consumed at t=0.433s.BackoutCountwas never incremented. The queue has no knowledge of M1’s processing. - t=72h+ — The reconciliation job finds Stripe charge
ch_Awithout a matchingBillingRecordrow. It reconstructs the billing event and publishes M2 toBILLING.EVENTS.QUEUE. - t=72h+0.430s —
handleBillingEvent(M2)runs.UUID.randomUUID()generatesUUID_B. Stripe’s 24-hour idempotency cache has expired. Stripe createsch_B.BillingRecord(UUID_B, ch_B)is saved and committed successfully. Customercust_GB_033’s card now has two charges.
The double charge is caused by a commit-phase DB failure — an event that is rare, transient, and invisible in application logs as a processing error. The TransactionSystemException logged at t=0.435s looks like a transient infrastructure problem, not a billing data-integrity issue. The reconciliation job three days later looks like a successful recovery. The two log entries are separated by 72 hours and attributed to different incident tickets.
Developer misconception
“
ChainedTransactionManagerprovides near-XA guarantees for JMS + DB. If DB fails, JMS rolls back. I’ve tested this: whenbillingRecordRepo.save()throws during normal execution, IBM MQ redelivers the message. The JMS-first registration order means JMS is the ‘outer’ resource — if the inner resource (DB) fails, the outer resource rolls back too.”
This misconception misidentifies when ChainedTransactionManager rolls back vs. when it is too late to roll back. During the execution phase (inside the @Transactional method body), any exception triggers a rollback before any commit is attempted — both managers roll back in reverse order, IBM MQ redelivers, and the system behaves correctly. This covers the common case (DB constraint violation, CannotGetJdbcConnectionException) and is what developers test. The commit phase is different: once ChainedTransactionManager starts committing, it cannot roll back a manager that has already committed. JMS-first registration means JMS commits first during the commit phase. A DB commit failure after the JMS commit is a window that cannot be closed by the ChainedTransactionManager design. The Javadoc for ChainedTransactionManager states explicitly: “In case of failure it is not guaranteed to roll back the first resource even if it participated in the transaction.” Most developers who read this treat it as a theoretical concern and do not account for commit-phase DB failures under infrastructure stress.
The fix
Two independent fixes, both required:
Fix 1: Register DataSourceTransactionManager first in ChainedTransactionManager. This reverses the commit order: DB commits first, then JMS commits. If the DB commits and the JMS commit fails (a much rarer event with IBM MQ’s reliable client), the IBM MQ message is not acknowledged and IBM MQ will redeliver it on the next poll — the stable idempotency key (Fix 2) makes this redelivery safe. If the DB commit fails, the JMS commit has not yet occurred, so the JMS transaction rolls back and IBM MQ redelivers M1 for a retry:
@Bean
public PlatformTransactionManager transactionManager(
ConnectionFactory connectionFactory,
DataSource dataSource) {
DataSourceTransactionManager dbTm = new DataSourceTransactionManager(dataSource);
JmsTransactionManager jmsTm = new JmsTransactionManager(connectionFactory);
// DB registered first → DB commits first → DB rolls back last
// If DB commit fails, JMS has not yet committed → JMS rolls back → IBM MQ redelivers
// If JMS commit fails (rare), DB already committed → stable key prevents double-charge
return new ChainedTransactionManager(dbTm, jmsTm);
}
Fix 2: Use a stable content-derived idempotency key. Even with DB-first ordering, there is a residual window: DB commits, JMS commit fails, IBM MQ redelivers M1, and the billing handler runs again. Without a stable key, the retry generates UUID_B → ch_B. With a stable key, Stripe returns ch_A from cache (within 24 hours) or the existsByIdempotencyKey guard short-circuits (beyond 24 hours):
@ServiceActivator(inputChannel = "billingChannel")
@Transactional
public void handleBillingEvent(Message<BillingEvent> message) {
BillingEvent event = message.getPayload();
String stableKey = "billing:" + sha256(
event.getAccountId() + ":" + event.getBillingPeriod() + ":" + event.getAmountCents()
);
if (billingRecordRepo.existsByIdempotencyKey(stableKey)) {
return; // Idempotent short-circuit
}
ChargeCreateParams params = ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("gbp")
.setCustomer(event.getCustomerId())
.build();
Charge charge = Charge.create(params,
RequestOptions.builder().setIdempotencyKey(stableKey).build());
billingRecordRepo.save(new BillingRecord(event, stableKey, charge.getId()));
}
With both fixes applied: DB-first commit ordering minimizes the commit-phase window (DB failures before JMS commit → IBM MQ redelivers); stable key makes all redeliveries safe (DMLC rollback redeliveries, residual JMS-commit-failure redeliveries, reconciliation republishes); the BACKOUT_QUEUE handler described in Mode 3 also becomes safe for the same reason.
Failure mode 3: IBM MQ BACKOUT_THRESHOLD + BACKOUT_QUEUE_NAME consumer using UUID.randomUUID() — retry exhaustion moves M1 to BACKOUT_QUEUE; BACKOUT consumer generates UUID_B; Stripe commits ch_B alongside ch_A from the final successful retry on the main queue
The setup
A team configures the IBM MQ billing queue with a backout threshold and a dedicated backout queue:
DEFINE QLOCAL(BILLING.EVENTS.QUEUE) BOTHRESH(5) BOQNAME(BILLING.EVENTS.BACKOUT) DEFPSIST(YES)
DEFINE QLOCAL(BILLING.EVENTS.BACKOUT) DEFPSIST(YES)
The main-queue consumer uses DefaultMessageListenerContainer with sessionTransacted=true. The billing ServiceActivator generates Stripe idempotency keys with UUID.randomUUID(). A separate Spring Integration pipeline consumes from BILLING.EVENTS.BACKOUT — the team’s “recovery handler” that re-attempts billing for messages the main queue gave up on:
// Main-queue consumer (on BILLING.EVENTS.QUEUE)
@ServiceActivator(inputChannel = "mainBillingChannel")
public void handleBillingEvent(Message<BillingEvent> message) {
BillingEvent event = message.getPayload();
String idempotencyKey = UUID.randomUUID().toString();
Charge charge = Charge.create(
ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("gbp")
.setCustomer(event.getCustomerId())
.build(),
RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
);
billingRecordRepo.save(new BillingRecord(event, idempotencyKey, charge.getId()));
}
// BACKOUT_QUEUE consumer (on BILLING.EVENTS.BACKOUT)
@ServiceActivator(inputChannel = "backoutBillingChannel")
public void handleBackoutBillingEvent(Message<BillingEvent> message) {
BillingEvent event = message.getPayload();
log.warn("Billing event in BACKOUT queue, re-attempting: {}", event.getId());
String idempotencyKey = UUID.randomUUID().toString(); // ← new random key
Charge charge = Charge.create(
ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("gbp")
.setCustomer(event.getCustomerId())
.build(),
RequestOptions.builder().setIdempotencyKey(idempotencyKey).build()
);
billingRecordRepo.save(new BillingRecord(event, idempotencyKey, charge.getId()));
}
The team’s mental model is that the BACKOUT_QUEUE handler only processes messages that the main queue completely failed to process — it is a safety net for catastrophic failure, not a parallel processing path.
The failure timeline
A billing event M1 arrives for customer cust_GB_058 during a period of intermittent DB connection pool exhaustion. The main-queue consumer retries multiple times before succeeding. IBM MQ’s BACKOUT_THRESHOLD=5 means IBM MQ moves M1 to BILLING.EVENTS.BACKOUT after the BackoutCount reaches 5.
- Attempt 1 (BackoutCount=0) — DMLC delivers M1 to the main billing
ServiceActivator.UUID.randomUUID()generatesUUID_A1. StripeCharge.create()begins. Stripe responds200 OKwithch_A1.billingRecordRepo.save()throwsCannotGetJdbcConnectionException(pool exhausted).@Transactionalrolls back the DB. Exception propagates to DMLC. DMLC callssession.rollback(). IBM MQ incrementsBackoutCountfor M1 to 1. M1 is requeued. Stripe has committedch_A1to the customer’s card. - Attempt 2 (BackoutCount=1) — DMLC redelivers M1.
UUID.randomUUID()generatesUUID_A2. StripeCharge.create()with new key. Stripe processes a new charge:ch_A2.billingRecordRepo.save()throws again. DMLC rolls back. IBM MQ incrementsBackoutCountto 2. Now two Stripe charges exist:ch_A1andch_A2. (This illustrates why stable keys matter even before reaching BACKOUT_THRESHOLD.) - Attempt 3 (BackoutCount=2) — Same pattern.
UUID_A3→ch_A3. DB fails. Rollback.BackoutCount=3. Three charges. - Attempt 4 (BackoutCount=3) — Same.
UUID_A4→ch_A4. DB fails. Rollback.BackoutCount=4. Four charges. - Attempt 5 (BackoutCount=4) — DB connection pool recovers.
UUID.randomUUID()generatesUUID_A5. StripeCharge.create()→ch_A5.billingRecordRepo.save(new BillingRecord(event, UUID_A5, ch_A5))succeeds. DMLC callssession.commit(). IBM MQ marks M1 as acknowledged (BackoutCount stays at 4; the message was successfully committed on this delivery attempt). - The BACKOUT_QUEUE complication — IBM MQ checks
BackoutCount >= BOTHRESHon each attempted redelivery before placing the message on the consumer’s queue. After Attempt 4’s rollback (BackoutCount=4, BOTHRESH=5), IBM MQ redelivers M1 for Attempt 5 because BackoutCount (4) is still less than BOTHRESH (5). After Attempt 5’s successful commit, M1 is consumed. IBM MQ’s BOTHRESH logic never fires — BackoutCount never reached 5. - However — in many IBM MQ topologies, the BACKOUT_QUEUE consumer is also triggered by the IBM MQ queue-management layer when BackoutCount is approaching the threshold, or by a parallel monitoring process that periodically scans for messages with high BackoutCount and moves them manually. In this team’s deployment, they have an IBM MQ operations script that moves any message with
BackoutCount >= BOTHRESH - 1(i.e., BackoutCount >= 4) to the BACKOUT_QUEUE proactively — the intent being to prevent BOTHRESH from being hit. After Attempt 4’s rollback, the operations script sees M1 with BackoutCount=4 onBILLING.EVENTS.QUEUEand places a copy onBILLING.EVENTS.BACKOUTbefore DMLC can pick it up for Attempt 5. - t=Attempt5_plus_0.001s — DMLC’s Attempt 5 picks up M1 from
BILLING.EVENTS.QUEUEconcurrently with the operations script copying toBILLING.EVENTS.BACKOUT. Attempt 5 succeeds:ch_A5committed,BillingRecordsaved. - t=Attempt5_plus_2.000s — The BACKOUT_QUEUE consumer picks up M1 from
BILLING.EVENTS.BACKOUT.UUID.randomUUID()generatesUUID_B. Stripe’s idempotency cache forUUID_A5exists, butUUID_Bis a different key. Stripe processes a new charge:ch_B.BillingRecord(UUID_B, ch_B)is saved. Customercust_GB_058’s card now has bothch_A5andch_B.
The net result is five Stripe charges from Attempt 1 through 4 (each with a different random key, each a real charge because Stripe’s 24-hour cache only protects retries with the same key within 24 hours), plus the successful ch_A5, plus the BACKOUT_QUEUE handler’s ch_B. The operations team’s “proactive backout” script was designed to protect the system, but without stable idempotency keys it created additional duplicate charges on top of the retry loop’s own duplicates.
Developer misconception
“The BACKOUT_QUEUE handler only processes messages that the main-queue consumer gave up on after exhausting all retries. If the main-queue consumer eventually succeeded, IBM MQ would have committed M1 and removed it from the queue before it was moved to the BACKOUT_QUEUE. The two paths cannot be active simultaneously for the same message. The BACKOUT_QUEUE handler is a safety net that runs after the main queue has given up — not a parallel charge path.”
This is incorrect in two ways. First, IBM MQ’s BOTHRESH check is not an application-level mutex — there is no exclusive lock that prevents a message from being on both the main queue and the BACKOUT_QUEUE simultaneously. Manual moves, monitoring scripts, and application-layer BACKOUT_QUEUE logic operate independently of IBM MQ’s own BOTHRESH logic. Second, even when relying only on IBM MQ’s native BOTHRESH behavior (without manual moves), the BOTHRESH check happens at the next attempted delivery after BackoutCount reaches the threshold. If the application’s retry loop has multiple concurrent consumer threads, it is possible for one thread to commit M1 successfully (consuming it from the main queue) while another thread is processing a message it received before the successful commit completed. More importantly: the main-queue retry loop itself generates multiple Stripe charges for M1 across attempts 1 through N-1 (before the first successful attempt), because each attempt uses UUID.randomUUID(). The BACKOUT_QUEUE handler generates one more. Any of these charges represents a real double charge to the customer.
The fix
Three independent fixes, all required:
Fix 1: Use a stable content-derived idempotency key in both the main-queue and BACKOUT_QUEUE handlers. A stable key means every retry attempt — whether on the main queue or the BACKOUT_QUEUE — produces the same Stripe idempotency key. Stripe returns the cached charge from the first successful attempt on every subsequent call:
// Main-queue consumer
@ServiceActivator(inputChannel = "mainBillingChannel")
@Transactional
public void handleBillingEvent(Message<BillingEvent> message) {
BillingEvent event = message.getPayload();
String stableKey = "billing:" + sha256(
event.getAccountId() + ":" + event.getBillingPeriod() + ":" + event.getAmountCents()
);
if (billingRecordRepo.existsByIdempotencyKey(stableKey)) {
return; // Already processed; ack the message without re-charging
}
Charge charge = Charge.create(
ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("gbp")
.setCustomer(event.getCustomerId())
.build(),
RequestOptions.builder().setIdempotencyKey(stableKey).build()
);
billingRecordRepo.save(new BillingRecord(event, stableKey, charge.getId()));
}
// BACKOUT_QUEUE consumer — same stable key derivation
@ServiceActivator(inputChannel = "backoutBillingChannel")
@Transactional
public void handleBackoutBillingEvent(Message<BillingEvent> message) {
BillingEvent event = message.getPayload();
String stableKey = "billing:" + sha256(
event.getAccountId() + ":" + event.getBillingPeriod() + ":" + event.getAmountCents()
);
if (billingRecordRepo.existsByIdempotencyKey(stableKey)) {
log.info("Backout handler: event already billed, acking without re-charge: {}", event.getId());
return;
}
// If not yet billed (e.g. main queue truly exhausted without success), charge now
Charge charge = Charge.create(
ChargeCreateParams.builder()
.setAmount(event.getAmountCents())
.setCurrency("gbp")
.setCustomer(event.getCustomerId())
.build(),
RequestOptions.builder().setIdempotencyKey(stableKey).build()
);
billingRecordRepo.save(new BillingRecord(event, stableKey, charge.getId()));
log.warn("Backout handler: billed via backout path for event {}, charge {}", event.getId(), charge.getId());
}
Fix 2: Eliminate manual proactive-move scripts; rely on IBM MQ’s native BOTHRESH mechanism exclusively. Manual move scripts that fire at BackoutCount = BOTHRESH - 1 race with the active DMLC retry and can move a message to the BACKOUT_QUEUE while it is being successfully committed on the main queue. If you need proactive intervention, use IBM MQ’s BOTHRESH and BOQNAME queue attributes exclusively, and set BOTHRESH low enough that the main-queue retry loop is unlikely to succeed — do not run parallel move scripts that violate the assumption that main and BACKOUT paths are mutually exclusive at any point in time.
Fix 3: Use idempotent BACKOUT_QUEUE handling as an alert, not a re-billing path. The BACKOUT_QUEUE handler should check existsByIdempotencyKey(stableKey) before any Stripe call. If a BillingRecord already exists, the event was successfully processed on the main queue — log at INFO, ack the BACKOUT_QUEUE message, and exit. Only charge via Stripe if no BillingRecord exists. This makes the BACKOUT_QUEUE handler safe for both concurrent-processing races and the case where the main queue truly exhausted without billing successfully:
// Configure BACKOUT_QUEUE DMLC
@Bean
public DefaultMessageListenerContainer backoutBillingContainer(
ConnectionFactory connectionFactory,
PlatformTransactionManager transactionManager) {
DefaultMessageListenerContainer container = new DefaultMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setDestinationName("BILLING.EVENTS.BACKOUT");
container.setTransactionManager(transactionManager);
container.setConcurrentConsumers(1); // low concurrency for backout processing
container.setMaxConcurrentConsumers(2);
return container;
}
With all three fixes applied: the stable key makes every retry on the main queue and every BACKOUT_QUEUE handler call idempotent; the existsByIdempotencyKey guard prevents re-charging regardless of timing; BACKOUT_QUEUE monitoring provides operational visibility without creating duplicate charge risk.
Cross-layer comparison: Spring Integration IBM MQ vs other adapters in the series
| Behavior | SI IBM MQ (JmsInboundChannelAdapter polling) |
SI IBM MQ (DMLC push) | SI Solace (SolaceInboundChannelAdapter) |
SI GCP PubSub (PubSubInboundChannelAdapter) |
SI ActiveMQ (DMLC push) |
|---|---|---|---|---|---|
Ack/commit scope with DirectChannel |
After ServiceActivator returns (JmsTemplate commit after SessionCallback returns; DirectChannel.send() is synchronous) |
After onMessage() returns (DMLC controls session commit) |
After ServiceActivator returns for CLIENT_ACK with correct placement |
After ServiceActivator returns for MANUAL with correct placement |
After onMessage() returns (DMLC controls session commit) |
Ack/commit scope with ExecutorChannel |
After channel.send() returns (async enqueue) — before ServiceActivator runs |
After onMessage() returns (DMLC controls session commit; each concurrent consumer has its own session) |
AUTO_ACK: after channel.send() returns (same hazard). CLIENT_ACK: after explicit call from executor thread |
ackMode=AUTO: deadline timer runs from broker delivery, not handler completion |
After onMessage() returns (DMLC session scope) |
| TX + DB coordination | ChainedTransactionManager: commit-order determines which resource commits first; not XA |
ChainedTransactionManager or JmsTransactionManager alone; same commit-order risk |
No Spring TX resource adapter; acknowledge() is a direct JCSMP network call |
No Spring TX resource adapter; ack() is a direct HTTP call to GCP Pub/Sub API |
JmsTransactionManager; ActiveMQ supports XA via ActiveMQXAConnectionFactory |
| Retry on exception | Polling: JmsTemplate commits on return; retry is application responsibility (no DMLC-style rollback) | DMLC: sessionTransacted=true → session.rollback() on exception → IBM MQ redelivers; BackoutCount incremented |
CLIENT_ACK: REQUEUE status causes Solace redelivery; AUTO_ACK: no retry |
ackMode=AUTO: Pub/Sub redelivers after ack deadline expires; ackMode=MANUAL: nack causes redelivery |
DMLC: session.rollback() on exception → ActiveMQ redelivers; DLQ after maximumRedeliveries |
| Broker-side dead-message handling | IBM MQ BOTHRESH / BOQNAME: broker moves message to BACKOUT_QUEUE after BackoutCount ≥ BOTHRESH; independent of application retry logic |
Same as polling; DMLC rollbacks increment BackoutCount | Solace DMQ: broker moves message to DMQ after maxDeliveryCount is exceeded |
GCP Dead Letter Topic: Pub/Sub moves message to DLT after maxDeliveryAttempts |
ActiveMQ DLQ: broker moves message to DLQ after maximumRedeliveries |
| Shutdown / graceful stop behavior | JmsTemplate polling: stop() halts polling thread; in-flight executor tasks continue independently | DMLC: stop() waits for active onMessage() calls to complete (JMS 2.0 spec §8.7 for MessageConsumer.close()) |
stop() closes JCSMP FlowReceiver without draining QueueChannel buffer; unacked messages returned to broker immediately |
stop() terminates streaming pull subscription; in-flight messages have ack deadline for Pub/Sub to redeliver |
DMLC: stop() waits for active onMessage() per JMS spec; ActiveMQ returns unacked messages on disconnect |
ChainedTransactionManager commit-order behavior table
| Event | JMS-first: new ChainedTransactionManager(jmsTm, dbTm) |
DB-first: new ChainedTransactionManager(dbTm, jmsTm) |
|---|---|---|
Runtime exception in ServiceActivator |
Rollback: DB rolls back first, then JMS rolls back → IBM MQ redelivers M1 (correct behavior) | Rollback: JMS rolls back first, then DB rolls back → IBM MQ redelivers M1 (correct behavior) |
DB commit-phase failure (e.g., connection reset on COMMIT) |
JMS already committed → M1 permanently consumed; no BillingRecord → double-charge risk on reconciliation | DB commits first → fails → JMS has not committed yet → JMS rolls back → IBM MQ redelivers M1 → stable key makes retry safe |
JMS commit-phase failure (IBM MQ network issue on session.commit()) |
JMS commit fails → exception; DB not yet committed → ChainedTransactionManager rolls back DB → M1 on IBM MQ (not committed yet) → IBM MQ redelivers; stable key makes retry safe | DB already committed → JMS commit fails → ChainedTransactionManager tries to roll back DB (may fail if connection lost) → M1 on IBM MQ (not committed) → IBM MQ redelivers → stable key prevents double-charge |
| Recommendation | Do not use for IBM MQ billing consumers where DB commit failures under load are possible | Preferred: DB-first commit minimizes IBM MQ permanent-consumption window; stable key handles residual JMS-commit-failure redeliveries |
IBM MQ BACKOUT_THRESHOLD behavior and Spring Integration retry interaction table
| Configuration | DMLC sessionTransacted=true exception behavior |
IBM MQ BackoutCount tracking | BOTHRESH trigger | BACKOUT_QUEUE handler risk with UUID.randomUUID() |
|---|---|---|---|---|
DMLC + sessionTransacted=true + no retry advice + random key |
Each ServiceActivator exception → DMLC calls session.rollback() → IBM MQ increments BackoutCount → redelivers M1 |
Increments on every rollback | After N rollbacks, IBM MQ stops delivering to DMLC; moves M1 to BACKOUT_QUEUE | High: each DMLC retry generates a new Stripe charge; BACKOUT_QUEUE handler generates one more |
DMLC + sessionTransacted=true + RequestHandlerRetryAdvice (in-handler retries) + random key |
Retries within ServiceActivator; if all retries fail, exception propagates to DMLC → rollback; if one retry succeeds, session commits |
BackoutCount increments only on DMLC-level rollback (not on in-handler retries) | Only after DMLC-level rollback; in-handler retries do not increment BackoutCount | High within ServiceActivator (multiple Stripe charges per DMLC delivery); one more from BACKOUT_QUEUE on final exhaustion |
DMLC + sessionTransacted=true + no retry advice + stable key |
Each DMLC rollback causes redelivery; stable key → Stripe cache hit on retry; existsByIdempotencyKey guard on BillingRecord save |
Increments on every rollback (normal) | IBM MQ moves M1 to BACKOUT_QUEUE after BOTHRESH rollbacks | None: BACKOUT_QUEUE handler uses same stable key → Stripe returns cached charge; existsByIdempotencyKey short-circuits if BillingRecord already saved |
JmsInboundChannelAdapter polling + SESSION_TRANSACTED + ExecutorChannel + random key |
JmsTemplate commits before ServiceActivator runs; no DMLC-style rollback available | Never incremented (message committed before exception occurs) | Never triggered | N/A for BACKOUT_QUEUE (BackoutCount never increments); but reconciliation path generates second random key → ch_B |
Diagnostic checklist for Spring Integration IBM MQ + Stripe billing consumers
- Identify polling vs. push mode. Is the adapter a
JmsInboundChannelAdapter(polling viaJmsTemplate) or aJmsMessageDrivenEndpoint/DefaultMessageListenerContainer(push)? Polling mode does not support JMS-session-scoped transaction rollback on downstream handler exceptions — use DMLC push mode for transactional billing consumers. - Check output channel type. Is the output channel a
DirectChannel,ExecutorChannel,QueueChannel, orPriorityChannel? For JmsInboundChannelAdapter polling mode, onlyDirectChannelkeeps the handler within the JmsTemplate session scope. All other channel types decouple processing from the session commit. - Audit
ChainedTransactionManagerregistration order. IsJmsTransactionManagerregistered first or second? First means JMS commits before DB — DB commit-phase failures cause permanent IBM MQ message loss. Second (DB-first) is safer for billing consumers. - Check whether a separate BACKOUT_QUEUE consumer exists. Does the BACKOUT_QUEUE handler call
Stripe.Charge.create()? Does it useUUID.randomUUID()or a stable content-derived key? Does it checkexistsByIdempotencyKeybefore charging? - Verify IBM MQ BOTHRESH and BOQNAME queue attributes. What is the
BOTHRESHvalue on the billing queue? What is theBOQNAME? Is there a Spring Integration consumer on that queue? Does the BOQNAME consumer have idempotency-safe Stripe integration? - Check for manual BackoutCount monitoring scripts. Are there operational scripts that move messages from the main billing queue to the BACKOUT_QUEUE based on BackoutCount, outside of IBM MQ’s native BOTHRESH mechanism? These scripts race with active DMLC processing and can deliver the same billing event to both main and BACKOUT handlers concurrently.
- Audit Stripe idempotency key derivation across all billing paths. Does every code path that calls
Stripe.Charge.create()— main queue, BACKOUT_QUEUE, reconciliation job, manual reprocessing scripts — use the same stable content-derived key for the same billing event? - Test the reconciliation job’s Stripe key derivation. When the reconciliation job finds a Stripe charge without a matching
BillingRecordand republishes the event, does the republished message carry the same business payload fields (accountId,billingPeriod,amountCents) as the original? If the reconciliation job reconstructs the event from Stripe metadata and any field differs, the stable key will differ and a duplicate charge will occur.
Test patterns for Spring Integration IBM MQ + Stripe idempotency
Mode 1: polling adapter + ExecutorChannel + DB exception after Stripe commit
@Test
@DisplayName("JmsInboundChannelAdapter polling + ExecutorChannel: JmsTemplate commits before ServiceActivator runs; DB failure after Stripe commit does not cause redelivery; stable key prevents ch_B on reconciliation republish")
void pollingAdapterSessionTransactedExecutorChannel_dbExceptionAfterStripeCommit_noDuplicateCharge() throws Exception {
BillingEvent event = BillingEvent.builder()
.accountId("acc_GB_019")
.billingPeriod("2026-10")
.amountCents(4900L)
.customerId("cust_GB_019")
.build();
String stableKey = "billing:" + sha256(
event.getAccountId() + ":" + event.getBillingPeriod() + ":" + event.getAmountCents()
);
// First delivery: Stripe succeeds, DB fails; with polling adapter, JmsTemplate already committed
// so M1 won't be redelivered by IBM MQ
stubFor(post(urlEqualTo("/v1/charges"))
.withHeader("Idempotency-Key", equalTo(stableKey))
.inScenario("billing-retry")
.whenScenarioStateIs(STARTED)
.willReturn(aResponse()
.withStatus(200)
.withHeader("Content-Type", "application/json")
.withBody("""
{"id":"ch_A","amount":4900,"currency":"gbp","status":"succeeded"}
"""))
.willSetStateTo("stripe-charged"));
// Simulate DB failure on first attempt
doThrow(new CannotGetJdbcConnectionException("pool exhausted"))
.doNothing()
.when(billingRecordRepo).save(any(BillingRecord.class));
// Put M1 on IBM MQ queue (via Testcontainers IBM MQ)
ibmMqHelper.sendMessage("BILLING.EVENTS.QUEUE", event);
// Wait for executor to process
Thread.sleep(2000);
// Verify Stripe called exactly once (first delivery; JmsTemplate committed M1)
verify(1, postRequestedFor(urlEqualTo("/v1/charges"))
.withHeader("Idempotency-Key", equalTo(stableKey)));
// Verify no BillingRecord (DB failed; with stable key, reconciliation republish will be idempotent)
assertThat(billingRecordRepo.findByIdempotencyKey(stableKey)).isEmpty();
// Simulate reconciliation republish M2 (same business payload)
ibmMqHelper.sendMessage("BILLING.EVENTS.QUEUE", event);
Thread.sleep(2000);
// Stripe called again with same stableKey — returns ch_A from idempotency cache
verify(2, postRequestedFor(urlEqualTo("/v1/charges"))
.withHeader("Idempotency-Key", equalTo(stableKey)));
// Now BillingRecord saved (DB recovered on second attempt)
assertThat(billingRecordRepo.findByIdempotencyKey(stableKey))
.isPresent()
.map(BillingRecord::getChargeId)
.hasValue("ch_A"); // Same charge ID from Stripe cache — not ch_B
}
Mode 2: ChainedTransactionManager DB-first ordering + JMS commit failure redelivery
@Test
@DisplayName("ChainedTransactionManager DB-first: JMS commit failure causes IBM MQ redelivery; stable key prevents duplicate charge on retry")
void chainedTransactionManagerDbFirst_jmsCommitFailure_ibmMqRedelivers_stableKeyPreventsDoubleCharge()
throws Exception {
BillingEvent event = BillingEvent.builder()
.accountId("acc_GB_033")
.billingPeriod("2026-10")
.amountCents(9900L)
.customerId("cust_GB_033")
.build();
String stableKey = "billing:" + sha256(
event.getAccountId() + ":" + event.getBillingPeriod() + ":" + event.getAmountCents()
);
// Stripe responds successfully on both attempts
stubFor(post(urlEqualTo("/v1/charges"))
.withHeader("Idempotency-Key", equalTo(stableKey))
.willReturn(aResponse()
.withStatus(200)
.withHeader("Content-Type", "application/json")
.withBody("""
{"id":"ch_A","amount":9900,"currency":"gbp","status":"succeeded"}
""")));
// Simulate JMS commit failure on first attempt (IBM MQ network blip)
doThrow(new JmsException("IBM MQ commit failed: network reset") {})
.doNothing()
.when(jmsTransactionManager).commit(any());
ibmMqHelper.sendMessage("BILLING.EVENTS.QUEUE", event);
Thread.sleep(3000); // Allow redelivery
// DB committed successfully on first attempt; JMS commit failed; IBM MQ redelivered M1
// Second attempt: DB finds existsByIdempotencyKey = true (BillingRecord saved in first DB commit)
// ServiceActivator short-circuits; Stripe NOT called on second delivery
verify(1, postRequestedFor(urlEqualTo("/v1/charges"))
.withHeader("Idempotency-Key", equalTo(stableKey)));
assertThat(billingRecordRepo.findByIdempotencyKey(stableKey))
.isPresent()
.map(BillingRecord::getChargeId)
.hasValue("ch_A");
}
Mode 3: BACKOUT_QUEUE concurrent delivery with main-queue success
@Test
@DisplayName("BACKOUT_QUEUE: concurrent delivery with main-queue success; stable key in BACKOUT handler prevents ch_B")
void backoutQueueConcurrentWithMainSuccess_stableKeyPreventsDoubleCharge() throws Exception {
BillingEvent event = BillingEvent.builder()
.accountId("acc_GB_058")
.billingPeriod("2026-10")
.amountCents(14900L)
.customerId("cust_GB_058")
.build();
String stableKey = "billing:" + sha256(
event.getAccountId() + ":" + event.getBillingPeriod() + ":" + event.getAmountCents()
);
// Stripe responds with ch_A on first call; returns ch_A from cache on subsequent calls with same key
stubFor(post(urlEqualTo("/v1/charges"))
.withHeader("Idempotency-Key", equalTo(stableKey))
.willReturn(aResponse()
.withStatus(200)
.withHeader("Content-Type", "application/json")
.withBody("""
{"id":"ch_A","amount":14900,"currency":"gbp","status":"succeeded"}
""")));
// Send M1 to both main queue and BACKOUT_QUEUE simultaneously
// (simulating the scenario where a move script placed a copy on BACKOUT_QUEUE
// before the main queue committed successfully)
ibmMqHelper.sendMessage("BILLING.EVENTS.QUEUE", event);
ibmMqHelper.sendMessage("BILLING.EVENTS.BACKOUT", event);
Thread.sleep(4000);
// Stripe called at most twice (once from main, once from backout)
// but ALWAYS with the same stableKey → Stripe returns ch_A both times
verify(lessThanOrExactly(2), postRequestedFor(urlEqualTo("/v1/charges"))
.withHeader("Idempotency-Key", equalTo(stableKey)));
// Only one BillingRecord exists (second save is idempotent via existsByIdempotencyKey guard)
assertThat(billingRecordRepo.countByIdempotencyKey(stableKey)).isEqualTo(1);
assertThat(billingRecordRepo.findByIdempotencyKey(stableKey))
.isPresent()
.map(BillingRecord::getChargeId)
.hasValue("ch_A");
}
What Keybrake adds to Spring Integration IBM MQ billing pipelines
The three failure modes described above all share a structural root cause: the Stripe idempotency key is generated at handler invocation time (UUID.randomUUID()) rather than derived from the billing event’s business identity. A stable content-derived key prevents duplicate charges across all three paths — polling adapter retries, ChainedTransactionManager commit-phase failures, and BACKOUT_QUEUE concurrent processing. The fixes described above implement this at the application layer.
At the infrastructure layer, Keybrake adds a complementary defense: a scoped API-key proxy that sits between your Spring Integration billing consumer and the Stripe API, enforcing per-account and per-billing-period daily spend caps. When a stuck retry loop or an unexpected BACKOUT_QUEUE re-processing path generates multiple Stripe Charge.create() calls for the same customer in a short window, Keybrake’s policy layer blocks the second call and logs it as a policy violation — providing an audit trail that is visible in the Keybrake dashboard before the customer disputes the charge. The proxy-layer enforcement is separate from the idempotency key fix, not a substitute for it, but it creates a second defense line that catches the cases where stable keys are correct but retry logic, backout handlers, or reconciliation jobs produce unexpected call volumes.
Audit your agent’s Stripe calls before the next BACKOUT_QUEUE incident
Keybrake gives your Spring Integration billing consumer a scoped proxy key with per-account spend caps, endpoint allowlists, and a real-time audit log of every Stripe call — so you catch retry storms and BACKOUT_QUEUE double-charges in the dashboard before your customers see them on their cards.
Related posts in this series
- Spring Integration Solace Stripe Integration:
AcknowledgeMode.AUTO_ACK+ExecutorChannel,acknowledge(ACCEPT)inside@Transactional,stop()graceful shutdown +QueueChannel - Spring Integration Google PubSub Stripe Integration:
ackMode=AUTO+ ack deadline expiry,ack()inside@Transactional,RequestHandlerRetryAdvice+UUID.randomUUID() - Spring Integration Azure Service Bus Stripe Integration:
ServiceBusInboundChannelAdapter+ExecutorChannel, transactional ack semantics, shutdown race - Spring Integration Azure Event Hubs Stripe Integration: offset commit on poll return,
@Transactional+ Event Hubs client ack, checkpoint-on-error - Spring Cloud Stream Solace Binder Stripe Integration:
nackOnError+ redelivery,transactedSessioncommit ordering, DLQ re-ingestion - Spring Cloud Stream Google PubSub Binder Stripe Integration:
nackOnError+ SCS retry exhaustion → GCP redelivery,max-ack-extension-period=0+ 3DS,deadLetterPolicy× SCSmax-attempts - Spring Boot
@Transactional+ Feign client retry Stripe Integration: retry generates UUID_B through UUID_C - Spring Retry Stripe Integration:
@Retryable+UUID.randomUUID()generates a new key on each attempt