Apache CXF and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance
Apache CXF’s AbstractPhaseInterceptor.handleMessage() is called once per outbound CXF message — not once per logical billing operation. A retry loop that re-calls the CXF proxy method re-runs the full phase interceptor chain, including an idempotency interceptor that calls UUID.randomUUID() inside handleMessage(). The initial proxy invocation sends the request to Stripe and creates ch_A before a ProcessingException wrapping a SocketTimeoutException from a stale HTTP conduit connection; the retry’s handleMessage() evaluates a new UUID, causing Stripe to create ch_B. Three Apache CXF-specific Stripe billing failure modes: an AbstractPhaseInterceptor<Message> computes UUID.randomUUID() inside handleMessage() — subtler variant: CXF’s async WebClient with an InvocationCallback.failed() retry that calls webClient.async().post(entity, newCallback) again in failed(), re-invoking the full phase chain with a fresh UUID per failed() execution; a JAX-WS SOAPHandler<SOAPMessageContext> registered on a CXF-generated dynamic proxy computes UUID.randomUUID() inside handleMessage() — Resilience4j @Retry re-invokes the service bean method, which makes another JAX-WS proxy call, firing the handler chain fresh per proxy invocation — subtler variant: @Retry on the CDI bean method calling a CXF proxy injected via @WebServiceRef — CXF’s dynamic proxy wraps the full handler chain execution per method call, handler chain fires per @Retry re-invocation; and a per-JVM ScheduledExecutorService billing job runs independently on all Kubernetes replicas — with replicas:3, all three pods pass the concurrent hasCompletedForPeriod() check before any pod commits the billing-started record (TOCTOU race), all three generate distinct UUID.randomUUID() per customer, and all three create ch_A, ch_B, ch_C per customer per billing period — subtler variant: ShedLock @SchedulerLock with lockAtMostFor shorter than billing P99.
This post covers all three failure modes with Java code, content-hash idempotency keys stable across handleMessage() re-invocations and JAX-WS handler chain re-executions, the Message context property bag as the key-passing mechanism from calling code to the CXF interceptor, BindingProvider.getRequestContext() as the key-passing mechanism for JAX-WS handlers, pg_try_advisory_lock() for cross-pod scheduler serialization, and pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For the ClientRequestFilter.filter() pattern in JAX-RS clients, see the Jersey and JAX-RS Stripe Integration post. For the Feign RequestInterceptor.apply() pattern, see the Feign and Spring Cloud OpenFeign Stripe Integration post. For the Apache HttpClient 5 HttpRequestInterceptor pattern, see the Apache HttpClient 5 and Stripe Integration post.
Failure mode 1: AbstractPhaseInterceptor.handleMessage() computes UUID.randomUUID() — application-level retry loop re-calls the CXF proxy method — full phase interceptor chain re-runs per proxy invocation — initial call creates ch_A before ProcessingException — retry creates ch_B
Apache CXF’s interceptor model organizes outbound processing into a PhaseInterceptorChain that executes registered interceptors in phase order before each outbound message is transmitted. Standard phases for outbound client interceptors include Phase.SETUP, Phase.PRE_LOGICAL, Phase.USER_LOGICAL, Phase.MARSHAL, Phase.PRE_STREAM, and Phase.WRITE. An AbstractPhaseInterceptor<Message> registered in any of these phases executes its handleMessage(Message message) method on every outbound message generated by that CXF client. The critical behavioral point is that a “message” in CXF corresponds to one HTTP request attempt, not one logical business operation. Each call to the CXF proxy method triggers a new Exchange and a new Message traversing the interceptor chain from the beginning.
This becomes a duplicate-charge risk when a retry loop re-calls the CXF proxy method on failure, and the registered interceptor computes the idempotency key inside handleMessage():
// StripeIdempotencyInterceptor.java
// UNSAFE: UUID.randomUUID() computed inside handleMessage() — called per outbound CXF message.
// A retry loop that re-invokes the CXF proxy method re-runs this interceptor
// with a fresh UUID on each attempt — ch_B created on the retry.
import org.apache.cxf.interceptor.Fault;
import org.apache.cxf.message.Message;
import org.apache.cxf.phase.AbstractPhaseInterceptor;
import org.apache.cxf.phase.Phase;
import org.apache.cxf.helpers.CastUtils;
import java.util.*;
public class StripeIdempotencyInterceptor extends AbstractPhaseInterceptor<Message> {
public StripeIdempotencyInterceptor() {
super(Phase.PRE_STREAM);
}
@Override
public void handleMessage(Message message) throws Fault {
// UNSAFE: UUID.randomUUID() called per handleMessage() invocation.
// First proxy call: UUID = "4c8a3b1d-2e5f-4b7e-9c0a-1a2b3c4d5e6f" → ch_A
// Retry proxy call: UUID = "d7e2f4a6-3b8c-4f1e-0d9a-2b3c4d5e6f7a" → ch_B ← duplicate
Map<String, List<String>> headers = CastUtils.cast(
message.get(Message.PROTOCOL_HEADERS));
if (headers == null) {
headers = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);
message.put(Message.PROTOCOL_HEADERS, headers);
}
headers.put("Idempotency-Key",
Collections.singletonList(UUID.randomUUID().toString()));
}
}
// BillingService.java — UNSAFE retry loop: each iteration re-invokes the CXF proxy method.
// The AbstractPhaseInterceptor fires per proxy method call —
// StripeIdempotencyInterceptor.handleMessage() evaluates a new UUID on every attempt.
import org.apache.cxf.jaxws.JaxWsProxyFactoryBean;
public class BillingService {
private final StripeBillingPort stripeProxy;
public BillingService() {
JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
factory.setServiceClass(StripeBillingPort.class);
factory.setAddress("https://api.stripe.com/v1/billing");
factory.getOutInterceptors().add(new StripeIdempotencyInterceptor()); // UNSAFE
this.stripeProxy = (StripeBillingPort) factory.create();
}
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
// UNSAFE: retry loop re-invokes stripeProxy.charge(), which triggers a new
// CXF Exchange → new outbound Message → handleMessage() fires fresh per attempt.
// Attempt 1: handleMessage() → UUID "4c8a3b1d-..." → Stripe creates ch_A
// → ProcessingException (SocketTimeoutException on stale conduit connection)
// Attempt 2: handleMessage() → UUID "d7e2f4a6-..." → Stripe sees new UUID
// → ch_B created — customer charged twice.
int maxAttempts = 3;
Exception lastException = null;
for (int attempt = 1; attempt <= maxAttempts; attempt++) {
try {
return stripeProxy.charge(customerId, amountCents, billingPeriod);
} catch (javax.xml.ws.WebServiceException e) {
lastException = e;
if (attempt < maxAttempts) {
try { Thread.sleep(150L * attempt); }
catch (InterruptedException ie) { Thread.currentThread().interrupt(); break; }
}
}
}
throw new BillingException("Billing failed after " + maxAttempts + " attempts", lastException);
}
}
The failure scenario: on attempt 1, stripeProxy.charge(customerId, amountCents, billingPeriod) is called. CXF creates a new Exchange and outbound Message. The PhaseInterceptorChain executes in phase order. At Phase.PRE_STREAM, StripeIdempotencyInterceptor.handleMessage() fires. UUID.randomUUID() evaluates to "4c8a3b1d-..." and is placed in Message.PROTOCOL_HEADERS. CXF’s HTTP conduit sends the request to Stripe with Idempotency-Key: 4c8a3b1d-.... Stripe processes the charge. ch_A is committed in Stripe’s ledger. The HTTP conduit was using a keep-alive connection from CXF’s internal connection pool. The connection had been idle for 61 seconds. The upstream AWS ALB’s idle timeout is 60 seconds. The ALB silently sent a TCP RST. CXF’s HTTP conduit reads the response after Stripe committed ch_A but before the RST was processed — it gets a SocketTimeoutException. CXF wraps this in a ProcessingException (or WebServiceException on the JAX-WS layer) and propagates it to the caller. The retry loop catches the exception, sleeps 150 ms, and increments the attempt counter.
On attempt 2, stripeProxy.charge() is called again. CXF creates a new Exchange and a new outbound Message with a fresh, empty header map. The PhaseInterceptorChain runs from the beginning. StripeIdempotencyInterceptor.handleMessage() fires again — it is a new handleMessage() invocation with a new Message instance. UUID.randomUUID() evaluates to "d7e2f4a6-...". The retry request reaches Stripe with Idempotency-Key: d7e2f4a6-.... Stripe looks up "d7e2f4a6-..." in its idempotency cache, finds nothing, processes the charge again, and creates ch_B. The customer is charged twice.
The failure is structurally invisible in code review because the interceptor looks correct in isolation: it is a clean AbstractPhaseInterceptor subclass with a clear single purpose. The invariant it violates is not in the CXF Javadoc: handleMessage() is documented as firing “before the message is sent”, which is precisely what it does. The developer who wrote the retry loop treats stripeProxy.charge() as a business-level operation and assumes the infrastructure plumbing (headers, authentication, idempotency) is configured once per client, not re-evaluated per call. In most middleware frameworks that assumption holds. In CXF’s phase interceptor model, it does not.
The subtler variant: CXF async WebClient with InvocationCallback.failed() retry — failed() re-calls webClient.async().post(entity, newCallback) — full phase interceptor chain fires fresh per post() call — ch_B without any change to the idempotency interceptor
CXF’s WebClient API provides a fluent JAX-RS-style HTTP client that shares CXF’s interceptor infrastructure. An async billing pattern that uses WebClient.async().post(entity, callback) with an InvocationCallback<Response> that retries in failed() suffers the same structural problem: each new webClient.async().post(entity, newCallback) call is a new CXF request invocation, driving a new outbound Message through the full phase chain:
// UNSAFE: InvocationCallback.failed() retries by re-calling webClient.async().post().
// Each new webClient.async().post() call is a new CXF request — phase chain runs fresh.
// StripeIdempotencyInterceptor.handleMessage() evaluates UUID.randomUUID() per chain execution.
import org.apache.cxf.jaxrs.client.WebClient;
import javax.ws.rs.client.InvocationCallback;
import java.util.concurrent.CompletableFuture;
public class AsyncBillingService {
private final WebClient webClient;
public AsyncBillingService() {
this.webClient = WebClient.create(
"https://api.stripe.com/v1/charges",
Collections.singletonList(new StripeIdempotencyInterceptor()) // UNSAFE
);
}
public CompletableFuture<ChargeResponse> chargeAsync(
String customerId, long amountCents, String billingPeriod) {
CompletableFuture<ChargeResponse> result = new CompletableFuture<>();
submitWithRetry(buildForm(customerId, amountCents, billingPeriod), result, 2);
return result;
}
private void submitWithRetry(
javax.ws.rs.core.Form form,
CompletableFuture<ChargeResponse> result,
int retriesLeft) {
// UNSAFE: each call to webClient.async().post() triggers a new CXF message.
// StripeIdempotencyInterceptor.handleMessage() fires fresh per post() call.
// Initial call: UUID "4c8a3b1d-..." → Stripe commits ch_A before SocketTimeoutException
// failed() fires → retriesLeft=1 → submitWithRetry() calls post() again
// New CXF message: UUID "d7e2f4a6-..." → Stripe creates ch_B ← duplicate
webClient.async().post(
javax.ws.rs.client.Entity.form(form),
new InvocationCallback<javax.ws.rs.core.Response>() {
@Override
public void completed(javax.ws.rs.core.Response response) {
if (response.getStatus() == 200 || response.getStatus() == 201) {
result.complete(response.readEntity(ChargeResponse.class));
} else if (response.getStatus() >= 400 && response.getStatus() < 500) {
result.completeExceptionally(
new BillingException("Stripe client error: " + response.getStatus()));
} else if (retriesLeft > 0) {
submitWithRetry(form, result, retriesLeft - 1); // ← re-calls post() → fresh UUID
} else {
result.completeExceptionally(
new BillingException("Stripe 5xx exhausted retries"));
}
}
@Override
public void failed(Throwable throwable) {
if (retriesLeft > 0) {
submitWithRetry(form, result, retriesLeft - 1); // ← re-calls post() → fresh UUID
} else {
result.completeExceptionally(throwable);
}
}
});
}
}
The coupling between the async retry pattern and the interceptor is invisible at the point of registration. The developer who wrote submitWithRetry() may have understood that each webClient.async().post() call is a new HTTP request — they may even have intended that, expecting the infrastructure to supply idempotency headers automatically. The developer who wrote StripeIdempotencyInterceptor may have written it for synchronous use, where the pattern was safe. When the async client is registered with the same interceptor class, the interaction produces a fresh UUID per post() call without any visible change to either component.
The fix for failure mode 1
The idempotency key must be computed once per billing operation, before any CXF proxy invocation or WebClient.post() call, and stored where the interceptor can read it across all retry attempts. CXF’s Message context property bag — the Message instance passed to handleMessage() — is per-message. For a retry loop that creates new Message instances per attempt, the right carrier is the RequestContext on the JAX-WS BindingProvider, or a thread-local scoped to the billing operation for the WebClient case. For the WebClient async case, compute the key once before the first post() call and pass it explicitly through the callback closure.
// Safe StripeIdempotencyInterceptor.java — reads caller-supplied key from Message properties.
// Falls back to RequestContext (set by calling code before each logical billing operation).
// Never calls UUID.randomUUID() inside handleMessage().
import org.apache.cxf.interceptor.Fault;
import org.apache.cxf.message.Message;
import org.apache.cxf.phase.AbstractPhaseInterceptor;
import org.apache.cxf.phase.Phase;
import org.apache.cxf.helpers.CastUtils;
public class StripeIdempotencyInterceptor extends AbstractPhaseInterceptor<Message> {
public static final String IDEMPOTENCY_KEY_PROPERTY = "stripe.idempotency.key";
public StripeIdempotencyInterceptor() {
super(Phase.PRE_STREAM);
}
@Override
public void handleMessage(Message message) throws Fault {
// Safe: read the key set by calling code before any proxy invocation.
// The key comes from RequestContext (JAX-WS BindingProvider) or from
// a property set directly on the outbound message before chain entry.
String key = (String) message.get(IDEMPOTENCY_KEY_PROPERTY);
if (key == null) {
// Check the exchange — BindingProvider.getRequestContext() values
// are propagated into the Exchange by CXF's binding layer.
key = (String) message.getExchange().get(IDEMPOTENCY_KEY_PROPERTY);
}
if (key == null || key.isBlank()) {
throw new Fault(new IllegalStateException(
"stripe.idempotency.key not set in RequestContext — " +
"compute stableKey() before invoking the proxy and set it via " +
"((BindingProvider) proxy).getRequestContext().put(IDEMPOTENCY_KEY_PROPERTY, key)"));
}
Map<String, List<String>> headers = CastUtils.cast(
message.get(Message.PROTOCOL_HEADERS));
if (headers == null) {
headers = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);
message.put(Message.PROTOCOL_HEADERS, headers);
}
headers.put("Idempotency-Key", Collections.singletonList(key));
}
}
// Safe BillingService.java — computes stableKey once before any proxy invocation.
// Retry loop passes the same key via RequestContext on every attempt.
import javax.xml.ws.BindingProvider;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
public class BillingService {
private final StripeBillingPort stripeProxy;
public BillingService() {
JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
factory.setServiceClass(StripeBillingPort.class);
factory.setAddress("https://api.stripe.com/v1/billing");
factory.getOutInterceptors().add(new StripeIdempotencyInterceptor()); // safe: reads from context
this.stripeProxy = (StripeBillingPort) factory.create();
}
private String stableKey(String customerId, String billingPeriod) {
try {
String raw = customerId + ":" + billingPeriod + ":cxf-billing";
byte[] hash = MessageDigest.getInstance("SHA-256")
.digest(raw.getBytes(StandardCharsets.UTF_8));
StringBuilder sb = new StringBuilder();
for (int i = 0; i < 16; i++) {
sb.append(String.format("%02x", hash[i]));
}
return sb.toString(); // 32-character stable hex string
} catch (Exception e) {
throw new RuntimeException("SHA-256 unavailable", e);
}
}
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
// Compute the stable key ONCE, before any proxy invocation.
String idempotencyKey = stableKey(customerId, billingPeriod);
// Set the key in the JAX-WS RequestContext.
// CXF propagates RequestContext values into the Exchange before the interceptor chain runs.
// All retry iterations read the same idempotencyKey from the same RequestContext.
((BindingProvider) stripeProxy).getRequestContext()
.put(StripeIdempotencyInterceptor.IDEMPOTENCY_KEY_PROPERTY, idempotencyKey);
// Pre-flight: attempt to claim this billing slot atomically.
// Returns true if this invocation is the first to claim (customer_id, billing_period).
// Returns false if a prior invocation already claimed it (safe: return existing charge).
if (!claimBillingSlot(customerId, billingPeriod, idempotencyKey)) {
return lookupExistingCharge(customerId, billingPeriod);
}
int maxAttempts = 3;
Exception lastException = null;
for (int attempt = 1; attempt <= maxAttempts; attempt++) {
try {
// Safe: each retry reads the same idempotencyKey from RequestContext.
// handleMessage() reads it from the Exchange, not from UUID.randomUUID().
return stripeProxy.charge(customerId, amountCents, billingPeriod);
} catch (javax.xml.ws.WebServiceException e) {
lastException = e;
if (attempt < maxAttempts) {
try { Thread.sleep(150L * attempt); }
catch (InterruptedException ie) { Thread.currentThread().interrupt(); break; }
}
}
}
throw new BillingException("Billing failed after " + maxAttempts + " attempts", lastException);
}
private boolean claimBillingSlot(String customerId, String billingPeriod, String idempotencyKey) {
// INSERT INTO billing_slots (customer_id, billing_period, idempotency_key, claimed_at)
// VALUES (?, ?, ?, NOW())
// ON CONFLICT (customer_id, billing_period) DO NOTHING
// Returns true if INSERT succeeded (this invocation owns the slot).
// Returns false if a prior INSERT already claimed it (safe: skip, return cached charge).
return db.executeInsertOrNothing(
"INSERT INTO billing_slots (customer_id, billing_period, idempotency_key, claimed_at) " +
"VALUES (?, ?, ?, NOW()) ON CONFLICT (customer_id, billing_period) DO NOTHING",
customerId, billingPeriod, idempotencyKey);
}
}
// Safe async WebClient: compute key once before any webClient.post() call.
// Pass idempotencyKey as a final parameter through the callback closure.
public CompletableFuture<ChargeResponse> chargeAsync(
String customerId, long amountCents, String billingPeriod) {
// Compute the stable key once before any post() call.
// Captured as a final local variable — all InvocationCallback.failed() retries use the same value.
final String idempotencyKey = stableKey(customerId, billingPeriod);
CompletableFuture<ChargeResponse> result = new CompletableFuture<>();
submitWithRetry(buildForm(customerId, amountCents), idempotencyKey, result, 2);
return result;
}
private void submitWithRetry(
javax.ws.rs.core.Form form,
String idempotencyKey, // same value on every retry
CompletableFuture<ChargeResponse> result,
int retriesLeft) {
// Safe: set the header explicitly on the WebClient for this specific call.
// WebClient.header() returns the same WebClient; the header value is stable.
webClient.header("Idempotency-Key", idempotencyKey)
.async()
.post(javax.ws.rs.client.Entity.form(form),
new InvocationCallback<javax.ws.rs.core.Response>() {
@Override
public void completed(javax.ws.rs.core.Response response) { /* ... */ }
@Override
public void failed(Throwable throwable) {
if (retriesLeft > 0) {
// Safe: same idempotencyKey passed through — not re-computed.
submitWithRetry(form, idempotencyKey, result, retriesLeft - 1);
} else {
result.completeExceptionally(throwable);
}
}
});
}
idempotencyKey is computed by stableKey(customerId, billingPeriod) before the first proxy invocation or post() call. For the JAX-WS path, it is stored in the CXF RequestContext map, which CXF propagates into the Exchange before the interceptor chain runs. The safe StripeIdempotencyInterceptor.handleMessage() reads it from the Exchange instead of calling UUID.randomUUID(). On the second and third retry attempts, the RequestContext still holds the same idempotencyKey value because it was set once before the loop and never overwritten inside the loop. Stripe receives the same Idempotency-Key header on every attempt and returns the cached ch_A result after the first successful commit. For the async WebClient path, idempotencyKey is captured as a final local variable in the outer scope and passed explicitly as a parameter to every submitWithRetry() invocation, so each new webClient.header(...).async().post() call sets the same header value.
Failure mode 2: JAX-WS SOAPHandler<SOAPMessageContext>.handleMessage() computes UUID.randomUUID() — Resilience4j @Retry re-invokes the containing service method — JAX-WS handler chain fires per proxy method invocation — initial invocation creates ch_A before WebServiceException — first re-invocation creates ch_B
Apache CXF generates JAX-WS dynamic proxies for service endpoints defined via @WebService interfaces or WSDL. Developers extend these proxies with JAX-WS handler chains: a list of SOAPHandler or LogicalHandler implementations registered via @HandlerChain annotation pointing to a handler chain XML file, or programmatically via ((BindingProvider) proxy).getBinding().setHandlerChain(handlers). The JAX-WS specification mandates that the handler chain executes for every outbound message. In CXF’s implementation, “every outbound message” means every call to the JAX-WS proxy method interface. A SOAPHandler that generates an idempotency key inside handleMessage() executes its UUID generation once per proxy method call.
The structural failure: Resilience4j @Retry on the service bean method containing the JAX-WS proxy call re-invokes the entire method body on each retry attempt. Each re-invocation calls the JAX-WS proxy method, triggering a new JAX-WS message exchange and a fresh handler chain execution. UUID.randomUUID() inside SOAPHandler.handleMessage() evaluates per chain execution:
// StripeIdempotencyHandler.java — SOAPHandler that computes UUID in handleMessage().
// UNSAFE: UUID.randomUUID() per outbound handleMessage() invocation.
// Fires fresh on every JAX-WS proxy method call — including every @Retry re-invocation
// of the containing service method, since each re-invocation calls the proxy again.
import javax.xml.namespace.QName;
import javax.xml.ws.handler.MessageContext;
import javax.xml.ws.handler.soap.SOAPHandler;
import javax.xml.ws.handler.soap.SOAPMessageContext;
import javax.xml.soap.SOAPHeader;
import java.util.Set;
public class StripeIdempotencyHandler implements SOAPHandler<SOAPMessageContext> {
private static final QName IDEMPOTENCY_QNAME =
new QName("https://api.stripe.com/headers", "IdempotencyKey");
@Override
public boolean handleMessage(SOAPMessageContext context) {
Boolean outbound = (Boolean) context.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY);
if (!Boolean.TRUE.equals(outbound)) return true;
try {
// UNSAFE: UUID.randomUUID() evaluated per handleMessage() call.
// First proxy call (attempt 1): UUID "4c8a3b1d-..." → ch_A
// → WebServiceException (transient Stripe error)
// @Retry re-invocation (attempt 2): UUID "d7e2f4a6-..." → ch_B ← duplicate
SOAPHeader header = context.getMessage().getSOAPPart().getEnvelope().getHeader();
if (header == null) {
header = context.getMessage().getSOAPPart().getEnvelope().addHeader();
}
header.addChildElement(IDEMPOTENCY_QNAME)
.setTextContent(UUID.randomUUID().toString());
} catch (javax.xml.soap.SOAPException e) {
throw new javax.xml.ws.WebServiceException(
"Failed to add idempotency key to SOAP header", e);
}
return true;
}
@Override
public boolean handleFault(SOAPMessageContext context) { return true; }
@Override
public void close(MessageContext context) { }
@Override
public Set<QName> getHeaders() { return Collections.emptySet(); }
}
// BillingBean.java — @Retry re-invokes chargeCustomer() — proxy call fires SOAPHandler fresh.
import io.github.resilience4j.retry.annotation.Retry;
import org.springframework.stereotype.Service;
@Service
public class BillingBean {
@javax.annotation.Resource
private StripeBillingService stripeService; // @WebServiceRef JAX-WS proxy
// UNSAFE: @Retry re-invokes chargeCustomer() method body on each retry attempt.
// chargeCustomer() calls stripeService.getStripeBillingPort().charge(...).
// Each proxy.charge() call fires StripeIdempotencyHandler.handleMessage().
// UUID.randomUUID() inside handleMessage() evaluates per re-invocation.
// Attempt 1: UUID "4c8a3b1d-..." → POST /v1/charges → ch_A → WebServiceException
// Attempt 2: UUID "d7e2f4a6-..." → POST /v1/charges → ch_B ← duplicate charge
@Retry(name = "stripeBilling", fallbackMethod = "billingFallback")
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
StripeBillingPort proxy = stripeService.getStripeBillingPort();
return proxy.charge(customerId, amountCents, billingPeriod);
}
public ChargeResponse billingFallback(String customerId, String billingPeriod,
long amountCents, Throwable t) {
throw new BillingException("Billing failed for " + customerId, t);
}
}
The failure is structurally identical to the Feign RequestInterceptor.apply() pattern, but the mechanism is different in a way that makes it harder to detect: the JAX-WS handler chain is configured in an XML file (handler-chain.xml) or in the @HandlerChain annotation on the proxy interface or on the @WebServiceRef injection site. The handler chain definition is separated from the calling service code by at least one layer of indirection. The developer writing BillingBean.chargeCustomer() annotated with @Retry does not see the handler chain definition — they see only a method call on an injected JAX-WS proxy. The handler chain file may have been written by a different team or in a different repository module. The failure emerges from the combination of two independently-correct choices: “use the handler chain to set the idempotency key” and “use @Retry on the service method for transient error tolerance.”
The subtler variant: @Retry on the CDI bean method calling a CXF proxy injected via @WebServiceRef — CXF generates a thread-safe dynamic proxy that wraps the full handler chain execution per method call — Resilience4j RetryAspect re-invokes the annotated method — each re-invocation fires the handler chain fresh — fresh UUID per @Retry attempt
The failure mode compounds when the JAX-WS proxy is injected via Spring’s @WebServiceRef or as a CXF-generated bean in a Spring XML configuration. CXF creates a thread-safe dynamic proxy backed by the handler chain. The handler chain is invoked synchronously on each proxy method call. Resilience4j’s RetryAspect is an AOP proxy around the @Service bean that intercepts the annotated method and calls MethodInvocation.proceed() for each retry attempt. Each proceed() call re-enters the BillingBean.chargeCustomer() method body from the first statement, which calls stripeService.getStripeBillingPort().charge() — a fresh JAX-WS proxy method invocation — which triggers a fresh handler chain execution:
// Configuration: CXF-generated JAX-WS proxy wired as a Spring bean.
// The proxy is a thread-safe singleton that wraps StripeIdempotencyHandler per method call.
// application-context.xml (or equivalent @Configuration):
//
// <jaxws:client id="stripePort"
// serviceClass="com.billing.StripeBillingPort"
// address="https://api.stripe.com/v1/billing">
// <jaxws:handlers>
// <bean class="com.billing.StripeIdempotencyHandler"/> <!-- UNSAFE: UUID in handleMessage() -->
// <bean class="com.billing.StripeAuthHandler"/>
// </jaxws:handlers>
// </jaxws:client>
// BillingBean.java (revised with injection):
@Service
public class BillingBean {
@Autowired
private StripeBillingPort stripePort; // CXF-generated proxy; StripeIdempotencyHandler attached
// UNSAFE: @Retry(maxAttempts=3) — RetryAspect calls proceed() 3 times on WebServiceException.
// Each proceed() re-enters chargeCustomer() from line 1.
// Each re-entry calls stripePort.charge() — new JAX-WS message exchange — handler chain fires.
// StripeIdempotencyHandler.handleMessage() evaluates UUID.randomUUID() per chain execution.
// stripePort is a singleton proxy but handler chain execution is per-call, not per-bean.
@Retry(name = "stripeBilling")
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
return stripePort.charge(customerId, amountCents, billingPeriod);
}
}
// The key observation: stripePort is a singleton proxy bean.
// Being a singleton does NOT mean handler chain runs only once.
// CXF's dynamic proxy invokes the handler chain on every method call to the proxy.
// A singleton proxy called 3 times (3 @Retry attempts) fires the handler chain 3 times.
// UUID.randomUUID() in StripeIdempotencyHandler fires 3 times → 3 distinct idempotency keys
// → potential for ch_A, ch_B, ch_C depending on which Stripe calls succeed before failure.
The singleton scope of the injected proxy is the source of the subtle misdirection: developers often associate singleton beans with “configured once, run once.” For stateful setup like establishing a connection pool or parsing a WSDL, that association is correct — these operations happen once when the bean is created. For handler chain execution, the singleton scope is irrelevant. The handler chain is a call-processing pipeline, not a configuration artifact. It runs per call regardless of the bean’s scope.
The fix for failure mode 2
The stable key must be computed before the first JAX-WS proxy method call and made available to StripeIdempotencyHandler.handleMessage() without re-evaluation on retry. JAX-WS provides two mechanisms for passing per-invocation context to a handler: BindingProvider.getRequestContext() (a Map of per-operation context values) and MessageContext properties set before the invocation. For a @Retry-annotated service method, the correct pattern is to refactor the method so the stable key is computed in a non-retried outer method and passed as a parameter to the retried inner method:
// Safe StripeIdempotencyHandler.java — reads key from MessageContext / RequestContext.
// Never calls UUID.randomUUID() inside handleMessage().
public class StripeIdempotencyHandler implements SOAPHandler<SOAPMessageContext> {
public static final String IDEMPOTENCY_KEY_PROPERTY = "stripe.idempotency.key";
private static final QName IDEMPOTENCY_QNAME =
new QName("https://api.stripe.com/headers", "IdempotencyKey");
@Override
public boolean handleMessage(SOAPMessageContext context) {
Boolean outbound = (Boolean) context.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY);
if (!Boolean.TRUE.equals(outbound)) return true;
// Safe: read from MessageContext properties set by calling code.
// Calling code sets the key via BindingProvider.getRequestContext()
// before each logical billing operation (not inside the @Retry scope).
String key = (String) context.get(IDEMPOTENCY_KEY_PROPERTY);
if (key == null || key.isBlank()) {
throw new javax.xml.ws.WebServiceException(
"stripe.idempotency.key not set in MessageContext — " +
"compute stableKey() before calling the proxy and set via " +
"((BindingProvider) proxy).getRequestContext().put(IDEMPOTENCY_KEY_PROPERTY, key)");
}
try {
SOAPHeader header = context.getMessage().getSOAPPart().getEnvelope().getHeader();
if (header == null) {
header = context.getMessage().getSOAPPart().getEnvelope().addHeader();
}
header.addChildElement(IDEMPOTENCY_QNAME).setTextContent(key);
} catch (javax.xml.soap.SOAPException e) {
throw new javax.xml.ws.WebServiceException("Failed to set idempotency key header", e);
}
return true;
}
@Override
public boolean handleFault(SOAPMessageContext context) { return true; }
@Override
public void close(MessageContext context) { }
@Override
public Set<QName> getHeaders() { return Collections.emptySet(); }
}
// Safe BillingBean.java — stableKey computed in outer non-retried method.
// Inner @Retry-annotated method receives idempotencyKey as a parameter.
// BindingProvider.getRequestContext() is set in outer method before @Retry scope.
@Service
public class BillingBean {
@Autowired
private StripeBillingPort stripePort;
// Outer method: not annotated with @Retry.
// Computes the stable key ONCE and sets it in RequestContext before @Retry scope.
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
String idempotencyKey = stableKey(customerId, billingPeriod);
// Set the key in RequestContext before entering @Retry scope.
// CXF propagates RequestContext values into MessageContext for each proxy call,
// so StripeIdempotencyHandler reads the same key across all retry attempts.
((BindingProvider) stripePort).getRequestContext()
.put(StripeIdempotencyHandler.IDEMPOTENCY_KEY_PROPERTY, idempotencyKey);
// Pre-flight: claim the billing slot atomically before any proxy call.
if (!claimBillingSlot(customerId, billingPeriod, idempotencyKey)) {
return lookupExistingCharge(customerId, billingPeriod);
}
// Delegate to inner method annotated with @Retry.
// Inner method receives idempotencyKey as a parameter — but since RequestContext
// is already set, the handler reads it from there rather than needing the parameter.
return chargeWithRetry(customerId, billingPeriod, amountCents);
}
// Inner method: @Retry re-invokes this method body on WebServiceException.
// BindingProvider.getRequestContext() was set by chargeCustomer() before this scope.
// The same idempotencyKey flows through RequestContext → MessageContext → handler
// on every @Retry re-invocation. StripeIdempotencyHandler reads it; no UUID regeneration.
@Retry(name = "stripeBilling")
protected ChargeResponse chargeWithRetry(
String customerId, String billingPeriod, long amountCents) {
return stripePort.charge(customerId, amountCents, billingPeriod);
}
private String stableKey(String customerId, String billingPeriod) {
try {
String raw = customerId + ":" + billingPeriod + ":cxf-billing";
byte[] hash = MessageDigest.getInstance("SHA-256")
.digest(raw.getBytes(StandardCharsets.UTF_8));
StringBuilder sb = new StringBuilder();
for (int i = 0; i < 16; i++) sb.append(String.format("%02x", hash[i]));
return sb.toString();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
The outer chargeCustomer() method computes the stable key and sets it on BindingProvider.getRequestContext() before entering the @Retry-annotated scope. The inner chargeWithRetry() method is what RetryAspect re-invokes on each retry attempt. CXF propagates the RequestContext map into the MessageContext for each proxy call, making the pre-computed key available inside StripeIdempotencyHandler.handleMessage() via context.get(IDEMPOTENCY_KEY_PROPERTY). Each @Retry re-invocation of chargeWithRetry() calls stripePort.charge(), which fires the handler chain, which reads the same key from RequestContext — no UUID regeneration. The pre-flight claimBillingSlot() ensures that even if two concurrent service instances both compute the same stable key (they will, because it is deterministic), only one proceeds to the proxy call; the other reads the existing charge result.
Failure mode 3: per-JVM ScheduledExecutorService billing on Kubernetes replicas:3 — CXF client Bus is per-JVM — TOCTOU race on hasCompletedForPeriod() — all 3 pods generate distinct UUID.randomUUID() per customer — ch_A, ch_B, ch_C per customer per billing period
Apache CXF’s Bus — the central service registry for interceptors, conduits, and extension points — is initialized per JVM at startup via CXFBusFactory.getDefaultBus() or Spring’s CXF namespace bean (<cxf:bus>). JAX-WS proxies and WebClient instances are backed by the JVM-local Bus. A ScheduledExecutorService that drives the billing loop is also per-JVM. Kubernetes deploys the billing service with replicas:3 for availability. All three pods have independent JVMs, independent Bus instances, and independent ScheduledExecutorService instances. When the scheduler fires at billing time, all three pods execute the billing loop concurrently with no cross-pod coordination:
// BillingScheduler.java — UNSAFE: runs independently on each Kubernetes replica.
// With replicas:3, three pods fire the billing loop simultaneously at cron time.
// All three query hasCompletedForPeriod() before any pod writes the billing-started record.
// TOCTOU race: all three see "not completed" and proceed to charge customers.
// Each pod's CXF proxy fires the StripeIdempotencyInterceptor independently.
// Even with the safe interceptor, each pod generates its own stableKey per customer —
// stableKey is deterministic, so the same idempotency key flows from all three pods.
// Stripe's idempotency cache deduplics on the same key — but only within a 24-hour window.
// If the key is truly stable, Stripe returns ch_A to all three pods.
// If any pod uses an UNSAFE interceptor with UUID.randomUUID(), each pod sends a different key.
// ch_A from pod 1, ch_B from pod 2, ch_C from pod 3 — per customer — 1,500 charges for 500 customers.
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.Executors;
@Component
public class BillingScheduler {
private final BillingService billingService;
private final CustomerRepository customers;
private final BillingPeriodTracker tracker;
// Spring @Scheduled fires on all pods simultaneously at cron expression time.
// No built-in cross-pod coordination.
@Scheduled(cron = "0 0 0 1 * *") // first of each month at midnight UTC
public void runMonthlyBilling() {
String billingPeriod = currentBillingPeriod();
// TOCTOU: all 3 pods reach hasCompletedForPeriod() at approximately the same millisecond.
// None have committed a billing-started record yet.
// All 3 read false. All 3 proceed to the loop below.
if (tracker.hasCompletedForPeriod(billingPeriod)) return;
for (Customer customer : customers.findAllActive()) {
// Pod 1: billingService calls CXF proxy → interceptor → UUID "4c8a3b1d-..." → ch_A
// Pod 2: billingService calls CXF proxy → interceptor → UUID "d7e2f4a6-..." → ch_B
// Pod 3: billingService calls CXF proxy → interceptor → UUID "f1e2d3c4-..." → ch_C
// 500 customers × 3 pods × distinct UUID per pod = 1,500 Stripe charge attempts
// 500 customers charged 3 times each = $X × 3 per customer for the month.
billingService.chargeCustomer(customer.getId(), billingPeriod, customer.getAmountCents());
}
tracker.markCompleted(billingPeriod); // only one pod's write wins; the others silently commit
}
}
The TOCTOU race: all three pods query hasCompletedForPeriod(billingPeriod) within the same scheduler tick. The tick fires at the millisecond level and all three pods are synchronized to the system clock. hasCompletedForPeriod() executes a SELECT against the shared billing database. None of the three pods has written the billing-started record yet — that write happens at the end of the loop. All three reads return false. All three pods enter the billing loop and begin charging customers. The damage scales with customer count and replica count: 500 customers with replicas:3 and an unsafe interceptor (fresh UUID per pod per customer) yields up to 1,500 Stripe charge attempts.
The safe content-hash idempotency key partially mitigates the damage: all three pods compute the same stable key for each customer, so Stripe’s idempotency cache returns ch_A to pods 2 and 3 for each customer (as long as the initial request is within Stripe’s 24-hour idempotency window). But the mitigation relies entirely on Stripe’s server-side deduplication. It provides no protection against a pod that uses an unsafe interceptor (which was the code before the fix), against a billing period that spans more than 24 hours between pod executions (rare but possible during downtime), or against the load: even with deduplication, 1,500 HTTP requests to Stripe’s API in one burst will hit Stripe’s rate limits and produce 429 errors that look like billing failures to the on-call engineer.
The subtler variant: ShedLock @SchedulerLock with lockAtMostFor shorter than billing P99 — lock expires while the first pod is still executing — second pod acquires the expired lock — reads hasCompletedForPeriod() = false for not-yet-processed customers — creates ch_B for those customers
ShedLock is a common addition to Spring services that need one-pod-at-a-time cron semantics. It writes a lock record to the shared database at cron time and deletes or expires it after the job completes. The lockAtMostFor parameter is a safety ceiling: if the lock holder crashes without releasing the lock, ShedLock forcibly expires the lock after lockAtMostFor duration so the job can be picked up by another pod on the next cron tick.
The failure: lockAtMostFor is set to a value shorter than the billing job’s actual P99 tail latency. A common misconfiguration is setting it to the P50 or a “comfortable” estimate that does not account for Stripe latency spikes during high-traffic billing periods (end-of-month, major US holiday weekends). When the billing job takes longer than lockAtMostFor on a high-latency day, ShedLock expires the lock. A second pod’s next scheduled check fires (at the next cron interval or via ShedLock’s poll interval), acquires the now-available lock, and reads hasCompletedForPeriod(billingPeriod) = false — because the first pod has not yet finished and thus has not called markCompleted(). The second pod begins billing from the top of the customer list. For customers the first pod has not yet processed (those later in the list), both pods now bill concurrently. The first pod sends ch_A with its UUID (or stable key); the second pod sends ch_B with its own UUID (or the same stable key). With a stable content-hash key, Stripe deduplicates. With an unsafe UUID interceptor, ch_B is a new charge:
// @SchedulerLock configuration — UNSAFE: lockAtMostFor shorter than billing P99.
// If billing takes 35 minutes (P99 on a high-latency month-end day) and lockAtMostFor=20min,
// ShedLock expires the lock at T+20min while pod 1 is still executing.
// Pod 2 acquires the lock at T+20min, reads hasCompletedForPeriod()=false (pod 1 not done),
// starts billing from customer 1, creates ch_B for all customers pod 1 hasn't processed yet,
// and creates ch_A-prime for customers pod 1 already processed with a safe stable key
// (Stripe deduplicates same-key re-requests within 24h — ch_A-prime → same as ch_A).
@Scheduled(cron = "0 0 0 1 * *")
@SchedulerLock(
name = "monthlyBilling",
lockAtLeastFor = "PT10M", // hold for at least 10 minutes even if job completes early
lockAtMostFor = "PT20M" // UNSAFE: P99 is 35 minutes on high-latency days → lock expires
)
public void runMonthlyBilling() {
String billingPeriod = currentBillingPeriod();
if (tracker.hasCompletedForPeriod(billingPeriod)) return;
for (Customer customer : customers.findAllActive()) {
billingService.chargeCustomer(
customer.getId(), billingPeriod, customer.getAmountCents());
}
tracker.markCompleted(billingPeriod);
}
The fix for failure mode 3
The pre-flight ON CONFLICT DO NOTHING constraint inside chargeCustomer() is the authoritative cluster-wide billing mutex. This is the only defense that operates at the unit of actual Stripe work (one per customer per billing period) rather than at the job level (one lock for the entire batch). Even if multiple pods are simultaneously executing the billing loop, each pod’s attempt to INSERT INTO billing_slots (...) ON CONFLICT DO NOTHING for a given (customer_id, billing_period) pair will succeed for only one pod. All other pods receive zero rows affected, read the existing charge result, and skip the Stripe call. PostgreSQL’s unique constraint enforcement is atomic and serializable — it is immune to TOCTOU races that defeat the application-level hasCompletedForPeriod() check:
// Fix: pre-flight ON CONFLICT DO NOTHING as the authoritative billing mutex.
// Even with replicas:3 all billing simultaneously, only one pod commits per (customer_id, period).
// Schema:
// CREATE TABLE billing_slots (
// customer_id TEXT NOT NULL,
// billing_period TEXT NOT NULL,
// idempotency_key TEXT NOT NULL,
// claimed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
// stripe_charge_id TEXT,
// CONSTRAINT billing_slots_pkey PRIMARY KEY (customer_id, billing_period)
// );
// In chargeCustomer() — runs per customer per billing cycle on every pod:
boolean claimed = db.execute(
"INSERT INTO billing_slots (customer_id, billing_period, idempotency_key) " +
"VALUES (?, ?, ?) ON CONFLICT (customer_id, billing_period) DO NOTHING",
customerId, billingPeriod, stableKey(customerId, billingPeriod));
// claimed = true → this pod is the winner for this customer → proceed to Stripe call
// claimed = false → another pod already claimed → return existing charge, skip Stripe
if (!claimed) {
return lookupExistingCharge(customerId, billingPeriod);
}
// Only one pod per customer reaches this line.
ChargeResponse charge = stripeProxy.charge(customerId, amountCents, billingPeriod);
db.execute("UPDATE billing_slots SET stripe_charge_id = ? " +
"WHERE customer_id = ? AND billing_period = ?",
charge.getId(), customerId, billingPeriod);
return charge;
// Additionally: replace @SchedulerLock with pg_try_advisory_lock() for the batch-level guard.
// pg_try_advisory_lock() releases automatically on session disconnect —
// it does not expire mid-run based on a misconfigured duration.
@Scheduled(cron = "0 0 0 1 * *")
public void runMonthlyBilling() throws Exception {
String billingPeriod = currentBillingPeriod();
long lockKey = Math.abs(("monthly-billing:" + billingPeriod).hashCode());
// Attempt to acquire a session-scoped advisory lock.
// Only one pod acquires it — all others return false immediately (non-blocking).
// Lock is released when the database connection closes (no lockAtMostFor misconfiguration).
Boolean acquired = db.queryForObject(
"SELECT pg_try_advisory_lock(?)", Boolean.class, lockKey);
if (!Boolean.TRUE.equals(acquired)) {
log.info("Monthly billing for {} already running on another pod — skipping", billingPeriod);
return;
}
try {
for (Customer customer : customers.findAllActive()) {
billingService.chargeCustomer(
customer.getId(), billingPeriod, customer.getAmountCents());
}
} finally {
db.execute("SELECT pg_advisory_unlock(?)", lockKey);
}
}
pg_try_advisory_lock() provides a session-scoped cross-pod mutex without a configurable expiration. The lock is held for exactly as long as the database session is open — if the pod crashes, the session closes and the lock is released automatically. This is structurally safer than ShedLock’s row-based lock with a lockAtMostFor duration, because it does not depend on a duration estimate matching the actual job runtime. The per-customer ON CONFLICT DO NOTHING pre-flight provides defense-in-depth: even if two pods simultaneously hold no advisory lock (e.g., pod 1 wins the advisory lock but crashes after charging 200 customers, pod 2 then acquires it and restarts from customer 1), the pre-flight constraint ensures customers already charged by pod 1 are not re-billed by pod 2. The combination of advisory lock (batch-level serialization) and per-customer constraint (charge-level idempotency) covers all reasonable concurrent execution patterns.
Vault keys as the financial backstop
Content-hash idempotency keys, pg_try_advisory_lock(), and pre-flight ON CONFLICT DO NOTHING prevent duplicate charges at the application logic level. A vault key scoped per billing period with a spend cap adds the financial layer that operates independently of application code:
// Per-billing-period vault key via Keybrake:
//
// Before the billing batch starts:
// 1. Issue a vault key scoped to Stripe with a daily USD cap:
// vault_key = keybrake.issueKey({
// vendor: "stripe",
// daily_usd_cap: expected_total_usd * 1.10, // 10% margin above expected
// allowed_endpoints: ["/v1/charges", "/v1/customers"],
// expires_at: billingPeriod.endOfMonth()
// })
//
// 2. Register the vault key as the Authorization bearer token in the CXF interceptor chain:
//
// public class StripeAuthInterceptor extends AbstractPhaseInterceptor<Message> {
// private final String vaultKey;
// public StripeAuthInterceptor(String vaultKey) {
// super(Phase.PRE_STREAM);
// this.vaultKey = vaultKey;
// }
// @Override
// public void handleMessage(Message message) throws Fault {
// Map<String, List<String>> headers = CastUtils.cast(
// message.get(Message.PROTOCOL_HEADERS));
// if (headers == null) {
// headers = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);
// message.put(Message.PROTOCOL_HEADERS, headers);
// }
// headers.put("Authorization", Collections.singletonList("Bearer " + vaultKey));
// }
// }
//
// 3. When the billing batch exceeds the cap (retry storm, logic bug, runaway loop),
// Keybrake returns HTTP 429 — CXF throws WebServiceException wrapping HTTP 429:
// {"error":"daily_usd_cap_exceeded","cap_usd":450.00,"spent_usd":450.01}
// The billing scheduler catches 429 as a non-retryable terminal error and stops cleanly.
// No surprise invoice: the cap stops the bleeding before it reaches the next card statement.
//
// 4. Keybrake's audit log records every proxied Stripe request with the vault key,
// billing period, customer ID (from the request body), and charge amount (from
// the Stripe response). Independent of the billing service's own database —
// cross-reference for reconciliation and dispute resolution.
The vault key’s spend cap catches the cases where software-layer guards succeed at preventing duplicate charges but application logic has a bug — charging 20% more customers than intended, double-counting amounts due to a currency precision error, or running the billing batch against the wrong billing period. The cap converts a silent financial discrepancy into a hard stop with a clear error message. The billing team learns about it from the Keybrake alert rather than from customer disputes filed two weeks later.
Putting it together
Apache CXF’s AbstractPhaseInterceptor, JAX-WS SOAPHandler, and per-JVM scheduler each represent a structurally distinct way that idempotency key generation ends up inside a scope that executes more than once per logical billing operation. The AbstractPhaseInterceptor.handleMessage() executes per outbound CXF Message; a retry loop that re-calls the proxy generates a new Message and re-executes the chain. The JAX-WS SOAPHandler.handleMessage() executes per JAX-WS proxy method invocation; Resilience4j @Retry re-invokes the containing service method, which re-invokes the proxy, which re-executes the handler chain. The per-JVM scheduler executes the billing loop on all Kubernetes replicas simultaneously; TOCTOU on the completion check means all pods enter the billing loop before any pod commits a guard record.
The fix in every case is the same pattern: compute a deterministic, content-hash key outside all retry and re-invocation boundaries — once per billing operation, before any CXF proxy call or WebClient.post() invocation — and carry it through all retry attempts in a form that no retry re-evaluation can change. For synchronous JAX-WS proxy calls with an AbstractPhaseInterceptor, set the key in BindingProvider.getRequestContext() before the retry loop; the interceptor reads it from the Exchange on each attempt. For JAX-WS SOAPHandler with Resilience4j @Retry, refactor so the key is computed in the non-retried outer method and the RequestContext is populated before entering the @Retry-annotated scope. For async WebClient with InvocationCallback, capture the key as a final local variable before the first post() call and pass it explicitly through the callback closure. For the scheduler, replace the application-level completion check with pg_try_advisory_lock() at the batch level and ON CONFLICT DO NOTHING at the per-customer level.
The layering of defenses — stable content-hash key, pre-flight ON CONFLICT DO NOTHING, advisory lock, and vault key spend cap — matches the layering of retry and concurrency mechanisms that production billing systems actually use. No single defense covers every failure class. The key insight specific to CXF is that its interceptor model is designed for transparency: the phase chain fires invisibly around every proxy method call, making it easy to add cross-cutting concerns like authentication, logging, and tracing without modifying service code. That transparency is a feature. It becomes a billing hazard when the cross-cutting concern — idempotency key generation — must behave differently from authentication or logging: it must fire exactly once per logical operation, not once per HTTP attempt.
The CXF interceptor that looks like infrastructure plumbing — no business logic, no side effects — becomes a billing hazard the day someone adds a retry loop in the service class. The stable key computed before the loop is what makes the interceptor safe regardless of how many times the retry policy decides to re-invoke the proxy.
For the JAX-RS ClientRequestFilter.filter() retry pattern, see Jersey and JAX-RS Stripe Integration. For the Feign RequestInterceptor.apply() pattern, see Feign and Spring Cloud OpenFeign Stripe Integration. For the Apache HttpClient 5 HttpRequestInterceptor pattern, see Apache HttpClient 5 and Stripe Integration. For the OkHttp application interceptor pattern, see OkHttp and Retrofit Stripe Integration. For the Spring Boot @Retryable and RestTemplate interceptor patterns, see Spring Boot and Stripe Integration. For the full series, see the blog index.
Put a spend cap on your agent’s Stripe key
Keybrake issues scoped vault keys for Stripe, Twilio, and Resend with per-period spend caps, allowed-endpoint allowlists, and an audit log of every proxied request. When the cap is hit, the agent gets a 429 — not a surprise invoice. Join the waitlist.