Feign and Spring Cloud OpenFeign Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance
Feign’s SynchronousMethodHandler executes an executeAndDecode() loop that re-invokes all registered RequestInterceptors on every retry attempt triggered by Retryer.Default — UUID.randomUUID() inside RequestInterceptor.apply() produces a different Idempotency-Key on the initial request and on Feign’s transparent retry: the initial POST /v1/charges creates ch_A before a RetryableException from a transient connection failure, and the retry’s apply() evaluates a fresh UUID, causing Stripe to create ch_B. Three Feign and Spring Cloud OpenFeign-specific Stripe billing failure modes: a RequestInterceptor computes UUID.randomUUID() per apply() invocation — subtler variant: Spring Cloud OpenFeign with spring-retry on the classpath enables Feign-level retry automatically, and a Retryer bean in the @FeignClient configuration class silently overrides the default no-retry policy, causing every FeignException to trigger interceptor re-invocation with a fresh UUID; a service method annotated with Resilience4j @Retry that computes UUID.randomUUID() at the @FeignClient interface call site — Resilience4j re-invokes the method body on each retry, re-evaluating the UUID argument expression, initial call creates ch_A before FeignException.ServiceUnavailable, first @Retry re-invocation creates ch_B — subtler variant: Feign’s own Retryer.Default is also configured on the client, so Resilience4j outer retry (3 attempts) multiplied by Feign inner retry (up to 3 executeAndDecode() loops per Resilience4j attempt) produces up to 9 distinct idempotency keys from a single service instance; and a per-JVM @Scheduled billing job fires the Feign-backed billing service independently on all Kubernetes replicas — with replicas:3, all three pods pass the concurrent hasCompletedForPeriod() database check before any pod commits the billing-started record (TOCTOU race), all three compute UUID.randomUUID() per customer independently, and all three create ch_A, ch_B, ch_C per customer per billing period.
This post covers all three failure modes with Java code, content-hash idempotency keys stable across RequestInterceptor re-invocations, Resilience4j and Feign retry compounding, and multi-pod concurrent billing loops, a RequestTemplate attribute as the key-passing mechanism between calling code and the interceptor, pg_try_advisory_lock() for cross-pod scheduler serialization with correct lockAtMostFor sizing, and pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For the Spring AOP @Retryable pattern and RestTemplate interceptors, see the Spring Boot and Stripe Integration post. For the Apache HttpClient 5 interceptor pattern, see the Apache HttpClient 5 and Stripe Integration post. For the OkHttp application interceptor pattern, see the OkHttp and Retrofit Stripe Integration post.
Failure mode 1: RequestInterceptor.apply() computes UUID.randomUUID() — Feign’s Retryer.Default triggers executeAndDecode() loop to re-run the full interceptor pipeline on every retry — initial request creates ch_A before RetryableException — retry’s apply() creates ch_B
Feign’s internal execution model is built around SynchronousMethodHandler, which holds the loop that drives retries. When you call a method on a Feign proxy, SynchronousMethodHandler.invoke() enters a while (true) loop that calls executeAndDecode(template, options) on each iteration. executeAndDecode() builds the HTTP request in three stages: it clones the RequestTemplate, applies all registered RequestInterceptors via template.applyTo(interceptors) (which calls interceptor.apply(template) for each), then calls client.execute(request, options). If the call throws an IOException, Feign wraps it in a RetryableException and calls retryer.continueOrPropagate(e). If the retryer decides to continue — which Retryer.Default does for up to maxAttempts — the loop iterates and calls executeAndDecode() again. This means every registered RequestInterceptor fires on the initial request and on each retry.
A StripeIdempotencyInterceptor that calls UUID.randomUUID() inside apply(RequestTemplate template) to set the Idempotency-Key header fires per executeAndDecode() call — once for the initial attempt and once for every retry the retryer permits:
// StripeIdempotencyInterceptor.java
// UNSAFE: UUID.randomUUID() computed inside apply() — called on the initial
// request AND on every retry triggered by Retryer.Default.
// If the initial request created ch_A before an IOException from a dropped
// connection, the retry's fresh UUID causes Stripe to create ch_B.
import feign.RequestInterceptor;
import feign.RequestTemplate;
import java.util.UUID;
public class StripeIdempotencyInterceptor implements RequestInterceptor {
@Override
public void apply(RequestTemplate template) {
// UNSAFE: UUID.randomUUID() called per apply() invocation.
// Initial request: UUID = "4c8a3b1d-2e5f-4b7e-9c0a-1a2b3c4d5e6f"
// Retryer retry 1: UUID = "d7e2f4a6-3b8c-4f1e-0d9a-2b3c4d5e6f7a"
// Retryer retry 2: UUID = "f1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c"
template.header("Idempotency-Key", UUID.randomUUID().toString());
}
}
// Feign client configuration — Retryer.Default retries on IOException:
StripeClient stripeClient = Feign.builder()
.requestInterceptor(new StripeIdempotencyInterceptor()) // UNSAFE interceptor
.retryer(new Retryer.Default(100, TimeUnit.SECONDS.toMillis(1), 3))
.client(new OkHttpClient())
.decoder(new GsonDecoder())
.target(StripeClient.class, "https://api.stripe.com");
The failure scenario: the billing service calls stripeClient.createCharge(chargeRequest). Feign’s SynchronousMethodHandler.invoke() enters the retry loop. executeAndDecode() is called for the first iteration. The interceptor pipeline fires. StripeIdempotencyInterceptor.apply() calls UUID.randomUUID(), returns "4c8a3b1d-2e5f-4b7e-9c0a-1a2b3c4d5e6f", and sets Idempotency-Key: 4c8a3b1d-.... Feign’s HTTP client submits POST /v1/charges to Stripe. The request body reaches Stripe’s servers. Stripe processes the charge. ch_A is committed in Stripe’s ledger. The underlying TCP connection to Stripe is reset before the HTTP response arrives at the billing service — a common occurrence with cloud load balancers that silently drop idle keep-alive connections between requests. Feign’s HTTP client throws an IOException reading the response. SynchronousMethodHandler wraps it in RetryableException and calls retryer.continueOrPropagate(e). Retryer.Default has seen only one attempt; it sleeps 100 ms and returns, allowing the loop to continue. executeAndDecode() is called for the second iteration. The interceptor pipeline fires again. StripeIdempotencyInterceptor.apply() is invoked again — it calls UUID.randomUUID() and returns a completely new value: "d7e2f4a6-3b8c-4f1e-0d9a-2b3c4d5e6f7a". The retry request reaches Stripe. Stripe looks up "d7e2f4a6-..." in its idempotency cache. It finds nothing — this UUID has never been seen. Stripe processes the charge again. ch_B is created. The customer is charged $99 twice for the same billing period.
The failure is structurally invisible because the interceptor design looks correct: a dedicated class, a single responsibility, a clean interface. The invariant that was violated is not part of the Feign documentation or the RequestInterceptor interface contract: apply() is called per HTTP attempt, not per logical API call. A developer who reads only the Feign guide’s “Request Interceptors” section without reading the retry internals will assume the interceptor fires once per proxy method invocation, because that is how Spring’s ClientHttpRequestInterceptor and OkHttp’s application Interceptor are typically described.
The subtler variant: Spring Cloud OpenFeign with spring-retry on the classpath — Feign retry enabled automatically — a Retryer bean in the @FeignClient configuration class silently activates retry — RequestInterceptor fires with a fresh UUID on every Feign retry
Spring Cloud OpenFeign’s auto-configuration has a conditional wiring: if spring-retry is on the classpath, Spring Cloud OpenFeign registers a FeignRetryer that delegates to Spring Retry. Additionally, a @FeignClient annotation can reference a configuration class via configuration = BillingFeignConfig.class, and that class can declare a @Bean of type feign.Retryer. If a developer adds a Retryer.Default bean to the configuration class to handle transient 5xx errors from Stripe and the billing team’s RequestInterceptor was written with the assumption that it fires once per proxy invocation, the retry wiring turns a safe interceptor into an unsafe one without any change to the interceptor code itself:
// BillingFeignConfig.java — @FeignClient configuration class.
// Declaring a Retryer bean here enables Feign-level retry for this client ONLY.
// If StripeIdempotencyInterceptor calls UUID.randomUUID() inside apply(),
// every retry triggered by this retryer fires apply() with a fresh UUID.
@Configuration
public class BillingFeignConfig {
// UNSAFE if StripeIdempotencyInterceptor.apply() calls UUID.randomUUID():
// Retryer.Default retries on RetryableException (IOException wrapper) up to 3 times.
// apply() fires on the initial attempt and on each retry — fresh UUID per attempt.
@Bean
public Retryer feignRetryer() {
return new Retryer.Default(100, TimeUnit.SECONDS.toMillis(1), 3);
}
@Bean
public RequestInterceptor stripeIdempotencyInterceptor() {
return new StripeIdempotencyInterceptor(); // UNSAFE — calls UUID.randomUUID() in apply()
}
}
// @FeignClient declaration references the configuration class:
@FeignClient(
name = "stripe-billing",
url = "${stripe.api.url:https://api.stripe.com}",
configuration = BillingFeignConfig.class
)
public interface StripeBillingClient {
@PostMapping(
value = "/v1/charges",
consumes = MediaType.APPLICATION_FORM_URLENCODED_VALUE
)
ChargeResponse createCharge(Map<String, ?> formData);
}
The coupling is invisible at the call site. The service class that injects StripeBillingClient and calls stripeClient.createCharge(formData) has no visibility into whether Feign retry is configured on the client bean. The service may have been written before BillingFeignConfig added the Retryer bean. A later addition of the retryer — perhaps in response to a different transient failure mode that had nothing to do with billing — activates retry across all methods on StripeBillingClient, including createCharge, without any change to the service layer or the interceptor.
The fix for failure mode 1
The idempotency key must be computed once per billing operation, before the Feign proxy method is called, and passed into the interceptor in a way that survives interceptor re-invocations. RequestTemplate carries per-request metadata and is the correct vehicle: the calling code sets a custom header or a query parameter that the interceptor reads, rather than generating. Alternatively, Feign’s @RequestHeader parameter on the interface method lets calling code pass the key as a method argument, which Feign binds to the header before any interceptor fires:
// Approach A: @RequestHeader parameter — calling code owns key computation.
// Feign binds the argument to the header value before the interceptor pipeline.
// The same value is bound on every executeAndDecode() iteration because
// Feign clones the RequestTemplate from the method template, which captured
// the parameter value at the point of the proxy method invocation.
@FeignClient(name = "stripe-billing", url = "${stripe.api.url}", configuration = BillingFeignConfig.class)
public interface StripeBillingClient {
@PostMapping(value = "/v1/charges", consumes = MediaType.APPLICATION_FORM_URLENCODED_VALUE)
ChargeResponse createCharge(
@RequestHeader("Idempotency-Key") String idempotencyKey, // safe: bound once per invocation
Map<String, ?> formData
);
}
// BillingService.java — calling code owns key computation.
@Service
public class BillingService {
private final StripeBillingClient stripeClient;
private final BillingRepository billingRepository;
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
// Computed once per billing operation — deterministic, stable across retries and pods.
String idempotencyKey = stableKey(customerId, billingPeriod);
// Pre-flight: claim the billing slot before any network activity.
boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
if (!claimed) {
return billingRepository.findExistingCharge(customerId, billingPeriod);
}
// Feign binds idempotencyKey to the Idempotency-Key header value at proxy invocation time.
// Retryer.Default retries re-run executeAndDecode() but re-clone the same RequestTemplate
// built from the method's parameter bindings — the header value is "4c8a3b1d-..."
// on the initial attempt AND on every retry.
Map<String, Object> formData = buildChargeForm(customerId, amountCents);
return stripeClient.createCharge(idempotencyKey, formData);
}
public static String stableKey(String customerId, String billingPeriod) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] hash = digest.digest(
(customerId + ":" + billingPeriod + ":feign-billing")
.getBytes(StandardCharsets.UTF_8)
);
StringBuilder sb = new StringBuilder();
for (int i = 0; i < 16; i++) {
sb.append(String.format("%02x", hash[i]));
}
return sb.toString();
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("SHA-256 not available", e);
}
}
}
// Approach B: interceptor reads a custom header set by calling code.
// If the interface cannot be modified to add @RequestHeader, the calling code
// sets a stable header before the call and the interceptor reads it.
// Safe interceptor — reads caller-supplied key, never calls UUID.randomUUID():
public class StripeIdempotencyInterceptor implements RequestInterceptor {
private static final String STABLE_KEY_HEADER = "X-Billing-Idempotency-Key";
@Override
public void apply(RequestTemplate template) {
// Safe: read the key that calling code set — survives any number of retries.
Collection<String> keys = template.headers().get(STABLE_KEY_HEADER);
if (keys == null || keys.isEmpty()) {
throw new IllegalStateException(
"Stripe POST missing X-Billing-Idempotency-Key header — " +
"compute stableKey() in calling code and set header before proxy call");
}
String idempotencyKey = keys.iterator().next();
template.header("Idempotency-Key", idempotencyKey);
template.header(STABLE_KEY_HEADER, (String) null); // remove before sending to Stripe
}
}
With the @RequestHeader approach, Feign’s template construction captures the method argument value at proxy invocation time. When Retryer.Default triggers a retry and executeAndDecode() runs again, Feign re-clones the RequestTemplate from the method’s base template — the one that was built from the parameter bindings when the proxy method was first called. The Idempotency-Key header is already set to the value passed by the caller. The interceptor pipeline fires, but apply() sees the header already present and the interceptor becomes a no-op for the key field (or can be removed entirely if the interface-level binding is trusted). Every retry to Stripe carries the same Idempotency-Key. Stripe finds ch_A cached under that key and returns the cached result without creating ch_B.
Failure mode 2: Resilience4j @Retry re-invokes service method — UUID.randomUUID() at the @FeignClient interface call site argument position re-evaluates per @Retry attempt — initial call creates ch_A before FeignException.ServiceUnavailable — first @Retry re-invocation creates ch_B
When the idempotency key is not inside the Feign interceptor but at the call site in the service method body, the failure mode shifts from the Feign retry layer to the outer retry layer. A common pattern in Spring Boot applications using Spring Cloud OpenFeign is to annotate service methods with Resilience4j’s @Retry annotation. Resilience4j’s AOP proxy intercepts the annotated method, calls it, catches retryable exceptions, and re-invokes the full method body on each retry attempt. If the service method computes UUID.randomUUID() inside the method body and passes it as a @RequestHeader parameter to the Feign client interface call, the UUID is re-evaluated on each Resilience4j retry re-invocation:
// BillingService.java — UNSAFE: @Retry re-invokes the method body.
// UUID.randomUUID() at the argument expression position re-evaluates per retry.
// Initial call creates ch_A before FeignException.ServiceUnavailable.
// First @Retry re-invocation evaluates a new UUID — Stripe creates ch_B.
import io.github.resilience4j.retry.annotation.Retry;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestHeader;
import java.util.UUID;
@Service
public class BillingService {
private final StripeBillingClient stripeClient;
@Retry(name = "billing", fallbackMethod = "chargeFallback")
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
// UNSAFE: UUID.randomUUID() evaluated at the argument expression position.
// Resilience4j's @Retry AOP proxy re-invokes this method body on each retry attempt.
// Each re-invocation evaluates a new argument expression for UUID.randomUUID().
//
// Attempt 1: UUID = "4c8a3b1d-2e5f-4b7e-9c0a-1a2b3c4d5e6f" → ch_A created before 503
// Attempt 2: UUID = "d7e2f4a6-3b8c-4f1e-0d9a-2b3c4d5e6f7a" → ch_B ← duplicate charge
// Attempt 3: UUID = "a9b3c7e1-4f5a-6b7c-8d9e-0f1a2b3c4d5e" → ch_C ← third charge
return stripeClient.createCharge(
UUID.randomUUID().toString(), // UNSAFE: re-evaluated on every @Retry re-invocation
buildChargeForm(customerId, amountCents)
);
}
private ChargeResponse chargeFallback(String customerId, String billingPeriod,
long amountCents, Exception e) {
log.error("Billing exhausted retries for customer {}", customerId, e);
throw new BillingException("Billing failed after retries", e);
}
}
// @FeignClient interface — accepts idempotency key as a header parameter:
@FeignClient(name = "stripe-billing", url = "${stripe.api.url}")
public interface StripeBillingClient {
@PostMapping(value = "/v1/charges", consumes = MediaType.APPLICATION_FORM_URLENCODED_VALUE)
ChargeResponse createCharge(
@RequestHeader("Idempotency-Key") String idempotencyKey,
Map<String, ?> formData
);
}
// application.yml — Resilience4j retry config for "billing":
# resilience4j.retry.instances.billing.maxAttempts=3
# resilience4j.retry.instances.billing.waitDuration=500ms
# resilience4j.retry.instances.billing.retryExceptions=feign.FeignException.ServiceUnavailable,
# java.net.SocketException
The failure scenario: Resilience4j’s RetryAspect intercepts the annotated chargeCustomer() call. On attempt 1, the method body executes. The argument expression UUID.randomUUID().toString() is evaluated by the Java runtime at call time, before the method body runs — this is not a lazy expression, it is evaluated immediately as part of the method call. The UUID value "4c8a3b1d-..." is computed and passed to stripeClient.createCharge(). Feign submits POST /v1/charges with Idempotency-Key: 4c8a3b1d-.... Stripe processes the charge. ch_A is committed. Stripe’s API gateway returns a 503 Service Unavailable due to a transient overload condition on Stripe’s backend. Feign wraps this in a FeignException.ServiceUnavailable and throws it. Resilience4j’s retry policy matches FeignException.ServiceUnavailable as a retryable exception. Resilience4j waits 500 ms and re-invokes the method body. On attempt 2, the method body executes from its first line. The argument expression UUID.randomUUID().toString() is evaluated again — it is a separate call to UUID.randomUUID(), entirely independent of the evaluation on attempt 1. The new UUID value "d7e2f4a6-..." is computed and passed to stripeClient.createCharge(). Stripe receives a POST with Idempotency-Key: d7e2f4a6-.... This UUID has never been seen by Stripe. Stripe processes the charge again. ch_B is created. The customer is charged twice.
The reason this is not caught in testing is that unit tests typically mock the Feign client, and the mock returns a success on the first call — Resilience4j never retries. Integration tests against a real Stripe test environment rarely simulate Stripe returning a 503 on the first attempt and succeeding on the second, because Stripe’s test mode infrastructure does not trigger these transient failures on demand. The failure only manifests in production under load, exactly when Stripe’s API is under strain.
The subtler variant: Feign’s own Retryer.Default also configured on the @FeignClient — Resilience4j outer retry (3 attempts) × Feign inner retry (3 executeAndDecode() loops) — up to 9 distinct idempotency keys from a single service instance per billing operation
Feign’s default Retryer is Retryer.NEVER_RETRY — Feign does not retry by default. But a @FeignClient configuration class that adds a Retryer.Default bean, combined with a service method annotated with Resilience4j @Retry, creates a layered retry stack that the billing team may not be aware of. Resilience4j’s outer retry re-invokes the service method body, evaluating a new UUID.randomUUID() argument expression. The Feign client, when it receives a connection-level IOException (as opposed to a Stripe-level 5xx), triggers its own internal Retryer.Default retry loop inside SynchronousMethodHandler. Each Feign inner retry fires the RequestInterceptor pipeline — if the interceptor also calls UUID.randomUUID() inside apply(), each of the 3 inner Feign retries creates a distinct UUID. With Resilience4j retrying the service method 3 times, and Feign retrying each service method invocation up to 3 times, the total distinct idempotency keys that can reach Stripe before both retry stacks exhaust is 9:
// Compounding retry stack — 9 distinct idempotency keys possible from a single service instance:
//
// Resilience4j @Retry attempt 1:
// → method body evaluates UUID #1 at argument position
// → Feign executeAndDecode() loop, attempt 1: interceptor apply() → UUID #1 (same — from @RequestHeader binding)
// → IOException → Feign retries internally
// → Feign executeAndDecode() loop, attempt 2: interceptor apply() → UUID #1 (same — RequestTemplate re-clones from binding)
// → 503 → FeignException.ServiceUnavailable thrown after Feign retries exhausted
// Resilience4j @Retry attempt 2:
// → method body re-invoked — evaluates UUID #2 at argument position (new UUID.randomUUID() call)
// → Feign executeAndDecode() loop, attempt 1: UUID #2
// → IOException → Feign retries
// → Feign executeAndDecode() loop, attempt 2: UUID #2 (same as attempt 2's binding)
// → 503 → ...
// Resilience4j @Retry attempt 3: UUID #3 — same pattern
//
// Result: up to 3 distinct UUIDs per Resilience4j invocation of the service method.
// If the FeignClient interceptor ALSO calls UUID.randomUUID() inside apply():
// → Feign inner retries generate UUID #1a, #1b, #1c per Resilience4j attempt 1
// → Resilience4j retry 2: UUID #2a, #2b, #2c
// → Resilience4j retry 3: UUID #3a, #3b, #3c
// → 9 distinct keys, 9 potential Stripe charges for a single billing operation.
//
// Reality: whether all 9 reach Stripe depends on which failure mode triggers
// the retry at each layer. But any overlap creates ch_B at minimum.
The compounding is not theoretical: in production, connection-level IOExceptions and application-level 503s can occur on the same request path. A billing job running during a rolling deploy of the billing service’s dependencies may hit both a connection reset (triggering Feign inner retry) and a Stripe API transient 503 (triggering Resilience4j outer retry) within the same billing run.
The fix for failure mode 2
The idempotency key must be computed outside any retry boundary and remain stable across all re-invocations at every retry layer. The only position that satisfies this requirement is before the outermost retry boundary — which, in a layered stack with Resilience4j wrapping the service method, means computing the key before the service method is called:
// BillingService.java — SAFE: stable key computed once, passed through all retry layers.
@Service
public class BillingService {
private final StripeBillingClient stripeClient;
private final BillingRepository billingRepository;
// Outer method — NOT annotated with @Retry.
// Computes stable key ONCE before entering any retry boundary.
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
// Computed once per billing operation — deterministic across all retries.
String idempotencyKey = stableKey(customerId, billingPeriod);
// Pre-flight: claim the billing slot before any network activity.
// ON CONFLICT DO NOTHING returns false if already claimed.
boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
if (!claimed) {
return billingRepository.findExistingCharge(customerId, billingPeriod);
}
// Delegate to the retryable inner method — key is captured before the retry boundary.
return chargeWithRetry(customerId, billingPeriod, amountCents, idempotencyKey);
}
// Inner method — annotated with @Retry.
// idempotencyKey is a parameter — passed by the outer method, NOT re-evaluated on retry.
@Retry(name = "billing")
private ChargeResponse chargeWithRetry(String customerId, String billingPeriod,
long amountCents, String idempotencyKey) {
// Safe: idempotencyKey is a parameter value captured before the retry boundary.
// Resilience4j re-invokes this method on retry but passes the same idempotencyKey
// that the outer method computed and passed.
return stripeClient.createCharge(idempotencyKey, buildChargeForm(customerId, amountCents));
}
// Alternative if parameter threading is impractical: use a ThreadLocal
// or Spring's RequestAttributes to pass the key through the retry boundary.
// The ThreadLocal approach works for synchronous code on the same thread:
//
// private static final ThreadLocal<String> BILLING_KEY = new ThreadLocal<>();
//
// public ChargeResponse chargeCustomer(...) {
// BILLING_KEY.set(stableKey(customerId, billingPeriod));
// try { return chargeWithRetry(...); }
// finally { BILLING_KEY.remove(); }
// }
//
// @Retry(name = "billing")
// private ChargeResponse chargeWithRetry(...) {
// return stripeClient.createCharge(BILLING_KEY.get(), buildChargeForm(...));
// }
public static String stableKey(String customerId, String billingPeriod) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] hash = digest.digest(
(customerId + ":" + billingPeriod + ":feign-billing")
.getBytes(StandardCharsets.UTF_8)
);
StringBuilder sb = new StringBuilder();
for (int i = 0; i < 16; i++) {
sb.append(String.format("%02x", hash[i]));
}
return sb.toString();
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("SHA-256 not available", e);
}
}
}
The outer chargeCustomer() method computes the stable key once and passes it as a parameter to the inner chargeWithRetry() method. Resilience4j re-invokes chargeWithRetry() on each retry, but it passes the same parameter value that the caller provided on the first invocation — Resilience4j does not re-evaluate the arguments of the outer call, only the method body of the retried method. idempotencyKey is a String passed by value; its value does not change between retry attempts. The Feign client receives the same Idempotency-Key header value on every attempt at every retry layer. Stripe returns the cached ch_A result on all retry attempts after the first success.
Failure mode 3: per-JVM @Scheduled billing job fires the Feign-backed billing service independently on Kubernetes replicas:3 — TOCTOU race on hasCompletedForPeriod() — all 3 pods generate distinct UUID.randomUUID() per customer — ch_A, ch_B, ch_C per customer per billing period
The first two failure modes concern retry logic within a single billing invocation. The third concerns concurrency across pods in a Kubernetes deployment. Spring Boot’s @Scheduled annotation fires the annotated method on every JVM that runs the application. With replicas:3 in Kubernetes, all three pods start their JVM, register the @Scheduled task with Spring’s ThreadPoolTaskScheduler, and the scheduler fires the billing job at the same wall-clock time on all three pods simultaneously — because all three pods synchronize to the same cron expression evaluated against the same system clock. The billing job iterates over the customer list, and for each customer it calls the BillingService.chargeCustomer() method, which calls the Feign client:
// BillingScheduler.java — UNSAFE: fires independently on every Kubernetes replica.
// With replicas:3, all 3 pods call chargeCustomer() for every customer simultaneously.
// TOCTOU race: all 3 pods read hasCompletedForPeriod() = false before any pod commits.
// All 3 call stripeClient.createCharge() with different UUIDs per customer.
// ch_A, ch_B, ch_C created per customer per billing period.
@Component
public class BillingScheduler {
private final BillingService billingService;
private final CustomerRepository customerRepository;
// UNSAFE: no cross-pod coordination. Fires on every pod.
@Scheduled(cron = "0 0 0 1 * *") // 1st of each month at midnight UTC
public void runMonthlyBilling() {
String billingPeriod = YearMonth.now().toString(); // e.g. "2026-09"
List<Customer> customers = customerRepository.findActive();
for (Customer customer : customers) {
// TOCTOU race: pods A, B, C all read false before any pod commits billing-started.
if (!billingService.hasCompletedForPeriod(customer.getId(), billingPeriod)) {
// Pod A: UUID.randomUUID() → "4c8a3b1d-..." → ch_A
// Pod B: UUID.randomUUID() → "d7e2f4a6-..." → ch_B ← duplicate charge
// Pod C: UUID.randomUUID() → "a9b3c7e1-..." → ch_C ← third charge
billingService.chargeCustomer(customer.getId(), billingPeriod, customer.getAmountCents());
}
}
}
}
The TOCTOU window is proportional to the time between the first pod’s hasCompletedForPeriod() read and the moment it commits the billing-started record to the database. For a billing job processing 500 customers sequentially, this window is open for every customer from the start of the job until the first pod has processed and committed that specific customer. All three pods begin processing at time T=0. Pod A reads hasCompletedForPeriod("cust_1", "2026-09") = false at T=0.001s. Pod B reads the same at T=0.002s. Pod C reads the same at T=0.003s. Pod A has not written the billing-started record yet — it will do so after the Feign call returns, at T=2.1s. All three pods proceed to call stripeClient.createCharge() with different UUID-derived keys. All three charges are committed by Stripe before any pod has written its billing-started record to the database. The hasCompletedForPeriod guard never fires for any pod for customer 1. 1,500 charges are created for 500 customers.
The subtler variant: ShedLock @SchedulerLock configured with lockAtMostFor shorter than the billing job’s tail latency — lock expires mid-run — a second pod acquires the lock and reads hasCompletedForPeriod() = false for not-yet-processed customers — ch_B created for those customers
ShedLock is a popular library for distributing Spring scheduled tasks across a cluster. When integrated correctly, only one pod acquires the lock for the scheduled job, and other pods skip the execution. The two critical configuration parameters are lockAtLeastFor (minimum time the lock is held even if the job finishes early — prevents rapid re-acquisition on a short job) and lockAtMostFor (maximum time the lock is held even if the holder crashes — prevents permanent lock on dead pod). The failure mode is a lockAtMostFor that is shorter than the billing job’s actual tail latency:
// BillingScheduler.java — UNSAFE if lockAtMostFor < billing job P99 duration.
// If the monthly billing job takes 8 minutes but lockAtMostFor=5m:
// - Pod A acquires the lock at T=0, starts processing customers.
// - At T=5m, lockAtMostFor expires. Pod A still holds the lock in memory
// and continues running, but ShedLock releases the database lock row.
// - Pod B's scheduler fires its next check (ShedLock checks every 10s by default).
// Pod B finds no lock held — lockAtMostFor has expired — acquires the lock.
// Pod B calls hasCompletedForPeriod() for customers 251-500 (not yet processed by pod A).
// hasCompletedForPeriod() returns false for all of them.
// Pod B starts charging customers 251-500 concurrently with Pod A.
// → ch_A (Pod A) and ch_B (Pod B) for customers 251-500.
@Component
public class BillingScheduler {
private final BillingService billingService;
private final CustomerRepository customerRepository;
// UNSAFE lockAtMostFor: assumes billing always completes in <5m.
// If billing runs slow (Stripe P99 latency spike, 500 customers × 15ms each = 7.5s,
// but Stripe rate limits cause backoff = 8+ minutes total):
@Scheduled(cron = "0 0 0 1 * *")
@SchedulerLock(
name = "monthlyBilling",
lockAtLeastFor = "PT4M",
lockAtMostFor = "PT5M" // UNSAFE if job takes >5 minutes
)
public void runMonthlyBilling() {
String billingPeriod = YearMonth.now().toString();
List<Customer> customers = customerRepository.findActive();
for (Customer customer : customers) {
if (!billingService.hasCompletedForPeriod(customer.getId(), billingPeriod)) {
billingService.chargeCustomer(customer.getId(), billingPeriod, customer.getAmountCents());
}
}
}
}
The lockAtMostFor value should be set to the billing job’s P99 duration under load plus a safety margin — not the average or the P50. A billing job that processes 500 customers with a Stripe rate limit of 100 req/s and 15ms average latency takes about 75 seconds in the normal case. Under Stripe’s rate-limit backoff with exponential delay, the same job can take 8 minutes or more. A lockAtMostFor set to 5 minutes was calibrated against the average case and fails under the slow path. The correct value is the maximum expected duration plus enough margin to detect a genuinely dead pod (typically 2× the expected maximum): lockAtMostFor = "PT20M". The lockAtLeastFor value should equal the cron period to prevent a fast job from re-running immediately.
The fix for failure mode 3
The correct architecture uses three independent guards, each effective against a different failure class: pg_try_advisory_lock() as the cross-pod distributed mutex that ensures at most one pod runs the billing job at a time, a per-customer pre-flight INSERT ... ON CONFLICT DO NOTHING as the authoritative cluster-wide billing record that prevents double-charging even if two pods acquire billing slots concurrently, and per-billing-period vault keys at Stripe as the financial backstop that caps the total spend regardless of how many charges are attempted:
// BillingScheduler.java — SAFE: pg_try_advisory_lock() as cross-pod distributed mutex.
@Component
public class BillingScheduler {
private final BillingService billingService;
private final CustomerRepository customerRepository;
private final DataSource dataSource;
@Scheduled(cron = "0 0 0 1 * *")
public void runMonthlyBilling() {
String billingPeriod = YearMonth.now().toString();
// Advisory lock key: deterministic integer derived from job name + billing period.
int lockKey = Math.abs(("feign-monthly-billing:" + billingPeriod).hashCode());
// pg_try_advisory_lock() is non-blocking: returns true if acquired, false if another
// pod holds the lock. The lock is automatically released when the session ends
// (pod crashes, connection closed) — no stuck lock on dead pods.
try (Connection conn = dataSource.getConnection()) {
Boolean locked = jdbcTemplate.queryForObject(
"SELECT pg_try_advisory_lock(?)", Boolean.class, (long) lockKey);
if (!Boolean.TRUE.equals(locked)) {
log.info("Monthly billing for {} already running on another pod — skipping",
billingPeriod);
return;
}
try {
runBillingBatch(billingPeriod);
} finally {
jdbcTemplate.update("SELECT pg_advisory_unlock(?)", (long) lockKey);
}
}
}
private void runBillingBatch(String billingPeriod) {
List<Customer> customers = customerRepository.findActive();
for (Customer customer : customers) {
billingService.chargeCustomer(customer.getId(), billingPeriod, customer.getAmountCents());
}
}
}
// BillingService.java — per-customer pre-flight ON CONFLICT DO NOTHING as cluster-wide mutex.
@Service
public class BillingService {
private final StripeBillingClient stripeClient;
private final JdbcTemplate jdbcTemplate;
public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
String idempotencyKey = stableKey(customerId, billingPeriod);
// Pre-flight: atomic cluster-wide billing slot claim.
// If another pod (or a previous run on this pod) already claimed this slot,
// rowsInserted = 0 and we return the existing charge without calling Stripe.
int rowsInserted = jdbcTemplate.update(
"INSERT INTO billing_records (customer_id, billing_period, idempotency_key, created_at) " +
"VALUES (?, ?, ?, NOW()) ON CONFLICT (customer_id, billing_period) DO NOTHING",
customerId, billingPeriod, idempotencyKey
);
if (rowsInserted == 0) {
// Already charged (or in-progress on another pod) — return existing record.
return jdbcTemplate.queryForObject(
"SELECT charge_id, amount_cents FROM billing_records WHERE customer_id = ? AND billing_period = ?",
(rs, n) -> new ChargeResponse(rs.getString("charge_id"), rs.getLong("amount_cents")),
customerId, billingPeriod
);
}
// Sole holder of the billing slot — safe to call Stripe.
// stableKey() ensures the same Idempotency-Key on every @Retry re-invocation.
return chargeWithRetry(customerId, billingPeriod, amountCents, idempotencyKey);
}
@Retry(name = "billing")
private ChargeResponse chargeWithRetry(String customerId, String billingPeriod,
long amountCents, String idempotencyKey) {
return stripeClient.createCharge(idempotencyKey, buildChargeForm(customerId, amountCents));
}
}
// Database schema — unique constraint on (customer_id, billing_period) prevents double-insert:
// CREATE TABLE billing_records (
// id BIGSERIAL PRIMARY KEY,
// customer_id TEXT NOT NULL,
// billing_period TEXT NOT NULL, -- e.g. "2026-09"
// idempotency_key TEXT NOT NULL,
// charge_id TEXT, -- filled after Stripe confirms
// amount_cents BIGINT,
// created_at TIMESTAMPTZ NOT NULL,
// CONSTRAINT billing_records_customer_period_unique UNIQUE (customer_id, billing_period)
// );
pg_try_advisory_lock() provides the first defense: at most one pod runs the billing job for a given billing period. The lock is session-scoped, so a pod crash automatically releases it — there is no lockAtMostFor misconfiguration risk. The pre-flight INSERT ON CONFLICT DO NOTHING provides the second defense: even if two pods somehow both acquire billing slots (e.g., a pod acquired the advisory lock before the previous pod’s session fully closed), the UNIQUE constraint on (customer_id, billing_period) ensures only one row is inserted per customer per period, and the pod that loses the insert conflict returns the existing record without calling Stripe. stableKey() provides the third defense at the Stripe layer: the content-hash key is the same across @Retry re-invocations, so even if Stripe’s API is called more than once due to a retry, Stripe’s own idempotency cache returns ch_A without creating ch_B.
Vault keys as the financial backstop
Content-hash idempotency keys, pg_try_advisory_lock(), and pre-flight ON CONFLICT DO NOTHING together prevent duplicate charges at the application logic level. But they do not prevent a misconfigured or poorly reviewed billing job from charging far more than expected in a period where the idempotency logic is working correctly but the customer list, amount calculation, or period logic has a bug. A vault key — a scoped Stripe API key issued per billing period with a spend cap set to the expected billing total plus a margin — caps the financial exposure at the Stripe credential level regardless of application logic:
// Per-billing-period vault key via Keybrake:
//
// Before the billing job starts:
// 1. Issue a vault key scoped to Stripe with a daily USD cap:
// vault_key = keybrake.issueKey({
// vendor: "stripe",
// daily_usd_cap: expected_total_usd * 1.10, // 10% margin above expected
// allowed_endpoints: ["/v1/charges", "/v1/customers"],
// expires_at: billingPeriod.endOfMonth()
// })
//
// 2. Configure the Feign client to use the vault key as the Stripe API bearer token:
// @FeignClient(name = "stripe-billing", url = "${stripe.api.url}",
// configuration = VaultKeyFeignConfig.class)
// where VaultKeyFeignConfig injects the vault key via RequestInterceptor.apply():
// template.header("Authorization", "Bearer " + vaultKey)
//
// 3. When the billing job exceeds the cap (runaway loop, retry storm, logic bug),
// Keybrake returns 429 Too Many Requests with body:
// {"error":"daily_usd_cap_exceeded","cap_usd":450.00,"spent_usd":450.01}
// The Feign client throws FeignException.TooManyRequests — the billing job
// stops cleanly rather than continuing to charge customers.
//
// 4. Keybrake's audit log records every proxied Stripe request with the vault key,
// billing period, customer ID (from the request body), and charge amount (from
// the Stripe response). The billing team sees a complete per-customer ledger
// from outside the billing application — independent of the application's own
// database — for reconciliation.
The vault key provides a spend cap that operates at the Stripe credential level, independent of the application’s retry logic, distributed locks, and database guards. A billing job with a subtle logic bug that charges 20% more customers than expected hits the cap after the expected total is reached, stops, and surfaces a FeignException.TooManyRequests that is easy to alert on. Without the cap, the same job charges every customer in the misconfigured list and the billing team learns about it from customer complaints. The cap converts a revenue recovery problem into a spend monitoring alert.
Putting it together
Feign’s RequestInterceptor.apply(), Resilience4j’s @Retry method re-invocation, and @Scheduled on multi-replica Kubernetes deployments each represent a different class of billing safety failure, but they share a common root cause: idempotency key generation happens inside a scope that executes more than once per logical billing operation. The fix in every case is the same pattern: compute a deterministic, content-hash key outside all retry boundaries — once per billing operation, before any network activity — and pass it through as a parameter or captured value that no retry re-evaluation can change.
The layering of defenses — stable key, pre-flight ON CONFLICT DO NOTHING, advisory lock, and vault key spend cap — corresponds to the layering of retry and concurrency mechanisms that production billing systems actually use. No single defense covers every failure class. The key insight is that each defense operates at a different layer: the stable key addresses the idempotency layer (prevents Stripe creating ch_B), the pre-flight constraint addresses the database layer (prevents two pods each believing they are the sole holder of the billing slot), the advisory lock addresses the scheduling layer (prevents two pods running the billing job simultaneously), and the vault key addresses the financial layer (caps exposure when all software-layer defenses fail).
The billing job that ran without issue for six months is the one that will have a silent retry compound with a new Resilience4j dependency you added for a different service. The stable key is the thing that keeps that from being a Friday night incident.
For the Spring Boot @Retryable and RestTemplate interceptor patterns, see Spring Boot and Stripe Integration. For the Apache HttpClient 5 HttpRequestInterceptor pattern with HttpClientContext, see Apache HttpClient 5 and Stripe Integration. For the OkHttp application interceptor and Retrofit Callback retry patterns, see OkHttp and Retrofit Stripe Integration. For the Jersey JAX-RS client filter retry pattern, see the blog index for the full series.
Put a spend cap on your agent’s Stripe key
Keybrake issues scoped vault keys for Stripe, Twilio, and Resend with per-period spend caps, allowed-endpoint allowlists, and an audit log of every proxied request. When the cap is hit, the agent gets a 429 — not a surprise invoice. Join the waitlist.