Resilience4j and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

By Keybrake · October 21, 2026

Resilience4j’s Retry.decorateCallable(retry, supplier) stores a reference to the supplier lambda and calls supplier.call() on every retry attempt — not once at decoration time. UUID.randomUUID() inside the lambda body is a call expression that evaluates when the lambda executes, meaning it fires fresh on the initial attempt and on every subsequent retry. The initial callable creates ch_A before a StripeException wrapping a socket timeout; the first retry invocation evaluates a new UUID.randomUUID(), causing Stripe to create ch_B. Three Resilience4j-specific Stripe billing failure modes: Retry.decorateCallable() re-invokes the supplier lambda with a fresh UUID.randomUUID() on every attempt — subtler variant: Decorators.ofCallable() chain wrapping multiple Resilience4j components — each decorator re-invokes the underlying callable on its own retry or transition logic; TimeLimiter combined with Retry — TimeLimiter fires TimeoutException after canceling the future, but the HTTP bytes are already in Stripe’s network stack and ch_A is committed at Stripe before the response arrives back — Retry fires with a fresh UUID.randomUUID() inside the callable creating ch_B — subtler variant: cancelRunningFuture(false) lets the original callable continue while Retry fires a concurrent second attempt, both committed at Stripe simultaneously; and per-JVM ScheduledExecutorService billing on Kubernetes replicas:3 — TOCTOU race — all three pods call Retry.executeCallable() with distinct UUID.randomUUID() per customer — ch_A, ch_B, ch_C per customer per billing period — subtler variant: thread pool ThreadPoolBulkhead sized too small for the billing load, BulkheadFullException triggering Retry with a fresh UUID on the same retry path that also handles StripeException.

This post covers all three failure modes with standalone Resilience4j code (no Spring auto-configuration, no Feign — core resilience4j-core and resilience4j-retry modules used directly), content-hash idempotency keys stable across all decorateCallable() re-invocations, pg_try_advisory_lock() for cross-pod ScheduledExecutorService serialization, and pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — plus per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For Resilience4j @Retry annotations used with Feign clients and Spring beans, see the Feign and Spring Cloud OpenFeign Stripe Integration post. For Resilience4j @Retry on CXF JAX-WS proxy methods, see the Apache CXF and Stripe Integration post.

Failure mode 1: `Retry.decorateCallable()` re-invokes the supplier lambda on every retry attempt — `UUID.randomUUID()` inside the lambda evaluates fresh per invocation — initial attempt creates ch_A before `StripeException` — first retry creates ch_B

Resilience4j’s Retry implementation operates by wrapping a callable in a loop. Internally, Retry.executeCallable(callable) calls callable.call() on each attempt and checks the thrown exception (or return value, if configured with retryOnResult) against the RetryConfig predicates. When a retryable exception is thrown, the retry context waits for waitDuration and calls callable.call() again. The callable reference does not change between attempts — Resilience4j calls the same object. But a lambda in Java is not a memoized thunk: it is an instance of a functional interface whose single method contains the code you wrote in the lambda body. Every call to callable.call() re-executes that lambda body from the top.

This distinction — between the lambda reference (fixed) and the lambda body execution (fresh per call) — is the root of the failure. Any call expression inside the lambda body, including UUID.randomUUID(), evaluates on every invocation:

// UNSAFE: UUID.randomUUID() inside Retry.decorateCallable() lambda body.
// Resilience4j re-invokes the lambda on every retry attempt.
// The initial attempt and every subsequent retry each generate a distinct UUID.

import com.stripe.model.Charge;
import com.stripe.net.RequestOptions;
import com.stripe.param.ChargeCreateParams;
import io.github.resilience4j.retry.Retry;
import io.github.resilience4j.retry.RetryConfig;

import java.time.Duration;
import java.util.UUID;
import java.util.concurrent.Callable;

Retry retry = Retry.of("stripe-billing", RetryConfig.custom()
    .maxAttempts(3)
    .waitDuration(Duration.ofMillis(500))
    .retryOnException(e -> e instanceof com.stripe.exception.StripeException)
    .build());

// Lambda is created once; the body executes on each Retry invocation.
Callable<Charge> billingCallable = Retry.decorateCallable(retry, () -> {
    // UNSAFE: evaluated per callable.call() invocation, not at lambda creation time.
    // Attempt 1: UUID = "4c8a3b1d-2e5f-4b7e-9c0a-1a2b3c4d5e6f" → ch_A
    // Attempt 2: UUID = "d7e2f4a6-3b8c-4f1e-0d9a-2b3c4d5e6f7a" → ch_B ← duplicate
    String idempotencyKey = UUID.randomUUID().toString();

    RequestOptions options = RequestOptions.builder()
        .setIdempotencyKey(idempotencyKey)
        .build();
    ChargeCreateParams params = ChargeCreateParams.builder()
        .setAmount(amountCents)
        .setCurrency("usd")
        .setCustomer(customerId)
        .build();
    return Charge.create(params, options);
});

try {
    Charge charge = billingCallable.call();
} catch (Exception e) {
    log.error("Billing failed for customer {}", customerId, e);
}
// Attempt 1: idempotencyKey="4c8a3b1d-..." → POST /v1/charges → StripeException
//            (socket timeout after 30s — Stripe committed ch_A before timeout fired)
// Attempt 2: idempotencyKey="d7e2f4a6-..." → POST /v1/charges → ch_B committed at Stripe
//            Stripe returns 200 for attempt 2 — caller sees success but two charges exist

The exact timing of the charge commitment matters. Stripe processes idempotency at the server: when a POST /v1/charges request is received, Stripe checks the Idempotency-Key header against its 24-hour idempotency cache for the same API key. If the key is new, Stripe begins processing the charge and atomically stores the key-to-result mapping when the charge succeeds. The HTTP response carrying the charge object (or error body) then travels back across the network. A socket timeout on the client fires when the client has been waiting longer than its configured read timeout for bytes to arrive on the socket — it does not fire because Stripe failed. Stripe may have completed the charge and started sending the response before the timeout fired. The client interrupted its wait, concluded the request failed, and Resilience4j’s retry mechanism fires attempt 2. Attempt 2 uses a different idempotency key. Stripe sees it as a new request, creates a new charge, and returns 200. Two charges for the same billing period.

The Resilience4j Decorators fluent API produces the same root failure but can mask it more thoroughly because the chain looks like a single composed callable:

// UNSAFE: Decorators.ofCallable() chain — UUID inside the innermost supplier lambda.
// Each decorator in the chain (Retry, CircuitBreaker, RateLimiter) may re-invoke
// the underlying callable on retries, half-open probes, or rate-limit backoffs.

import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.ratelimiter.RateLimiter;
import io.github.resilience4j.decorators.Decorators;

CircuitBreaker cb = CircuitBreaker.ofDefaults("stripe-cb");
Retry retry = Retry.ofDefaults("stripe-retry");
RateLimiter rl = RateLimiter.ofDefaults("stripe-rl");

// The supplier lambda is captured as the innermost callable.
// Every layer in the decorator chain calls this lambda when it decides to proceed.
Callable<Charge> decorated = Decorators.ofCallable(() -> {
    // UNSAFE: evaluated per invocation through any decorator layer
    String key = UUID.randomUUID().toString();
    return chargeCustomerViaStripe(customerId, amountCents, key);
})
.withCircuitBreaker(cb)
.withRetry(retry)
.withRateLimiter(rl)
.decorate();

Charge charge = decorated.call();
// Retry re-invokes the entire inner callable including the supplier lambda on each attempt.
// UUID.randomUUID() fires per invocation regardless of the chain depth.

The fix is to compute the stable key once in the calling scope — before the lambda is constructed — and capture it as a closed-over variable. A closed-over variable in a Java lambda is evaluated at the point in code where it appears in the outer scope, not inside the lambda body. Resilience4j re-invokes the lambda body but re-invocations of the lambda body resolve the closed-over variable to the same value that was assigned in the outer scope:

// SAFE: stableKey computed before the lambda, captured as effectively-final variable.
// Resilience4j re-invokes the lambda body on each retry,
// but stableKey resolves to the same String object every time.

import java.security.MessageDigest;
import java.nio.charset.StandardCharsets;

private static String computeStableKey(String customerId, String billingPeriod, String scope) {
    try {
        MessageDigest digest = MessageDigest.getInstance("SHA-256");
        String input = customerId + ":" + billingPeriod + ":" + scope;
        byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8));
        StringBuilder hex = new StringBuilder();
        for (byte b : hash) hex.append(String.format("%02x", b));
        return hex.substring(0, 32); // Stripe idempotency key: up to 255 chars
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

// ---

// Outer scope: stableKey computed once per billing intent, never inside the lambda.
final String stableKey = computeStableKey(customerId, billingPeriod, "resilience4j-billing");

Callable<Charge> billingCallable = Retry.decorateCallable(retry, () -> {
    // stableKey is a closed-over reference — same value on every re-invocation.
    // Attempt 1: stableKey="a3f2c1b9d8e7f654" → POST /v1/charges → StripeException
    // Attempt 2: stableKey="a3f2c1b9d8e7f654" → Stripe sees same Idempotency-Key
    //            → Stripe returns the ch_A result without creating ch_B
    RequestOptions options = RequestOptions.builder()
        .setIdempotencyKey(stableKey)
        .build();
    ChargeCreateParams params = ChargeCreateParams.builder()
        .setAmount(amountCents)
        .setCurrency("usd")
        .setCustomer(customerId)
        .build();
    return Charge.create(params, options);
});

The content-hash key sha256(customerId + ":" + billingPeriod + ":resilience4j-billing")[:32] encodes only stable, deterministic inputs that are known before any retry loop starts. It excludes UUID.randomUUID() (different per call), System.currentTimeMillis() (different per millisecond), Thread.currentThread().getId() (different per thread), and any Resilience4j retry state (attempt count, delay timestamp). The resulting key is the same on every retry attempt, on every pod in a Kubernetes deployment, and on every retry after a JVM restart — which is exactly what Stripe’s idempotency guarantee requires to be useful.

The same rule applies to the Decorators chain. Pass the stable key as a parameter to the innermost business method rather than computing it inside the supplier lambda:

// SAFE: Decorators chain with stable key computed before the chain is decorated.
final String stableKey = computeStableKey(customerId, billingPeriod, "resilience4j-billing");

Callable<Charge> decorated = Decorators.ofCallable(() ->
    chargeCustomerViaStripe(customerId, amountCents, stableKey) // stableKey from outer scope
)
.withCircuitBreaker(cb)
.withRetry(retry)
.withRateLimiter(rl)
.decorate();

Failure mode 2: `TimeLimiter` combined with `Retry` — timeout fires after Stripe commits ch_A — `Retry` fires a second invocation with fresh `UUID.randomUUID()` — Stripe creates ch_B

Resilience4j’s TimeLimiter enforces a wall-clock timeout on a callable by submitting it to a thread pool executor and scheduling a cancellation task. When the configured timeoutDuration elapses without the callable completing, TimeLimiter calls future.cancel(interruptIfRunning) on the underlying Future and throws TimeoutException to the calling thread. This mechanism cancels the JVM thread — it does not cancel the network bytes already in-flight to Stripe.

The sequence that produces a duplicate charge:

TimeLimiter submits the billing callable to a ScheduledExecutorService thread. The thread constructs the request, opens a TCP connection (or acquires a pooled connection), and writes the HTTP bytes for POST /v1/charges to the socket send buffer. The OS kernel flushes the send buffer to the network interface. The bytes reach Stripe’s load balancer, then Stripe’s charge processing backend.
Stripe’s backend receives the request, validates the API key, validates the Idempotency-Key header as new, runs the charge workflow, debits the card, and atomically stores the Idempotency-Key → charge_object mapping. ch_A is now committed in Stripe’s ledger and idempotency cache. Stripe’s server starts writing the HTTP response (200 with the charge JSON) to the TCP socket toward the client.
The HTTP response bytes are traveling from Stripe’s datacenter to the client. Network round-trip time is 40–80ms for a cross-region call. The timeoutDuration configured as Duration.ofSeconds(5) fires because Stripe’s charge workflow — including card network authorization and fraud scoring — took 4.9 seconds, and the response has not yet been fully received by the client’s read buffer.
TimeLimiter calls future.cancel(true). The executor thread receives InterruptedException while blocked on InputStream.read() waiting for the response. The socket read is interrupted, the thread throws SocketException: Socket closed or InterruptedIOException, which wraps into a StripeException or propagates as a raw TimeoutException through TimeLimiter’s TimeoutException wrapper.
Retry’s retryOnException predicate matches (TimeoutException or StripeException → true). Retry waits waitDuration and re-invokes the decorated callable. The callable body executes again. UUID.randomUUID() inside the callable evaluates fresh. Stripe receives a new POST /v1/charges with a distinct Idempotency-Key, treats it as a new charge request, and creates ch_B.

// UNSAFE: TimeLimiter + Retry composition with UUID.randomUUID() inside the callable.
// TimeLimiter may fire after ch_A is committed at Stripe; Retry creates ch_B.

import io.github.resilience4j.timelimiter.TimeLimiter;
import io.github.resilience4j.timelimiter.TimeLimiterConfig;

import java.util.concurrent.*;

TimeLimiter timeLimiter = TimeLimiter.of("stripe-tl", TimeLimiterConfig.custom()
    .timeoutDuration(Duration.ofSeconds(5))
    .cancelRunningFuture(true)
    .build());

Retry retry = Retry.of("stripe-retry", RetryConfig.custom()
    .maxAttempts(3)
    .waitDuration(Duration.ofSeconds(1))
    // retryOnException matches both TimeoutException and StripeException —
    // same predicate handles the timeout-then-retry path and the stripe-error path
    .retryOnException(e -> e instanceof TimeoutException
        || e instanceof com.stripe.exception.StripeException)
    .build());

ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(4);

// TimeLimiter wraps a FutureSupplier. The Callable inside CompletableFuture.supplyAsync()
// is re-executed on every Retry attempt because Retry.decorateCallable() re-invokes
// the entire TimeLimiter callable (which creates a new CompletableFuture each time).
Callable<Charge> withTimeLimiter = TimeLimiter.decorateFutureSupplier(
    timeLimiter,
    scheduler,
    () -> CompletableFuture.supplyAsync(() -> {
        // UNSAFE: UUID.randomUUID() inside the supplyAsync() body — re-evaluated per attempt.
        String key = UUID.randomUUID().toString();
        return chargeCustomer(customerId, amountCents, key);
    }, scheduler)
);

Callable<Charge> withRetry = Retry.decorateCallable(retry, withTimeLimiter);

try {
    Charge charge = withRetry.call();
} catch (Exception e) {
    // Attempt 1: key="4c8a3b1d-..." → POST /v1/charges → Stripe commits ch_A
    //            → response in-flight → TimeLimiter fires TimeoutException
    // Attempt 2: key="d7e2f4a6-..." → Stripe creates ch_B ← duplicate charge
    log.error("Billing failed", e);
}

Subtler variant: `cancelRunningFuture(false)` creates concurrent duplicate charges

The TimeLimiterConfig.cancelRunningFuture(false) option instructs TimeLimiter to throw TimeoutException to the calling thread but leave the underlying Future running in the thread pool. This is sometimes used to avoid spurious InterruptedException propagation in downstream libraries that do not tolerate thread interruption. But for Stripe billing, cancelRunningFuture(false) means attempt 1 and attempt 2 can both be in-flight to Stripe simultaneously:

// EVEN MORE UNSAFE: cancelRunningFuture(false) — attempt 1 continues running
// while Retry fires attempt 2 after the timeout.

TimeLimiter timeLimiter = TimeLimiter.of("stripe-tl", TimeLimiterConfig.custom()
    .timeoutDuration(Duration.ofSeconds(5))
    .cancelRunningFuture(false) // ← does NOT cancel the executor thread
    .build());

// Sequence:
// t=0.000s: attempt 1 starts — key="4c8a3b1d-..." → POST /v1/charges sent to Stripe
// t=5.000s: timeoutDuration fires — TimeoutException thrown to calling thread
//           BUT the executor thread is still running (cancelRunningFuture=false)
//           The executor thread is still reading Stripe's response for attempt 1
// t=6.000s: Retry fires attempt 2 — key="d7e2f4a6-..." → POST /v1/charges sent to Stripe
// t=6.200s: attempt 1's executor thread receives Stripe's 200 for ch_A (committed at t=4.8s)
// t=6.400s: attempt 2 reaches Stripe — distinct Idempotency-Key → ch_B created
// Result: ch_A and ch_B both committed at Stripe. Caller sees attempt 2's success result.
//         Attempt 1's result is discarded (the Future returned by CompletableFuture.supplyAsync()
//         has already been abandoned by the timeout machinery).

The fix combines two defenses. The primary fix is the stable key computed before the TimeLimiter scope — Stripe’s idempotency deduplication then handles the case where both attempt 1 and attempt 2 reach Stripe, returning ch_A for attempt 2 because the key is the same. The secondary fix is cancelRunningFuture(true) (the default) to interrupt the original thread and stop sending bytes to the network where possible:

// SAFE: stable key computed before TimeLimiter scope; captured as final variable.
// Stripe deduplicates attempts 1 and 2 via idempotency key match.
final String stableKey = computeStableKey(customerId, billingPeriod, "resilience4j-billing");

TimeLimiter timeLimiter = TimeLimiter.of("stripe-tl", TimeLimiterConfig.custom()
    .timeoutDuration(Duration.ofSeconds(5))
    .cancelRunningFuture(true) // cancel the running future on timeout
    .build());

Callable<Charge> withTimeLimiter = TimeLimiter.decorateFutureSupplier(
    timeLimiter,
    scheduler,
    () -> CompletableFuture.supplyAsync(() -> {
        // stableKey captured from outer scope — same on every invocation
        return chargeCustomer(customerId, amountCents, stableKey);
    }, scheduler)
);

Callable<Charge> withRetry = Retry.decorateCallable(retry, withTimeLimiter);

// Attempt 1: stableKey="a3f2c1b9d8e7f654" → POST /v1/charges → Stripe commits ch_A
//            → response in-flight → TimeLimiter fires TimeoutException
// Attempt 2: stableKey="a3f2c1b9d8e7f654" → Stripe sees same Idempotency-Key
//            → Stripe returns ch_A result (committed in attempt 1) → no ch_B
//            → caller sees successful charge result on attempt 2

There is one important nuance about Stripe’s idempotency deduplication behavior. If attempt 1 reached Stripe but Stripe was still processing the charge (not yet committed) when the timeout fired, and attempt 2 arrives at Stripe with the same idempotency key while attempt 1’s processing is still in progress, Stripe returns a 409 idempotency_key_in_use error. This is the correct behavior — Stripe is protecting you from concurrent duplicate processing. Resilience4j’s retryOnException predicate should not retry on 409 errors. Configure it to match only transient failures, not Stripe’s idempotency conflict signal:

// Narrow the retryOnException predicate to exclude Stripe 409 (idempotency conflict).
Retry retry = Retry.of("stripe-retry", RetryConfig.custom()
    .maxAttempts(3)
    .waitDuration(Duration.ofSeconds(2))
    .retryOnException(e -> {
        if (e instanceof com.stripe.exception.IdempotencyException) return false; // 409
        if (e instanceof com.stripe.exception.InvalidRequestException) return false; // 4xx
        if (e instanceof com.stripe.exception.AuthenticationException) return false; // 401
        if (e instanceof com.stripe.exception.PermissionException) return false;    // 403
        return e instanceof com.stripe.exception.ApiConnectionException             // network
            || e instanceof com.stripe.exception.ApiException                       // 5xx
            || e instanceof TimeoutException;
    })
    .build());

Failure mode 3: per-JVM `ScheduledExecutorService` billing on Kubernetes `replicas:3` — TOCTOU race — all three pods call `Retry.executeCallable()` with distinct `UUID.randomUUID()` per customer — ch_A, ch_B, ch_C per customer per billing period

A billing job implemented as a ScheduledExecutorService.scheduleAtFixedRate() Runnable creates one independent timer per JVM. When a Kubernetes Deployment with replicas: 3 runs this code, three pods execute the billing job simultaneously with no cross-pod coordination. The TOCTOU (time-of-check-to-time-of-use) race condition occurs between the “check whether this customer has been billed this period” database read and the “write the billing-started record” database write:

// UNSAFE: per-JVM ScheduledExecutorService on Kubernetes replicas:3.
// All three pods fire the billing loop simultaneously and independently.

import io.github.resilience4j.retry.Retry;
import io.github.resilience4j.retry.RetryConfig;

import java.util.concurrent.*;

ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();

scheduler.scheduleAtFixedRate(() -> {
    // All three pods query this at the same time, before any pod writes billing-started.
    List<String> customers = db.getCustomersDueForBilling(billingPeriod);
    // Pod 1: customers = [cust_A, cust_B, ..., cust_500]  (before any pod writes)
    // Pod 2: customers = [cust_A, cust_B, ..., cust_500]  (before any pod writes)
    // Pod 3: customers = [cust_A, cust_B, ..., cust_500]  (before any pod writes)
    // All 500 customers billed 3 times each.

    Retry retry = Retry.of("stripe", RetryConfig.custom()
        .maxAttempts(3)
        .retryOnException(e -> e instanceof com.stripe.exception.StripeException)
        .build());

    for (String customerId : customers) {
        try {
            retry.executeCallable(() -> {
                // UNSAFE: UUID.randomUUID() is distinct per pod, per attempt.
                // Pod 1: cust_A → key="4c8a3b1d-..." → ch_A
                // Pod 2: cust_A → key="d7e2f4a6-..." → ch_B
                // Pod 3: cust_A → key="9f1e3c7b-..." → ch_C
                String key = UUID.randomUUID().toString();
                return chargeCustomer(customerId, billingPeriod, key);
            });
        } catch (Exception e) {
            log.error("Billing failed for customer {}", customerId, e);
        }
    }
}, 0, 1, TimeUnit.DAYS);
// Result: 1500 charges for 500 customers. Per-billing-period overcharge of 3x.

Subtler variant: thread pool `ThreadPoolBulkhead` with `BulkheadFullException` on pool saturation — `Retry` catches `BulkheadFullException` with the same predicate as `StripeException` — ch_B from a single pod

A thread pool bulkhead is often added to a billing service to cap the number of concurrent Stripe requests and avoid overwhelming the connection pool or Stripe’s rate limiter. The ThreadPoolBulkhead accepts tasks up to its maxThreadPoolSize + queueCapacity limit and throws BulkheadFullException for any excess. In a billing run of 500 customers, a bulkhead configured with maxThreadPoolSize=4 and queueCapacity=10 will throw BulkheadFullException for approximately 486 of the 500 submissions before the thread pool drains.

The duplicate-charge risk arises not from BulkheadFullException itself (which fires before the request reaches Stripe) but from a Retry configuration that catches both BulkheadFullException and StripeException without distinguishing them. A StripeException may fire after Stripe has already committed ch_A (for example, a socket timeout fires after Stripe processed the charge but before the response arrives). When that StripeException reaches the same retryOnException predicate that also catches BulkheadFullException, Resilience4j fires a retry with a fresh UUID.randomUUID() inside the callable — creating ch_B:

// UNSAFE: Retry catches BulkheadFullException and StripeException with the same predicate.
// A StripeException from an attempt that committed ch_A triggers the same retry path
// as BulkheadFullException — Retry fires with fresh UUID.randomUUID() creating ch_B.

import io.github.resilience4j.bulkhead.ThreadPoolBulkhead;
import io.github.resilience4j.bulkhead.ThreadPoolBulkheadConfig;
import io.github.resilience4j.bulkhead.BulkheadFullException;

ThreadPoolBulkheadConfig bulkheadConfig = ThreadPoolBulkheadConfig.custom()
    .maxThreadPoolSize(4)
    .coreThreadPoolSize(2)
    .queueCapacity(10) // saturated by a billing run of 500 customers
    .keepAliveDuration(Duration.ofMillis(20))
    .build();
ThreadPoolBulkhead bulkhead = ThreadPoolBulkhead.of("stripe-bulkhead", bulkheadConfig);

// UNSAFE: retryOnException catches both BulkheadFullException and StripeException.
// A StripeException from an attempt that already committed ch_A will hit this predicate
// and Retry fires with fresh UUID.randomUUID() → ch_B.
Retry retry = Retry.of("stripe-retry", RetryConfig.custom()
    .maxAttempts(3)
    .waitDuration(Duration.ofMillis(200))
    .retryOnException(e ->
        e instanceof BulkheadFullException
        || e instanceof com.stripe.exception.StripeException) // ← unsafe conflation
    .build());

for (String customerId : customers) {
    Callable<Charge> billingCallable = Retry.decorateCallable(retry, () -> {
        // Wrapped inside Retry, which wraps the bulkhead submission.
        // UUID.randomUUID() re-evaluates on every Retry attempt.
        final String key = UUID.randomUUID().toString();
        try {
            CompletableFuture<Charge> future = bulkhead.executeSupplier(
                () -> chargeCustomer(customerId, billingPeriod, key)
            );
            return future.get();
        } catch (BulkheadFullException e) {
            throw e; // propagates to Retry — fires with fresh UUID on next attempt
        }
    });
    billingCallable.call();
}

The bulkhead-specific fix has two parts. First, separate the retryOnException predicates so that BulkheadFullException does not trigger a retry that re-invokes the charge logic with a new UUID — instead, BulkheadFullException should trigger a wait-and-enqueue strategy, not a full re-execution of the charge callable. Second, compute the stable key outside the Retry scope:

// SAFE: stable key computed before the Retry scope;
// BulkheadFullException handled separately from StripeException.

// Strategy for BulkheadFullException: use a simple blocking retry with backoff,
// or queue the customer ID for a secondary pass — do NOT re-invoke the charge callable.
Retry bulkheadRetry = Retry.of("bulkhead-retry", RetryConfig.custom()
    .maxAttempts(10)
    .waitDuration(Duration.ofMillis(100))
    .retryOnException(e -> e instanceof BulkheadFullException)
    .build());

// Separate Retry for transient Stripe errors that may fire AFTER charge commitment.
// Uses stable key and a narrow predicate — does NOT catch BulkheadFullException.
Retry stripeRetry = Retry.of("stripe-retry", RetryConfig.custom()
    .maxAttempts(3)
    .waitDuration(Duration.ofSeconds(2))
    .retryOnException(e -> {
        if (e instanceof com.stripe.exception.IdempotencyException) return false;
        if (e instanceof com.stripe.exception.InvalidRequestException) return false;
        return e instanceof com.stripe.exception.ApiConnectionException
            || e instanceof com.stripe.exception.ApiException;
    })
    .build());

for (String customerId : customers) {
    // Stable key computed per customer per billing period, outside ALL retry scopes.
    final String stableKey = computeStableKey(customerId, billingPeriod, "resilience4j-billing");

    // Pre-flight: authoritative cluster-wide billing mutex.
    // ON CONFLICT DO NOTHING prevents double-billing if two pods race.
    boolean inserted = db.insertBillingRecord(customerId, billingPeriod); // ON CONFLICT DO NOTHING
    if (!inserted) {
        log.info("Billing already started for customer {} period {}", customerId, billingPeriod);
        continue;
    }

    Callable<Charge> chargeCallable = stripeRetry.decorateCallable(() ->
        chargeCustomer(customerId, billingPeriod, stableKey) // stable key from outer scope
    );

    // Bulkhead retry wraps the enqueue attempt, not the charge callable.
    bulkheadRetry.executeRunnable(() -> {
        CompletableFuture<Charge> future = bulkhead.executeCallable(chargeCallable);
        // future.get() blocks the billing loop thread — use async pattern for real systems
    });
}

Fixing the cross-pod TOCTOU race with `pg_try_advisory_lock()`

The pre-flight ON CONFLICT DO NOTHING on the billing record is the authoritative cluster-wide mutex at the per-customer level. But 500 per-customer INSERT statements still leave a window between the getCustomersDueForBilling() read and the per-customer INSERT: all three pods may read the same 500-customer list before any pod’s per-customer INSERT commits, causing each pod to attempt 500 inserts with only 500 successes total spread across pods. A batch-level cross-pod lock eliminates the redundant work:

// SAFE: pg_try_advisory_lock() as cross-pod batch-level mutex.
// Only one pod executes the billing run; others exit immediately.

private void runMonthlyBilling(String billingPeriod, Connection conn) throws SQLException {
    long lockKey = Math.abs(("resilience4j-monthly-billing:" + billingPeriod).hashCode());
    try (PreparedStatement lock = conn.prepareStatement(
            "SELECT pg_try_advisory_lock(?)")) {
        lock.setLong(1, lockKey);
        ResultSet rs = lock.executeQuery();
        rs.next();
        if (!rs.getBoolean(1)) {
            // Another pod holds the lock — this billing period is already being processed.
            log.info("Billing lock held by another pod for period {}", billingPeriod);
            return;
        }
    }

    try {
        List<String> customers = db.getCustomersDueForBilling(billingPeriod);
        for (String customerId : customers) {
            // Per-customer pre-flight: authoritative charge-level mutex.
            boolean inserted = db.insertBillingRecord(customerId, billingPeriod);
            if (!inserted) continue; // already processed by this pod earlier in the run

            final String stableKey = computeStableKey(customerId, billingPeriod, "resilience4j-billing");
            Retry retry = Retry.of("stripe", narrowStripeRetryConfig());
            retry.executeCallable(() -> chargeCustomer(customerId, billingPeriod, stableKey));
        }
    } finally {
        try (PreparedStatement unlock = conn.prepareStatement(
                "SELECT pg_advisory_unlock(?)")) {
            unlock.setLong(1, lockKey);
            unlock.execute();
        }
    }
}

The combination of pg_try_advisory_lock() at the batch level and INSERT ... ON CONFLICT DO NOTHING at the per-customer level provides defense in depth. The advisory lock serializes which pod runs the billing loop. The per-customer INSERT guards against the window between lock acquisition and per-customer processing in case a previous billing run was interrupted mid-loop. The stable idempotency key ensures that any Stripe retry within the per-customer Retry decorator uses the same key, so Stripe’s deduplication handles the case where a charge was committed on attempt 1 but the response was not received.

Vault keys as a financial backstop

Content-hash idempotency keys, pg_try_advisory_lock(), and per-customer ON CONFLICT DO NOTHING together prevent duplicate charges under normal operating conditions and most failure scenarios. A vault key provides a financial backstop for the scenarios they don’t cover: a subtle key-computation bug that produces collisions across customers, a database split-brain during the advisory lock check, or a misconfigured Retry predicate deployed by a future engineer who doesn’t know the billing contract. The vault key caps the maximum financial exposure of a single billing run at expected_total × 1.10:

// Vault key for Resilience4j-backed billing: capped at expected total × 1.10.
// Even if all defenses fail and Resilience4j fires 3× retries for 500 customers,
// the vault key spend cap limits total Stripe charges to configured maximum.

// In your Keybrake policy (pseudo-code, illustrative):
{
    "vendor": "stripe",
    "daily_usd_cap": (expected_monthly_billing_usd / 28) * 1.10,
    "allowed_endpoints": ["/v1/charges", "/v1/customers"],
    "description": "Resilience4j billing service — capped at 110% of expected monthly total"
}

A vault key spend cap enforced at the proxy layer fires regardless of what logic is running in your Resilience4j service. It is the one defense that survives a completely buggy billing implementation — if the Retry configuration retries 100 times on every exception and every retry fires a new charge with a fresh UUID, the spend cap halts the bleeding at a fixed dollar amount rather than letting it run to exhaustion. It is not a replacement for correct idempotency key handling, but it is the safety net that makes every other defense less critical to get perfectly right on the first deploy.

Summary table

Failure mode	Root cause	Subtler variant	Fix
`Retry.decorateCallable()` re-invokes lambda	`UUID.randomUUID()` inside lambda body evaluates per invocation — initial attempt ch_A, retry ch_B	`Decorators.ofCallable()` chain — multiple decorator layers each re-invoke the inner callable	Compute `stableKey` before lambda construction; capture as closed-over `final` variable
`TimeLimiter` + `Retry` composition	`TimeLimiter` fires `TimeoutException` after ch_A committed at Stripe; Retry fires with fresh UUID — ch_B	`cancelRunningFuture(false)` — original callable continues; attempt 1 and attempt 2 both in-flight simultaneously	Stable key before `TimeLimiter` scope; `cancelRunningFuture(true)`; narrow `retryOnException` predicate
Per-JVM `ScheduledExecutorService` on `replicas:3`	TOCTOU race — all three pods read customer list before any pod writes billing-started — ch_A, ch_B, ch_C per customer	`ThreadPoolBulkhead` + `Retry` with undifferentiated `retryOnException` predicate — `BulkheadFullException` and `StripeException` both trigger retry with fresh UUID	`pg_try_advisory_lock()` as batch mutex; per-customer `ON CONFLICT DO NOTHING`; stable key; vault key cap at `expected × 1.10`

Implementation checklist for Resilience4j + Stripe billing

Never call UUID.randomUUID() inside a Resilience4j decorated callable. Compute the idempotency key before the lambda, capture it as an effectively-final variable in the outer scope, and pass it as a parameter or reference inside the lambda. This applies to Retry.decorateCallable(), Retry.decorateSupplier(), Retry.executeCallable(), and every Decorators.ofCallable() chain.
Never call System.currentTimeMillis(), Instant.now(), or Thread.currentThread().getId() inside a decorated callable as part of an idempotency key. These values differ per millisecond and per thread. Use deterministic inputs: customer ID, billing period string, and a fixed scope suffix.
Place the TimeLimiter timeout inside the Stripe Java SDK read timeout, not as a replacement for it. Stripe’s SDK has a default read timeout of 80 seconds. If your TimeLimiter fires at 5 seconds, you are timing out before Stripe’s own timeout — but the request is often in-flight and Stripe may commit the charge in the remaining 75 seconds while Retry is firing a second attempt. Configure Stripe’s SDK read timeout to align with your TimeLimiter duration: set RequestOptions.builder().setReadTimeout(5000) to match TimeLimiterConfig.timeoutDuration(Duration.ofSeconds(5)), so both the client and the SDK agree on when to give up.
Separate BulkheadFullException retry behavior from StripeException retry behavior. BulkheadFullException fires before the request reaches Stripe — no charge has been committed — and the appropriate response is to wait and retry the enqueue, not to re-invoke the charge callable with a fresh UUID. StripeException may fire after the charge is committed — the stable key approach handles this, but the predicates should not conflate the two exception types.
Add a vault key spend cap at expected_total × 1.10. The cap fires at the proxy layer regardless of what Resilience4j configuration is deployed. It is the last line of defense after all code-level idempotency fixes.

Put a spend cap on your Resilience4j billing service

Keybrake issues scoped vault keys for Stripe, Twilio, and Resend — with per-vendor daily spend caps, allowlisted endpoints, and a one-click kill switch. A billing service running Resilience4j retries gets a hard financial ceiling even when the idempotency logic has a bug.

Failure mode 1: Retry.decorateCallable() re-invokes the supplier lambda on every retry attempt — UUID.randomUUID() inside the lambda evaluates fresh per invocation — initial attempt creates ch_A before StripeException — first retry creates ch_B

Failure mode 2: TimeLimiter combined with Retry — timeout fires after Stripe commits ch_A — Retry fires a second invocation with fresh UUID.randomUUID() — Stripe creates ch_B

Subtler variant: cancelRunningFuture(false) creates concurrent duplicate charges

Failure mode 3: per-JVM ScheduledExecutorService billing on Kubernetes replicas:3 — TOCTOU race — all three pods call Retry.executeCallable() with distinct UUID.randomUUID() per customer — ch_A, ch_B, ch_C per customer per billing period

Subtler variant: thread pool ThreadPoolBulkhead with BulkheadFullException on pool saturation — Retry catches BulkheadFullException with the same predicate as StripeException — ch_B from a single pod

Fixing the cross-pod TOCTOU race with pg_try_advisory_lock()