Apache HttpClient 5 and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Apache HttpClient 5’s DefaultHttpRequestRetryStrategy re-runs the full HttpRequestInterceptor chain on every retry attempt — UUID.randomUUID() inside Interceptor.process() to generate the Idempotency-Key header produces a different value on the initial request and on HC5’s transparent retry: the initial POST /v1/charges creates ch_A before a NoHttpResponseException from a stale pooled connection, and RetryExec fires the interceptor chain again on the retry, evaluates a fresh UUID, and Stripe creates ch_B. Three Apache HttpClient 5-specific Stripe billing failure modes: an HttpRequestInterceptor computes UUID.randomUUID() per process() invocation — subtler variant: HttpComponentsClientHttpRequestFactory in Spring 6.x’s RestClient re-runs the underlying CloseableHttpClient interceptor chain on every @Retryable re-invocation; a CloseableHttpAsyncClient retry that rebuilds the AsyncRequestProducer inside the FutureCallback.failed() handler — UUID.randomUUID() at producer-construction time creates ch_B on the first retry — subtler variant: HC5’s HTTP/2 GOAWAY-triggered transparent retry rebuilds the producer with a fresh UUID; and a per-JVM ScheduledExecutorService fires the billing job independently on all Kubernetes replicas — with replicas:3, three pods pass the concurrent database check before any pod commits, three distinct UUID.randomUUID() values per customer, ch_A, ch_B, and ch_C per billing period.

This post covers all three failure modes with Java code, content-hash idempotency keys stable across HC5 interceptor re-invocations, async producer rebuilds, and multi-pod concurrent billing loops, HttpClientContext as the key-passing mechanism that decouples key computation from interceptor execution, pg_try_advisory_lock() for cross-pod scheduler serialization, pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For the AOP proxy retry pattern in Spring Boot (@Retryable), see the Spring Boot and Stripe Integration post. For the Apache HttpClient 4.x interceptor pattern in Dropwizard, see the Dropwizard and Stripe Integration post. For the OkHttp application interceptor pattern, see the OkHttp and Retrofit Stripe Integration post.

Failure mode 1: HttpRequestInterceptor.process() computes UUID.randomUUID() — DefaultHttpRequestRetryStrategy triggers RetryExec to re-execute the full interceptor chain on each retry — initial request creates ch_A before NoHttpResponseException — retry’s process() creates ch_B

Apache HttpClient 5 rewrote the HTTP execution model relative to HC4. The InternalHttpClient.execute() method dispatches requests through an ExecChain — a sequence of ExecChainHandler implementations that includes RetryExec, RedirectExec, ProtocolExec, and ConnectExec. RetryExec wraps the inner chain and, on catching a recoverable IOException, consults HttpRequestRetryStrategy.retryRequest(response, executionCount, context) and, if the strategy returns true, re-executes the entire inner chain from the beginning. This inner chain re-execution includes ProtocolExec, which runs all HttpRequestInterceptors registered via addRequestInterceptorFirst() or addRequestInterceptorLast(). The default DefaultHttpRequestRetryStrategy retries on NoHttpResponseException, ConnectTimeoutException, and a set of connection-reset exceptions — all of which are common when HC5’s PoolingHttpClientConnectionManager hands out a connection that the server has quietly closed.

A StripeBillingInterceptor registered via addRequestInterceptorLast() that calls UUID.randomUUID() inside its process() override appears to implement a clean separation of concerns — a dedicated interceptor for a dedicated header. But process() is called per interceptor-chain execution, not per billing operation. When RetryExec fires a retry, process() is invoked again with the same ClassicHttpRequest context but a fresh call to the method body — including a fresh UUID.randomUUID():

// StripeBillingInterceptor.java
// UNSAFE: UUID.randomUUID() computed inside process() — called on the initial
// request AND on every retry triggered by DefaultHttpRequestRetryStrategy.
// If the initial request created ch_A before the pooled connection was reset,
// the retry's fresh UUID causes Stripe to create ch_B.

import org.apache.hc.core5.http.EntityDetails;
import org.apache.hc.core5.http.HttpRequest;
import org.apache.hc.core5.http.HttpRequestInterceptor;
import org.apache.hc.core5.http.protocol.HttpContext;
import java.util.UUID;

public class StripeBillingInterceptor implements HttpRequestInterceptor {

    @Override
    public void process(HttpRequest request, EntityDetails entity, HttpContext context)
            throws HttpException, IOException {

        // UNSAFE: UUID.randomUUID() called per process() invocation.
        // Initial request:  UUID = "4c8a3b1d-2e5f-4b7e-9c0a-1a2b3c4d5e6f"
        // RetryExec retry:  UUID = "d7e2f4a6-3b8c-4f1e-0d9a-2b3c4d5e6f7a"
        String idempotencyKey = UUID.randomUUID().toString();

        request.setHeader("Idempotency-Key", idempotencyKey);
    }
}

// Registration — DefaultHttpRequestRetryStrategy retries on NoHttpResponseException:
CloseableHttpClient httpClient = HttpClients.custom()
    .addRequestInterceptorLast(new StripeBillingInterceptor())  // UNSAFE interceptor
    .setRetryStrategy(new DefaultHttpRequestRetryStrategy(3, TimeValue.ofSeconds(1)))
    .setConnectionManager(PoolingHttpClientConnectionManagerBuilder.create()
        .setMaxConnPerRoute(20)
        .setMaxConnTotal(100)
        .build())
    .build();

The failure scenario: the billing service calls chargeCustomer("cust_123", "2026-09", 9900L). HC5 selects a pooled HTTPS connection to api.stripe.com. The connection has been idle for 50 seconds in the pool. Stripe’s load balancer has already closed the server-side socket with a TCP FIN that arrived at HC5’s OS buffer while the connection sat idle. HC5’s connection validation — controlled by setValidateAfterInactivity(Duration.ofSeconds(10)) on the connection manager — did not fire because 50 seconds is less than the configured threshold. The POST /v1/charges request is dispatched through the interceptor chain. StripeBillingInterceptor.process() fires. UUID.randomUUID() returns "4c8a3b1d-2e5f-4b7e-9c0a-1a2b3c4d5e6f". The request body is written to the stale connection. Stripe’s HTTP server receives the complete request before the RST propagates back to HC5. Stripe processes the charge. ch_A is committed in Stripe’s ledger. The RST arrives at HC5 and manifests as a NoHttpResponseException when HC5 attempts to read the response. RetryExec catches the exception. DefaultHttpRequestRetryStrategy.retryRequest() returns true for NoHttpResponseException. RetryExec re-executes the inner chain. ProtocolExec re-runs all registered interceptors. StripeBillingInterceptor.process() is called again. UUID.randomUUID() returns a new, completely independent value: "d7e2f4a6-3b8c-4f1e-0d9a-2b3c4d5e6f7a". The retry reaches Stripe with a different Idempotency-Key. Stripe has ch_A cached against the original UUID. The retry’s UUID is new to Stripe. Stripe processes it as a fresh charge. ch_B is created. Customer 123 is charged $99 twice for September 2026.

This failure is structurally invisible at code review because the interceptor looks identical to a correctly implemented version that reads a pre-computed key from the request. The difference is where UUID generation happens: inside process() it fires per chain-execution; outside process(), in calling code before the request is dispatched, it fires once per billing operation regardless of how many chain-executions RetryExec triggers. The HC5 documentation does not prominently surface the fact that RetryExec re-runs the full interceptor chain, not just the network layer.

The subtler variant: HttpComponentsClientHttpRequestFactory in Spring 6.x’s RestClient — Spring’s @Retryable re-calls restClient.post()…retrieve() on each retry — the underlying CloseableHttpClient interceptor chain fires fresh per RestClient call — new UUID per Spring retry invocation

Spring Framework 6.1 introduced RestClient as the synchronous HTTP client API, with HttpComponentsClientHttpRequestFactory as the recommended factory when HC5 is on the classpath. Each call to restClient.post().uri(...).body(body).retrieve().body(ChargeResponse.class) creates a new ClassicHttpRequest via the factory and dispatches it through CloseableHttpClient.execute(). This is a separate execution from HC5’s own RetryExec retry — it is a Spring-layer retry initiated by @Retryable. When @Retryable re-invokes the annotated billing method body, the method calls restClient.post()…retrieve() again, which creates a fresh ClassicHttpRequest and dispatches it through CloseableHttpClient.execute() — running the registered interceptor chain afresh:

// BillingService.java — UNSAFE: @Retryable re-invokes the method body.
// Each method invocation calls restClient.post()...retrieve() once.
// Each restClient call dispatches a fresh ClassicHttpRequest through the
// CloseableHttpClient's interceptor chain — StripeBillingInterceptor.process()
// fires again, evaluates UUID.randomUUID() independently.

import org.springframework.retry.annotation.Retryable;
import org.springframework.retry.annotation.Backoff;
import org.springframework.web.client.RestClient;

@Service
public class BillingService {

    private final RestClient restClient;

    public BillingService(RestClient.Builder builder) {
        // HttpComponentsClientHttpRequestFactory backed by CloseableHttpClient
        // with StripeBillingInterceptor registered on the underlying httpClient.
        this.restClient = builder
            .requestFactory(new HttpComponentsClientHttpRequestFactory(httpClientWithUnsafeInterceptor()))
            .build();
    }

    @Retryable(
        retryFor = { ResourceAccessException.class, HttpServerErrorException.ServiceUnavailable.class },
        maxAttempts = 3,
        backoff = @Backoff(delay = 1000, multiplier = 2)
    )
    public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents) {
        // @Retryable re-invokes this method body on each retry attempt.
        // Each invocation calls restClient.post(), which:
        //   1. Creates a new ClassicHttpRequest
        //   2. Dispatches via CloseableHttpClient.execute()
        //   3. Runs all interceptors including StripeBillingInterceptor.process()
        //   4. StripeBillingInterceptor.process() calls UUID.randomUUID() — UNSAFE
        //
        // Attempt 1: UUID = "4c8a3b1d..." → POST creates ch_A before 503
        // Attempt 2: UUID = "d7e2f4a6..." → POST creates ch_B  ← duplicate charge
        // Attempt 3: UUID = "a9b3c7e1..." → POST creates ch_C  ← third charge
        return restClient.post()
            .uri("https://api.stripe.com/v1/charges")
            .contentType(MediaType.APPLICATION_FORM_URLENCODED)
            .body(buildFormBody(customerId, amountCents))
            .retrieve()
            .body(ChargeResponse.class);
    }
}

The double-retry failure is especially pernicious here because HC5’s RetryExec and Spring’s @Retryable are both active. If HC5 retries a NoHttpResponseException internally (up to 3 times by default) and each HC5 internal retry fires the interceptor with a fresh UUID, and Spring’s @Retryable then re-invokes the billing method on a Spring-layer transient error, the total number of distinct idempotency keys sent to Stripe can reach 9 (3 Spring retries × 3 HC5 retries each) before the final failure is thrown. Each distinct UUID that reaches Stripe while ch_A is not yet in Stripe’s idempotency cache creates a new charge.

The fix for failure mode 1

The idempotency key must be computed once per billing operation and stored in a location that survives interceptor chain re-executions. HttpClientContext is the correct vehicle: it is passed through the entire ExecChain and is accessible to every interceptor. The interceptor reads the key from context rather than generating one:

// Safe: stable key computed ONCE in calling code — stored in HttpClientContext.
// Interceptor reads from context on every process() call — same value on every retry.

// BillingService.java — calling code owns key computation.
public class BillingService {

    private final CloseableHttpClient httpClient;

    public ChargeResponse chargeCustomer(String customerId, String billingPeriod, long amountCents)
            throws IOException {

        // Computed once per billing operation — deterministic across retries and pod restarts.
        String idempotencyKey = stableKey(customerId, billingPeriod);

        // Pre-flight: claim the billing slot before any network activity.
        boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
        if (!claimed) {
            return billingRepository.findExistingCharge(customerId, billingPeriod);
        }

        // Store stable key in HttpClientContext — survives interceptor chain re-executions.
        HttpClientContext context = HttpClientContext.create();
        context.setAttribute("stripe.idempotency_key", idempotencyKey);

        ClassicHttpRequest request = ClassicRequestBuilder.post("https://api.stripe.com/v1/charges")
            .addHeader("Authorization", "Bearer " + stripeKey)
            .setEntity(buildFormEntity(customerId, amountCents))
            .build();

        // RetryExec re-invokes the chain — interceptor reads from context each time.
        // context is the same object across all retries; "stripe.idempotency_key" never changes.
        try (CloseableHttpResponse response = httpClient.execute(request, context)) {
            return parseResponse(response);
        }
    }

    public static String stableKey(String customerId, String billingPeriod) throws NoSuchAlgorithmException {
        MessageDigest digest = MessageDigest.getInstance("SHA-256");
        byte[] hash = digest.digest(
            (customerId + ":" + billingPeriod + ":httpclient5-billing")
                .getBytes(StandardCharsets.UTF_8)
        );
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < 16; i++) {
            sb.append(String.format("%02x", hash[i]));
        }
        return sb.toString();
    }
}

// Safe interceptor — reads from context, never calls UUID.randomUUID().
public class StripeBillingInterceptor implements HttpRequestInterceptor {

    private static final String KEY_ATTR = "stripe.idempotency_key";

    @Override
    public void process(HttpRequest request, EntityDetails entity, HttpContext context)
            throws HttpException, IOException {

        // Safe: read from HttpClientContext — key was computed by calling code.
        // Survives any number of RetryExec re-executions: context is the same object.
        String idempotencyKey = (String) context.getAttribute(KEY_ATTR);
        if (idempotencyKey == null) {
            throw new IllegalStateException(
                "Stripe POST missing stripe.idempotency_key in HttpClientContext — " +
                "compute stableKey() in calling code and store in context before execute()");
        }

        request.setHeader("Idempotency-Key", idempotencyKey);
    }
}

The HttpClientContext is the same object across all RetryExec re-executions of the inner chain. The attribute "stripe.idempotency_key" is set once before httpClient.execute(request, context) and read by the interceptor on every process() call — initial request and all retries. Stripe receives the same Idempotency-Key on every attempt and returns the cached ch_A result from the first successful charge without creating ch_B. The pattern also decouples key computation from the interceptor entirely: the interceptor becomes a pure pass-through that validates presence and sets the header, while all key logic lives in calling code where it can be tested independently.

Failure mode 2: CloseableHttpAsyncClient AsyncRequestProducer built inside FutureCallback.failed() retry handler — UUID.randomUUID() at producer-construction time re-evaluates per retry — initial async charge creates ch_A before ConnectionRequestTimeoutException — retry’s fresh producer creates ch_B

Apache HttpClient 5 introduced a full asynchronous HTTP client API via CloseableHttpAsyncClient (from HttpAsyncClients.createDefault()). Async billing calls follow a pattern where the billing method submits a request via httpAsyncClient.execute(producer, consumer, context, callback) and the FutureCallback receives the result or exception. Teams that implement retry inside FutureCallback.failed() must decide how to produce the retry request. The unsafe pattern builds a fresh BasicRequestProducer (or a custom AsyncRequestProducer) inside the failed() handler, passing UUID.randomUUID() as the idempotency key argument:

// BillingService.java — UNSAFE async retry that builds a fresh AsyncRequestProducer on failure.
// UUID.randomUUID() evaluated at producer-construction time — a new producer means a new UUID.
// If the initial request created ch_A before ConnectionRequestTimeoutException,
// the retry's fresh UUID causes Stripe to create ch_B.

import org.apache.hc.client5.http.async.methods.BasicRequestProducer;
import org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient;
import org.apache.hc.core5.concurrent.FutureCallback;
import org.apache.hc.core5.http.message.BasicHttpRequest;

public class BillingService {

    private final CloseableHttpAsyncClient asyncClient;

    public CompletableFuture<ChargeResponse> chargeCustomerAsync(
            String customerId, String billingPeriod, long amountCents) {

        CompletableFuture<ChargeResponse> future = new CompletableFuture<>();

        // UNSAFE: buildProducer() evaluates UUID.randomUUID() at construction time.
        // If called again in failed(), the second call evaluates UUID independently.
        submitCharge(buildProducer(customerId, amountCents), future, customerId, billingPeriod, amountCents, 0);
        return future;
    }

    private BasicRequestProducer buildProducer(String customerId, long amountCents) {
        // UNSAFE: UUID.randomUUID() evaluated at call time — fresh UUID per buildProducer() call.
        String idempotencyKey = UUID.randomUUID().toString();

        BasicHttpRequest request = new BasicHttpRequest("POST", "https://api.stripe.com/v1/charges");
        request.setHeader("Authorization", "Bearer " + stripeKey);
        request.setHeader("Idempotency-Key", idempotencyKey);  // UNSAFE: different per call

        return new BasicRequestProducer(request, buildEntityProducer(customerId, amountCents));
    }

    private void submitCharge(BasicRequestProducer producer, CompletableFuture<ChargeResponse> future,
            String customerId, String billingPeriod, long amountCents, int attempt) {

        asyncClient.execute(producer, new BasicResponseConsumer<>(ChargeResponse.class),
            HttpClientContext.create(), new FutureCallback<>() {

            @Override
            public void completed(ChargeResponse result) {
                future.complete(result);
            }

            @Override
            public void failed(Exception ex) {
                if (isTransient(ex) && attempt < 3) {
                    // UNSAFE: buildProducer() called again — evaluates UUID.randomUUID() fresh.
                    // Initial request created ch_A before ConnectionRequestTimeoutException.
                    // Retry's new producer carries a different UUID — Stripe creates ch_B.
                    submitCharge(buildProducer(customerId, amountCents), future,
                                 customerId, billingPeriod, amountCents, attempt + 1);
                } else {
                    future.completeExceptionally(ex);
                }
            }

            @Override
            public void cancelled() {
                future.cancel(false);
            }
        });
    }
}

The failure scenario: chargeCustomerAsync("cust_456", "2026-09", 9900L) is called. buildProducer() is called once. UUID.randomUUID() evaluates at producer-construction time and returns "5e9a3c7b-1d4f-4b8e-0a2c-1b3c4d5e6f7a". The BasicRequestProducer carries this UUID in its Idempotency-Key header. The async client submits the request. The request enters the HC5 async I/O reactor. Before the response arrives, the connection pool is exhausted by concurrent billing operations for 499 other customers: HC5’s RequestConfig.connectionRequestTimeout expires while waiting to lease a connection. ConnectionRequestTimeoutException is thrown. But the original request body was fully sent to Stripe before the timeout fired — Stripe received the complete charge request, authorized the card, and committed ch_A. The FutureCallback.failed() handler receives ConnectionRequestTimeoutException. isTransient(ex) returns true. buildProducer() is called again. UUID.randomUUID() evaluates a second, completely independent time and returns "b3d7e9f1-2a5c-4e0b-8c1a-2d3e4f5a6b7c". The retry’s producer carries this different UUID. Stripe sees the retry as a new charge. Stripe creates ch_B. Customer 456 is charged $99 twice for September 2026.

The subtler variant: HC5 HTTP/2 with setVersionPolicy(HttpVersionPolicy.NEGOTIATE) — Stripe’s backend sends a GOAWAY frame on graceful shutdown — HC5’s H2 client transparently retries the request on a new connection — if the AsyncRequestProducer factory re-evaluates UUID.randomUUID() on retry, the GOAWAY-triggered retry creates ch_B

Apache HttpClient 5 supports HTTP/2 multiplexing via HttpAsyncClientBuilder.setVersionPolicy(HttpVersionPolicy.NEGOTIATE) or via HttpAsyncClients.createHttp2Default(). HTTP/2 allows multiple streams to share a single TCP connection. When Stripe’s backend server is gracefully shutting down — for a rolling restart or capacity change — it sends an HTTP/2 GOAWAY frame that tells the client which stream IDs it has processed and which it has not. HC5’s H2 async client transparently retries any request whose stream ID is above the GOAWAY’s lastStreamId on a new connection — the client code never sees an exception for these retries.

This transparent H2 retry path calls AsyncRequestProducer.produce(DataStreamChannel channel) to re-send the request body on the new connection. If the AsyncRequestProducer was designed to generate the Idempotency-Key inside its produce() method or at its factory construction time during the retry callback, the GOAWAY-triggered transparent retry evaluates a fresh UUID. Unlike HC5’s RetryExec retry path (which re-runs the interceptor chain and is visible to HttpRequestInterceptor.process()), the H2 transparent retry path calls produce() directly on the existing producer — so the danger is specifically when the producer is rebuilt fresh per retry, not when it is reused:

// UNSAFE: AsyncRequestProducer built inside a retry supplier that is called per attempt.
// If HC5's H2 client retries on GOAWAY (transparent, no exception raised),
// the retry path rebuilds the producer via the supplier — UUID re-evaluates.

// H2-capable async client:
CloseableHttpAsyncClient h2Client = HttpAsyncClients.custom()
    .setVersionPolicy(HttpVersionPolicy.NEGOTIATE)  // prefer HTTP/2
    .setRetryStrategy(new DefaultHttpRequestRetryStrategy(2, TimeValue.ofSeconds(1)))
    .build();
h2Client.start();

// UNSAFE retry pattern with producer factory:
Supplier<BasicRequestProducer> producerFactory = () -> {
    // UNSAFE: every call to producerFactory.get() evaluates UUID.randomUUID() fresh.
    // H2 GOAWAY retry calls producerFactory.get() on the transparent retry.
    String key = UUID.randomUUID().toString();
    BasicHttpRequest req = new BasicHttpRequest("POST", "https://api.stripe.com/v1/charges");
    req.setHeader("Idempotency-Key", key);  // UNSAFE: different per factory call
    req.setHeader("Authorization", "Bearer " + stripeKey);
    return new BasicRequestProducer(req, buildEntityProducer(customerId, amountCents));
};

// If the H2 retry path calls producerFactory.get() again, UUID re-evaluates → ch_B.
h2Client.execute(producerFactory.get(), consumer, context, callback);

The GOAWAY failure is harder to detect than an explicit FutureCallback.failed() retry because it is transparent — the billing code does not see an exception, does not explicitly trigger a retry, and does not have an obvious place to inspect the retry count. The duplicate charge appears in Stripe’s dashboard without any corresponding error in the billing service’s logs. The fix applies the same principle as failure mode 1: compute the stable key once, before the first execute() call, and build the AsyncRequestProducer with that key; do not rebuild the producer on retry.

The fix for failure mode 2

The stable key must be computed before any AsyncRequestProducer is constructed and stored as a final variable or HttpClientContext attribute. The AsyncRequestProducer is built once and reused for the retry — not rebuilt inside the failed() handler:

// Safe: stable key computed ONCE before any producer is built.
// Producer built with stable key — reused on retry via the same producer object.

public CompletableFuture<ChargeResponse> chargeCustomerAsync(
        String customerId, String billingPeriod, long amountCents) throws Exception {

    // Computed once per billing operation — stable across all retries and reconnects.
    String idempotencyKey = BillingService.stableKey(customerId, billingPeriod);

    // Pre-flight: claim the billing slot before async dispatch.
    boolean claimed = billingRepository.claimSlotAsync(customerId, billingPeriod, idempotencyKey)
        .get(5, TimeUnit.SECONDS);
    if (!claimed) {
        return billingRepository.findExistingChargeAsync(customerId, billingPeriod);
    }

    // Build producer ONCE with stable key — not rebuilt on retry.
    BasicHttpRequest request = new BasicHttpRequest("POST", "https://api.stripe.com/v1/charges");
    request.setHeader("Authorization", "Bearer " + stripeKey);
    request.setHeader("Idempotency-Key", idempotencyKey);  // stable, pre-computed
    BasicRequestProducer producer = new BasicRequestProducer(request,
        buildEntityProducer(customerId, amountCents));

    CompletableFuture<ChargeResponse> future = new CompletableFuture<>();

    submitRetryable(producer, future, 0);  // pass producer by reference — same object on retry
    return future;
}

private void submitRetryable(BasicRequestProducer producer, CompletableFuture<ChargeResponse> future,
        int attempt) {

    asyncClient.execute(producer, new BasicResponseConsumer<>(ChargeResponse.class),
        HttpClientContext.create(), new FutureCallback<>() {

        @Override
        public void completed(ChargeResponse result) {
            future.complete(result);
        }

        @Override
        public void failed(Exception ex) {
            if (isTransient(ex) && attempt < 3) {
                // Safe: reuse the SAME producer object — same request with same Idempotency-Key.
                // Does NOT call UUID.randomUUID() — idempotencyKey was captured at outer scope.
                submitRetryable(producer, future, attempt + 1);
            } else {
                future.completeExceptionally(ex);
            }
        }

        @Override
        public void cancelled() { future.cancel(false); }
    });
}

The single BasicRequestProducer object is constructed once with idempotencyKey computed by stableKey(customerId, billingPeriod). The failed() handler passes the same producer reference to submitRetryable() on retry. HC5’s async executor calls produce() on the same producer object. The Idempotency-Key header was set in the BasicHttpRequest at construction time and is part of the request that produce() serializes — it does not re-evaluate UUID.randomUUID() on each produce() invocation. Stripe receives the same key on the initial attempt and on the retry and returns the cached ch_A result. For the H2 GOAWAY-triggered transparent retry, the same producer object is reused by HC5 internally, so the key remains stable on the transparent retry as well.

Failure mode 3: per-JVM ScheduledExecutorService billing job runs on all Kubernetes replicas independently — TOCTOU race on hasCompletedForPeriod() — three pods generate distinct UUID.randomUUID() per customer — ch_A, ch_B, and ch_C per customer per billing period

Apache HttpClient 5 is commonly used in server-side Java applications deployed as Kubernetes Deployments with multiple replicas. HC5’s PoolingHttpClientConnectionManager maintains a per-JVM connection pool; there is one pool per pod, with no cross-pod state. Billing jobs in such applications are frequently scheduled via a plain ScheduledExecutorService — Executors.newSingleThreadScheduledExecutor() or Spring’s ThreadPoolTaskScheduler — created at application startup. Each JVM creates its own independent executor and its own independent connection pool. No cross-pod coordination exists between them.

When the billing job fires on the executor, every replica runs it at the same scheduled time. If the job includes a guard like hasCompletedForPeriod(billingPeriod) that queries a shared database but does not atomically claim the billing slot, all three pods can pass the guard concurrently before any pod writes the billing-started record. All three proceed to list the 500 active customers and call UUID.randomUUID() per customer independently, producing three distinct charges per customer:

// MonthlyBillingJob.java
// UNSAFE: scheduler started at application startup on EVERY Kubernetes pod.
// PoolingHttpClientConnectionManager is per-JVM — no cross-pod connection or lock sharing.
// hasCompletedForPeriod() is a non-atomic SELECT — all three pods pass before any commits.

import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.UUID;

public class MonthlyBillingJob {

    private final CloseableHttpClient httpClient;
    private final BillingRepository billingRepository;
    private final ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();

    public void start() {
        // UNSAFE: called at startup on EVERY Kubernetes pod — three independent schedulers.
        // computeInitialDelay() computes to 0 on all three pods at the same clock second.
        scheduler.scheduleAtFixedRate(
            () -> runMonthlyBilling(currentBillingPeriod()),
            computeInitialDelay(),
            30L * 24 * 60 * 60,
            TimeUnit.SECONDS
        );
    }

    private void runMonthlyBilling(String billingPeriod) {
        // UNSAFE: hasCompletedForPeriod() is a plain SELECT — no exclusive lock.
        // September billing hasn't started: all three pods return false simultaneously.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) return;

        // All three pods reach this point simultaneously (TOCTOU race).
        List<Customer> customers = billingRepository.listActive();
        for (Customer customer : customers) {
            // UNSAFE: UUID.randomUUID() per customer per pod — three distinct values.
            // Pod A: cust_123 → UUID_podA → Stripe creates ch_A ($99)
            // Pod B: cust_123 → UUID_podB → Stripe creates ch_B ($99)
            // Pod C: cust_123 → UUID_podC → Stripe creates ch_C ($99)
            String idempotencyKey = UUID.randomUUID().toString();

            HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
            post.setHeader("Authorization", "Bearer " + stripeKey);
            post.setHeader("Idempotency-Key", idempotencyKey);
            post.setEntity(buildFormEntity(customer.getId(), customer.getMonthlyAmountCents()));

            try (CloseableHttpResponse response = httpClient.execute(post)) {
                billingRepository.recordCharge(customer.getId(), billingPeriod,
                    parseChargeId(response));
            } catch (IOException e) {
                log.error("Charge failed for {}: {}", customer.getId(), e.getMessage());
            }
        }

        billingRepository.markCompleted(billingPeriod);
    }
}

The failure scenario: the monthly billing job fires at midnight UTC on October 1. All three Kubernetes pods call runMonthlyBilling("2026-10") within milliseconds. All three query hasCompletedForPeriod("2026-10"). The row does not exist. All three return false. All three proceed to list 500 active customers. All three iterate the list. For customer cust_123: pod A generates UUID_podA and creates a POST /v1/charges request; pod B generates UUID_podB and creates a different POST /v1/charges request; pod C generates UUID_podC and creates a third. All three requests reach Stripe within milliseconds of each other. Stripe processes each as an independent charge with a distinct idempotency key. ch_A, ch_B, and ch_C are committed for customer 123 for October 2026. Customer 123 is charged $99 three times. The same triple-charge pattern repeats for all 500 customers: 1,500 charges created for 500 customers. All three pods then call markCompleted("2026-10"); the last write wins. The next hasCompletedForPeriod check returns true. The 1,500 charges are already in Stripe’s ledger.

The subtler variant: RequestConfig.setConnectionRequestTimeout(Duration.ofSeconds(3)) — connection pool exhaustion under peak billing load causes ConnectionRequestTimeoutException — retry wrapper calls chargeCustomer() again — UUID.randomUUID() at method entry creates ch_B from a single pod

HC5’s PoolingHttpClientConnectionManager limits concurrent connections per route via setMaxConnPerRoute() (default 5 in HC5) and total connections via setMaxConnTotal() (default 25). When a billing job issues 500 concurrent charge requests and the connection pool is configured for only 20 total connections, most threads block waiting to lease a connection. If RequestConfig.setConnectionRequestTimeout(Duration.ofSeconds(3)) is set, threads waiting longer than 3 seconds throw ConnectionRequestTimeoutException. Teams that treat ConnectionRequestTimeoutException as a transient error and retry by re-calling chargeCustomer(customerId, billingPeriod, amountCents) trigger this failure:

// BillingJob.java — UNSAFE: retry wrapper calls chargeCustomer() again on pool timeout.
// chargeCustomer() computes UUID.randomUUID() at method entry — re-evaluates per call.
// The original request may have been sent to Stripe before the timeout (ch_A committed),
// but the retry's fresh UUID creates ch_B.

public void chargeWithRetry(Customer customer, String billingPeriod) {
    for (int attempt = 0; attempt < 3; attempt++) {
        try {
            // UNSAFE: chargeCustomer() calls UUID.randomUUID() at its first line.
            // Each attempt call evaluates UUID.randomUUID() independently.
            // If attempt 0 sent the request and Stripe committed ch_A before the
            // ConnectionRequestTimeoutException was thrown, attempt 1 creates ch_B.
            chargeCustomer(customer.getId(), billingPeriod, customer.getMonthlyAmountCents());
            return;
        } catch (ConnectionRequestTimeoutException e) {
            // Pool exhausted — treat as transient. But the original request may already be committed.
            if (attempt == 2) throw new RuntimeException("Billing failed after 3 attempts", e);
            sleepFor(Duration.ofSeconds(2));
        }
    }
}

private void chargeCustomer(String customerId, String billingPeriod, long amountCents)
        throws IOException {
    // UNSAFE: UUID.randomUUID() at method entry — re-evaluates on every chargeCustomer() call.
    String idempotencyKey = UUID.randomUUID().toString();

    HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
    post.setHeader("Idempotency-Key", idempotencyKey);
    // ... build and execute request
}

ConnectionRequestTimeoutException is thrown by HC5’s connection manager before the request is dispatched to the network in the common case — the thread could not obtain a connection from the pool within the timeout. In that common case no request reached Stripe and no charge was committed. But the failure is not guaranteed to be pre-network. If a connection was leased and the request was sent, and then the pool timeout fired on a different request for the same customer (from a parallel thread), and the retry loop retries using the same method call pattern, ch_B can be created. The safe pattern is the same regardless of which exception triggers the retry: compute stableKey() before the first chargeCustomer() call and pass it as a parameter so every retry invocation uses the same value.

The fix for failure mode 3

Cross-pod coordination requires a lock at a scope that spans all Kubernetes pods. A PostgreSQL advisory lock acquired before the billing loop begins serializes pod execution: only one pod runs the billing loop for a given billing period; all others skip it:

// Safe: pg_try_advisory_lock() as cross-pod distributed mutex.
// Only the pod that acquires the lock runs the billing loop.
// Content-hash key stable across all retries, pods, and HC5 interceptor re-executions.

import org.springframework.jdbc.core.JdbcTemplate;

@Component
public class MonthlyBillingJob {

    private final JdbcTemplate jdbc;
    private final CloseableHttpClient httpClient;
    private final BillingRepository billingRepository;

    // Stable lock key derived from billing period — identical on every pod.
    private long advisoryLockKey(String billingPeriod) {
        return Math.abs(("httpclient5-monthly-billing:" + billingPeriod).hashCode());
    }

    public void runMonthlyBilling() {
        String billingPeriod = currentBillingPeriod();
        long lockKey = advisoryLockKey(billingPeriod);

        // Non-blocking: returns false immediately if another pod holds the lock.
        Boolean acquired = jdbc.queryForObject(
            "SELECT pg_try_advisory_lock(?)", Boolean.class, lockKey);

        if (!Boolean.TRUE.equals(acquired)) {
            // Another pod acquired the lock — skip this pod's billing run entirely.
            log.info("Billing lock for {} held by another pod — skipping", billingPeriod);
            return;
        }

        try {
            runBillingUnderLock(billingPeriod);
        } finally {
            // Release the session-level advisory lock when done.
            jdbc.execute("SELECT pg_advisory_unlock(" + lockKey + ")");
        }
    }

    private void runBillingUnderLock(String billingPeriod) {
        // Double-check after acquiring the lock: another pod may have completed
        // billing between the lock attempt and the lock grant.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) return;

        List<Customer> customers = billingRepository.listActive();
        for (Customer customer : customers) {
            // Safe: content-hash key — same on every pod, every retry, every interceptor re-execution.
            String idempotencyKey = BillingService.stableKey(customer.getId(), billingPeriod);

            // Per-customer pre-flight INSERT — belt-and-suspenders against advisory lock races.
            int inserted = jdbc.update(
                "INSERT INTO billing_charges (customer_id, billing_period, idempotency_key) " +
                "VALUES (?, ?, ?) ON CONFLICT (customer_id, billing_period) DO NOTHING",
                customer.getId(), billingPeriod, idempotencyKey
            );

            if (inserted == 0) {
                // Another pod already claimed this customer (possible on lock boundary edge case).
                continue;
            }

            // Store stable key in HttpClientContext — interceptor reads from context.
            HttpClientContext context = HttpClientContext.create();
            context.setAttribute("stripe.idempotency_key", idempotencyKey);

            HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
            post.setHeader("Authorization", "Bearer " + stripeKey);
            post.setEntity(buildFormEntity(customer.getId(), customer.getMonthlyAmountCents()));

            try (CloseableHttpResponse response = httpClient.execute(post, context)) {
                billingRepository.recordCharge(customer.getId(), billingPeriod,
                    parseChargeId(response));
            } catch (IOException e) {
                log.error("Charge failed for {}: {}", customer.getId(), e.getMessage());
                // Do not retry here — content-hash key + ON CONFLICT DO NOTHING
                // ensures the next billing job run picks this customer up safely.
            }
        }

        billingRepository.markCompleted(billingPeriod);
    }
}

The advisory lock key "httpclient5-monthly-billing:" + billingPeriod evaluates identically on every pod because billingPeriod is derived from the calendar (“2026-10”), not from hostname, pod UID, UUID, or startup timestamp. Only one pod can hold this lock at a time. All other pods that call pg_try_advisory_lock() while the first pod holds it receive false and return without billing. The per-customer ON CONFLICT DO NOTHING insert provides a durable claim even if two pods race past the advisory lock due to a database connection error, an application crash between lock acquisition and the first insert, or a lock timeout during high load. The UNIQUE (customer_id, billing_period) constraint ensures only one pod’s insert wins per customer. The HttpClientContext passes the stable key to the interceptor so StripeBillingInterceptor.process() reads the same value on the initial attempt and on any RetryExec-triggered retry — all three layers (advisory lock, per-customer pre-flight, content-hash key) work together.

Patterns that reliably cause these failures in Apache HttpClient 5

The three failure modes share the same structural pattern: a value that must be stable across all retry attempts and all pod executions (UUID.randomUUID()) is computed at a scope that is smaller than the retry or coordination boundary. In FM1, the scope is the interceptor process() method, which is inside HC5’s RetryExec retry boundary. In FM2, the scope is the AsyncRequestProducer constructor, which is inside the async retry callback boundary. In FM3, the scope is the per-JVM scheduler invocation, which is inside the pod-level execution boundary with no cross-pod coordination.

Patterns that reliably produce these failure modes in Apache HttpClient 5:

• UUID.randomUUID() or System.currentTimeMillis() inside HttpRequestInterceptor.process() registered via addRequestInterceptorLast() when DefaultHttpRequestRetryStrategy is active (the default when using HttpClients.custom().setRetryStrategy(...))
• UUID.randomUUID() inside an HC5 interceptor used via Spring 6.x’s HttpComponentsClientHttpRequestFactory with Spring @Retryable — the Spring retry layer re-calls restClient.post()…retrieve() which re-runs the underlying HC5 interceptor chain, compounding HC5’s own RetryExec retries
• UUID.randomUUID() computed at AsyncRequestProducer construction time when the producer is rebuilt inside FutureCallback.failed() retry handlers rather than constructed once and reused
• AsyncRequestProducer factory lambda called per retry attempt with UUID.randomUUID() inside the factory, triggered by HC5’s H2 GOAWAY-handling transparent retry in HTTP/2-negotiated connections
• Billing logic in a ScheduledExecutorService (from Executors.newSingleThreadScheduledExecutor(), Spring @Scheduled, or ThreadPoolTaskScheduler) without a distributed lock across Kubernetes replicas — HC5’s PoolingHttpClientConnectionManager is per-JVM and provides no cross-pod coordination
• Retry-on-timeout wrapper that calls chargeCustomer(customerId, billingPeriod, amount) again on ConnectionRequestTimeoutException when UUID.randomUUID() is evaluated at chargeCustomer() method entry

Patterns that are safe in Apache HttpClient 5:

• stableKey(customerId, billingPeriod) computed before httpClient.execute(request, context); stored as context.setAttribute("stripe.idempotency_key", idempotencyKey) in HttpClientContext; interceptor reads from context and never calls UUID.randomUUID()
• AsyncRequestProducer built once with stable key and reused — not rebuilt in FutureCallback.failed(); submitRetryable(producer, future, attempt + 1) passes the same producer reference on retry
• Spring RestClient with HttpComponentsClientHttpRequestFactory and @Retryable where the stable key is computed before the @Retryable method entry and passed as a parameter, and the interceptor reads from HttpClientContext rather than calling UUID.randomUUID()
• pg_try_advisory_lock(Math.abs(("httpclient5-monthly-billing:" + billingPeriod).hashCode())) acquired before the billing loop with lock key derived from billing period string — not from hostname, pod ID, UUID, or timestamp
• Per-customer INSERT INTO billing_charges ... ON CONFLICT (customer_id, billing_period) DO NOTHING as a belt-and-suspenders durable claim before each Stripe call

Summary

The pattern connecting all three: the idempotency key must be derived from the business intent of the billing operation — customer ID, billing period, vendor namespace — not from any ephemeral runtime value that re-evaluates between interceptor chain executions, AsyncRequestProducer rebuilds, or concurrent pod invocations. UUID.randomUUID(), System.currentTimeMillis() at producer-construction time, a per-process()-call UUID, and any per-pod or per-call value all produce different results each time the expression is evaluated. A key derived from sha256(customerId + ":" + billingPeriod + ":httpclient5-billing")[:32] produces the same 32-character hex string on every HC5 RetryExec interceptor re-invocation, on every FutureCallback retry, and on every Kubernetes pod — stable across all three failure modes. Backing it with a PostgreSQL-level UNIQUE (customer_id, billing_period) constraint and a pg_try_advisory_lock() distributed mutex moves the deduplication guarantee out of ephemeral per-JVM state into durable shared storage that survives NoHttpResponseExceptions, GOAWAY frames, connection pool exhaustion, and multi-pod concurrent billing loops.

The vault key spend cap adds a hard financial boundary as a last line of defence: a per-billing-period vault key issued with a max_amount set to expected_total × 1.10 caps the maximum spend that any combination of HC5 interceptor retry re-invocations, async producer rebuilds, and multi-pod billing races can produce. Once the cap is reached, Stripe rejects further charges with a 402 Payment Required on the vault key, containing the blast radius to a known maximum regardless of how many RetryExec retries, FutureCallback retries, or concurrent billing pods are in flight.