Apache HttpComponents Core 5 Async HTTP Client and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

The Apache HttpClient 5 sync post covered three failure modes rooted in the synchronous execution chain: HttpRequestInterceptor.process() re-running on every retry attempt, PoolingHttpClientConnectionManager scheduler races across Kubernetes replicas, and the DefaultHttpRequestRetryStrategy re-sending with a fresh UUID.randomUUID(). The async client — CloseableHttpAsyncClient from HttpAsyncClients.createDefault() or H2AsyncClientBuilder.create().build() — introduces three structurally different Stripe billing failure modes. First: layered retry in FutureCallback.failed() — HC5’s internal async retry re-sends the same request object (UUID_A preserved) on its built-in attempts, then fires failed() on final exhaustion; a developer who adds external retry in failed() builds a new SimpleHttpRequest with UUID.randomUUID() and sends UUID_B — Stripe creates ch_B for a customer whose ch_A was committed before the connection failure. Second: concurrent NIO billing loop where all N failed() callbacks fire simultaneously — when Stripe returns HTTP 500 to all in-flight requests in a parallel batch, all N callbacks execute concurrently on HC5’s NIO reactor threads; each independently generates UUID.randomUUID(), producing N distinct UUID_B values; customers with committed ch_A receive ch_B. Third: HTTP/2 GoAway frame triggering mass simultaneous failures — HC5’s H2 multiplexer closes all in-flight streams when a GOAWAY frame arrives, firing failed() for every pending request; the developer retries all with fresh UUIDs; Stripe silently returns cached ch_A for already-committed customers (correct behavior) but returns ch_B for new UUID_B requests when idempotency is absent. This post covers all three with HC5 5.2.x Java code, content-hash stable keys, pre-flight database guards, and vault key spend caps.

How the HC5 async client’s execution model differs from the sync client

The HC5 sync client blocks the calling thread from the moment execute() is called until the response arrives or an exception is thrown. Retry is implemented either by wrapping the call in a loop on the calling thread or by the internal RetryExec exec chain element, which calls the next executor in the chain again before returning to the calling thread. The calling thread never leaves the synchronous execution scope, so local variables — including a pre-computed idempotency key — remain in scope across all retry attempts.

The HC5 async client works differently. CloseableHttpAsyncClient.execute(request, callback) returns immediately after submitting the request to the NIO I/O reactor. The calling thread is free. When the response arrives (or the connection fails), HC5’s internal I/O dispatch thread calls callback.completed(response) or callback.failed(ex). These callbacks run on the I/O dispatch thread, not on the thread that called execute(). The original calling-thread stack frame — including any local variables like a pre-computed UUID — is gone. The callback has no access to the original thread’s local state unless the developer explicitly captures it in a closure or passes it as a field on the callback instance.

This separation of execution contexts is what causes the three failure modes described in this post. In each case, the developer needs to reason across the original execute() callsite and the failed() callback as a single logical billing operation, but the async model makes that non-obvious because the two run on different threads at different times.

Failure mode 1: FutureCallback<SimpleHttpResponse>.failed() builds new SimpleHttpRequest with UUID.randomUUID() — HC5’s internal retry preserves UUID_A, external retry in failed() sends UUID_B — ch_B when ch_A committed before final retry exhaustion

HC5’s async execution chain includes AsyncRetryExec, which implements StandardHttpRequestRetryStrategy by default. When the async client encounters a retriable I/O error (connection reset, NoHttpResponseException, socket timeout during response read), AsyncRetryExec re-submits the original request object to the connection pool. Because the same SimpleHttpRequest object is re-submitted, the Idempotency-Key header set on that object carries UUID_A on every internal retry attempt. This is the correct behavior — no new UUID is generated by HC5’s internal retry layer.

The failure mode arises when the developer adds a second retry layer in FutureCallback.failed(). After AsyncRetryExec exhausts all internal retry attempts, it propagates the final exception to the callback’s failed() method. A developer who sees failed() as the “all retries exhausted, try from scratch” signal naturally rebuilds the request and fires another execute() from inside failed():

// BillingService.java — UNSAFE: external retry in failed() builds new request with UUID.randomUUID().
// HC5's internal AsyncRetryExec correctly preserves UUID_A on its built-in retries.
// When failed() fires after exhaustion, UUID.randomUUID() generates UUID_B.
import org.apache.hc.client5.http.async.methods.SimpleHttpRequest;
import org.apache.hc.client5.http.async.methods.SimpleHttpResponse;
import org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient;
import org.apache.hc.client5.http.impl.async.HttpAsyncClients;
import org.apache.hc.core5.concurrent.FutureCallback;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicInteger;

public class BillingService {
    private final CloseableHttpAsyncClient asyncClient =
        HttpAsyncClients.createDefault();

    public void chargeCustomer(String customerId, int amountCents) {
        // BUG: UUID.randomUUID() evaluated here, at the outer callsite.
        // This is UUID_A for the initial attempt.
        // But this method is also called from failed() on retry —
        // re-entering chargeCustomer() evaluates UUID.randomUUID() again — UUID_B.
        String idempotencyKey = UUID.randomUUID().toString();

        SimpleHttpRequest request = SimpleHttpRequest.create(
            org.apache.hc.core5.http.Method.POST,
            "https://api.stripe.com/v1/charges"
        );
        request.setHeader("Authorization", "Bearer rk_live_xxx");
        request.setHeader("Idempotency-Key", idempotencyKey); // UUID_A on first call, UUID_B on retry
        request.setHeader("Content-Type", "application/x-www-form-urlencoded");
        request.setBodyText(
            "amount=" + amountCents + "¤cy=usd&customer=" + customerId,
            org.apache.hc.core5.http.ContentType.APPLICATION_FORM_URLENCODED
        );

        asyncClient.execute(request, new FutureCallback<SimpleHttpResponse>() {
            @Override public void completed(SimpleHttpResponse response) {
                System.out.println("Charged: " + response.getBody());
            }

            @Override public void failed(Exception ex) {
                // HC5's internal AsyncRetryExec has exhausted its retries.
                // Developer reads this as "permanent failure, let's try from scratch."
                // BUG: chargeCustomer() is called again — re-evaluates UUID.randomUUID()
                // at method entry — UUID_B sent to Stripe.
                // If ch_A was committed before the final retry failure
                // (e.g., Stripe committed on attempt 1 but the response RST arrived
                // after HC5's internal retries also failed to get a response),
                // this retry creates ch_B alongside ch_A.
                System.err.println("All internal retries failed: " + ex.getMessage());
                chargeCustomer(customerId, amountCents); // ← generates UUID_B
            }

            @Override public void cancelled() { /* no-op */ }
        });
    }
}

The timing that makes this dangerous: HC5’s AsyncRetryExec retries on connection-level errors, not on successful HTTP responses with error status codes. A NoHttpResponseException means the server closed the connection without sending any response bytes. In that case, the charge was almost certainly not committed. But a java.net.SocketException: Connection reset received after Stripe has finished processing and begun sending the response is ambiguous — the charge may have been committed before the reset occurred. HC5’s StandardHttpRequestRetryStrategy retries on SocketException by default, so after two internal retries (all with UUID_A, all failing with Connection reset), failed() fires. At that point the developer does not know whether the first attempt’s charge was committed. The external retry with UUID_B bets “it wasn’t” — a bet that Stripe’s billing semantics does not honor without an idempotency key match.

Subtler variant: retry counter in AtomicInteger closed over by callback — counter reset between outer and inner retry layers — UUID_B generated on inner retry layer’s first attempt

A developer who is aware of the layered retry issue sometimes tries to unify the retry logic into a single AtomicInteger counter shared across both HC5’s internal retries and the failed() handler. The intent is to limit total attempts across both layers to, say, three. The bug is that HC5’s internal retry layer does not decrement or read the developer’s counter — it runs independently based on StandardHttpRequestRetryStrategy.retryRequest(). So the developer’s counter counts only the failed() invocations, not the internal retry attempts:

// BillingRetrier.java — UNSAFE: AtomicInteger counts failed() invocations,
// not HC5's internal retry attempts. HC5 retries 3x with UUID_A internally.
// failed() is then called once; counter = 0, so retry fires with UUID.randomUUID() — UUID_B.
public void chargeWithUnifiedRetry(String customerId, int amountCents) {
    AtomicInteger externalAttempts = new AtomicInteger(0);
    // Pre-compute UUID here — but this method is re-called from failed() —
    // re-entry re-evaluates UUID.randomUUID() on the next chargeWithUnifiedRetry() call.
    // The AtomicInteger does NOT prevent re-entry from generating UUID_B.
    String idempotencyKey = UUID.randomUUID().toString();

    submitRequest(customerId, amountCents, idempotencyKey, externalAttempts);
}

private void submitRequest(String customerId, int amountCents, String idempotencyKey,
                           AtomicInteger externalAttempts) {
    SimpleHttpRequest request = buildRequest(customerId, amountCents, idempotencyKey);
    asyncClient.execute(request, new FutureCallback<SimpleHttpResponse>() {
        @Override public void completed(SimpleHttpResponse r) { /* success */ }
        @Override public void failed(Exception ex) {
            if (externalAttempts.getAndIncrement() < 2) {
                // BUG: submits a NEW call to chargeWithUnifiedRetry(),
                // which re-evaluates UUID.randomUUID() at entry — UUID_B.
                // The intent was to pass the SAME idempotencyKey, but the method
                // generates a fresh one each time it's called.
                chargeWithUnifiedRetry(customerId, amountCents);
            }
        }
        @Override public void cancelled() { /* no-op */ }
    });
}

The developer intended the AtomicInteger to be a unified counter across all retry layers. But because the external retry re-calls chargeWithUnifiedRetry() instead of submitRequest() with the original idempotencyKey, a new key is generated on every chargeWithUnifiedRetry() invocation.

Fix: compute the stable content-hash key once, capture it in the callback closure, re-use it on retry without re-calling the billing method

// BillingService.java — FIXED.
import java.security.MessageDigest;
import java.nio.charset.StandardCharsets;

private static String stableIdempotencyKey(String customerId, String billingPeriod) {
    String input = customerId + ":" + billingPeriod + ":hc5-async-billing";
    byte[] digest = MessageDigest.getInstance("SHA-256")
        .digest(input.getBytes(StandardCharsets.UTF_8));
    StringBuilder hex = new StringBuilder(32);
    for (int i = 0; i < 16; i++) hex.append(String.format("%02x", digest[i]));
    return hex.toString();
}

public void chargeCustomer(String customerId, String billingPeriod, int amountCents) {
    // Stable key computed once, before the first execute() call.
    // stableIdempotencyKey() is a pure function: same inputs → same output
    // on any thread, any pod, any time. Re-computing it in failed() gives UUID_A.
    final String idempotencyKey = stableIdempotencyKey(customerId, billingPeriod);

    submitAttempt(customerId, billingPeriod, amountCents, idempotencyKey, 0);
}

private void submitAttempt(String customerId, String billingPeriod, int amountCents,
                            String idempotencyKey, int externalAttempt) {
    // Build request with the SAME idempotencyKey value on every call to submitAttempt().
    // UUID_A on attempt 0, attempt 1, attempt 2 — always.
    SimpleHttpRequest request = SimpleHttpRequest.create(
        org.apache.hc.core5.http.Method.POST,
        "https://api.stripe.com/v1/charges"
    );
    request.setHeader("Authorization", "Bearer rk_live_xxx");
    request.setHeader("Idempotency-Key", idempotencyKey); // always UUID_A
    request.setHeader("Content-Type", "application/x-www-form-urlencoded");
    request.setBodyText(
        "amount=" + amountCents + "¤cy=usd&customer=" + customerId,
        org.apache.hc.core5.http.ContentType.APPLICATION_FORM_URLENCODED
    );

    asyncClient.execute(request, new FutureCallback<SimpleHttpResponse>() {
        @Override public void completed(SimpleHttpResponse response) {
            System.out.println("Charged " + customerId + ": " + response.getBody());
        }

        @Override public void failed(Exception ex) {
            if (externalAttempt < 2) {
                long delayMs = 500L * (externalAttempt + 1);
                scheduler.schedule(
                    () -> submitAttempt(customerId, billingPeriod, amountCents,
                                        idempotencyKey,  // same UUID_A, not regenerated
                                        externalAttempt + 1),
                    delayMs, TimeUnit.MILLISECONDS
                );
            } else {
                System.err.println("Billing failed after retries for " + customerId);
            }
        }

        @Override public void cancelled() { /* no-op */ }
    });
}

Three properties of this fix matter. First, stableIdempotencyKey() is a deterministic pure function — two concurrent billing coroutines for cus_123 in 2026-09 compute the same 32-character hex string; Stripe serializes the two requests via its idempotency cache and returns the cached ch_A result for the second. Second, submitAttempt() is called directly from failed() with the pre-computed idempotencyKey value, not via chargeCustomer() (which would re-evaluate UUID). Third, the retry scheduling uses ScheduledExecutorService.schedule() rather than recursion, which avoids stack overflow on pathological retry sequences and gives the idempotencyKey closure explicit transfer to the next retry lambda.

Failure mode 2: NIO batch billing loop — asyncClient.execute() for each customer in parallel — server-side error fires all N FutureCallback.failed() callbacks concurrently on reactor threads — N independent UUID.randomUUID() retries — ch_B for customers whose ch_A committed before the error

The primary use case for an async HTTP client is concurrent request dispatch — sending multiple requests in parallel without blocking. A billing service that uses HC5 async typically submits one execute() per customer in a loop, relying on the NIO reactor to multiplex all connections efficiently. This is exactly the right approach for throughput. The failure mode is not in the concurrency itself but in how failed() callbacks interact when a server-side error causes all of them to fire simultaneously.

Consider a monthly billing run that charges 200 customers. The billing service starts the async client, submits 200 execute() calls, then blocks a wait thread on a CountDownLatch or CompletableFuture.allOf() for all 200 to complete. Stripe’s API rate limit is normally not a concern at 200 requests — but a momentary HTTP 529 (“Too Many Requests” from Stripe) or a transient HTTP 500 — which Stripe sends during brief infrastructure events — can cause all 200 responses to arrive as failures within the same reactor poll cycle. All 200 failed() callbacks fire concurrently on HC5’s I/O reactor threads:

// BatchBillingService.java — UNSAFE: UUID.randomUUID() inside failed() closure.
// When all 200 failed() callbacks fire concurrently, all 200 generate independent
// UUID_B_1 through UUID_B_200. Customers whose ch_A committed before the 500
// will receive ch_B from their respective retry.
import java.util.List;
import java.util.concurrent.CountDownLatch;

public void runMonthlyBillingBatch(List<BillingCustomer> customers, String billingPeriod) {
    CountDownLatch latch = new CountDownLatch(customers.size());

    for (BillingCustomer customer : customers) {
        // BUG: UUID.randomUUID() evaluated inside the failed() lambda.
        // Each lambda captures its own (customer, billingPeriod) closure
        // but generates a FRESH UUID on every failed() invocation.
        SimpleHttpRequest request = buildRequest(customer.id(), billingPeriod,
            UUID.randomUUID().toString()); // UUID_A for the initial request

        asyncClient.execute(request, new FutureCallback<SimpleHttpResponse>() {
            @Override public void completed(SimpleHttpResponse r) {
                latch.countDown();
            }

            @Override public void failed(Exception ex) {
                // This fires on the I/O reactor thread.
                // With 200 concurrent requests all failing simultaneously,
                // 200 threads enter this block concurrently.
                // Each independently evaluates UUID.randomUUID() — UUID_B_1..UUID_B_200.
                String retryKey = UUID.randomUUID().toString(); // UUID_B for this customer
                SimpleHttpRequest retryRequest = buildRequest(
                    customer.id(), billingPeriod, retryKey);
                asyncClient.execute(retryRequest, new FutureCallback<SimpleHttpResponse>() {
                    @Override public void completed(SimpleHttpResponse r) { latch.countDown(); }
                    @Override public void failed(Exception e) { latch.countDown(); }
                    @Override public void cancelled() { latch.countDown(); }
                });
            }

            @Override public void cancelled() { latch.countDown(); }
        });
    }

    latch.await(); // wait for all 200 (+ retries) to finish
}

The danger is proportional to the fraction of requests that were committed before the server-side error. If Stripe’s infrastructure processed, say, 40 of the 200 requests before a brief 500-burst hit the remaining 160 (and the 40 responses were in flight when the burst caused connection errors), those 40 customers have ch_A committed at Stripe. The 160 that received the 500 synchronously before any commit are safe to retry with any UUID. The 40 that were committed and then had their response connections disrupted will receive ch_B from the retry — 40 double charges from a single transient server event.

Subtler variant: ScheduledExecutorService retry with UUID generated inside the scheduled lambda — lambda executes on scheduler thread, not at scheduling time — UUID_B per customer at lambda execution

Some developers add exponential backoff by scheduling the retry instead of executing it immediately from failed(). If the UUID is generated inside the scheduled lambda, it is evaluated at execution time — not at scheduling time. This means the UUID is still generated fresh per retry, just on a different thread and at a later time:

// BatchBillingService.java — UNSAFE: UUID.randomUUID() inside scheduled lambda.
// Lambda executes on scheduler thread at delay expiry, not on the reactor thread.
// UUID is generated at execution time — UUID_B on retry regardless of delay.
@Override public void failed(Exception ex) {
    scheduler.schedule(() -> {
        // BUG: UUID.randomUUID() evaluated when this lambda runs (at delay expiry),
        // not when it is passed to scheduler.schedule(). UUID_B per customer.
        String retryKey = UUID.randomUUID().toString();
        SimpleHttpRequest retryRequest = buildRequest(customer.id(), billingPeriod, retryKey);
        asyncClient.execute(retryRequest, secondAttemptCallback);
    }, 500, TimeUnit.MILLISECONDS);
}

The developer may assume that writing UUID.randomUUID().toString() inside a lambda is equivalent to evaluating it at the lambda creation site. In Java, lambda bodies are evaluated lazily — the lambda’s body executes when the lambda is invoked, not when the lambda object is created. UUID.randomUUID() inside the lambda body is a new invocation per lambda execution, not a value captured at creation time. Moving UUID.randomUUID() before the scheduler.schedule() call and capturing it as a final String retryKey in the outer scope is not sufficient either if that outer scope is inside the failed() method, because failed() itself generates a new UUID per invocation. The stable content-hash key must be computed before the first execute() call, outside the callback, and closed over by all lambdas.

Fix: compute stable key per customer before the batch loop — close over it in both the initial callback and the retry callback

// BatchBillingService.java — FIXED.
public void runMonthlyBillingBatch(List<BillingCustomer> customers, String billingPeriod) {
    CountDownLatch latch = new CountDownLatch(customers.size());

    for (BillingCustomer customer : customers) {
        // Key computed once per customer, before the first execute().
        // All callbacks for this customer close over the same key value.
        final String idempotencyKey = stableIdempotencyKey(customer.id(), billingPeriod);

        SimpleHttpRequest request = buildRequest(customer.id(), billingPeriod, idempotencyKey);

        asyncClient.execute(request, new FutureCallback<SimpleHttpResponse>() {
            @Override public void completed(SimpleHttpResponse r) {
                latch.countDown();
            }

            @Override public void failed(Exception ex) {
                // Retry with the SAME idempotencyKey — UUID_A for this customer.
                // Whether 1 or 200 callbacks fire concurrently, all pass UUID_A to Stripe.
                // Stripe returns cached ch_A for already-charged customers,
                // and creates ch_A (first charge) for customers not yet charged.
                scheduler.schedule(() -> {
                    // idempotencyKey is closed over from the outer for-loop scope —
                    // the scheduler lambda captures the same reference as the failed() callback.
                    SimpleHttpRequest retryRequest = buildRequest(
                        customer.id(), billingPeriod, idempotencyKey); // UUID_A, not UUID_B
                    asyncClient.execute(retryRequest, new FutureCallback<SimpleHttpResponse>() {
                        @Override public void completed(SimpleHttpResponse r) { latch.countDown(); }
                        @Override public void failed(Exception e) { latch.countDown(); }
                        @Override public void cancelled() { latch.countDown(); }
                    });
                }, 500, TimeUnit.MILLISECONDS);
            }

            @Override public void cancelled() { latch.countDown(); }
        });
    }

    try { latch.await(); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
}

Two properties make this fix robust under concurrency. First, each idempotencyKey is computed in the for-loop body — one key per customer, computed before the first execute(). All callbacks for that customer (the initial callback and any retry callbacks) close over the same key value from the enclosing for-loop iteration. Second, because stableIdempotencyKey() is deterministic, if the entire batch billing run crashes mid-way (JVM OOM, pod eviction) and is restarted, the same keys are re-computed from the same customer IDs and billing period. Stripe’s idempotency cache returns cached results for all customers already charged, and creates new charges for those that weren’t. The pre-flight INSERT INTO billing_runs (customer_id, billing_period, idempotency_key) ... ON CONFLICT (customer_id, billing_period) DO NOTHING check provides the authoritative cluster-wide mutex for the case where two instances of the billing service run concurrently (for example, during a Kubernetes rolling deployment).

Failure mode 3: HC5 HTTP/2 GOAWAY frame — all in-flight streams fail simultaneously — FutureCallback.failed() for N pending billing requests — developer retries all with UUID.randomUUID() — ch_B for customers whose streams were processed by Stripe before GOAWAY

Apache HttpComponents Core 5 supports HTTP/2 via H2AsyncClientBuilder.create().build() or by setting HttpVersionPolicy.FORCE_HTTP_2 on the standard HttpAsyncClientBuilder. HTTP/2 multiplexes multiple requests over a single TCP connection using independent streams. Stripe’s API supports HTTP/2. This combination — HC5 async client with HTTP/2 against Stripe — introduces a failure mode specific to HTTP/2’s connection-lifecycle semantics: the GOAWAY frame.

A GOAWAY frame is sent by the server to initiate graceful connection shutdown. Stripe’s infrastructure sends GOAWAY frames during load balancer rotations, API gateway deployments, and graceful restarts of backend servers. The GOAWAY frame includes a Last-Stream-Id field: all streams with IDs ≤ Last-Stream-Id were processed by the server; all streams with IDs > Last-Stream-Id were not received and are safe to retry on a new connection.

HC5’s H2 multiplexer handles GOAWAY by closing the affected connection and completing all pending stream futures. The completion is a failure — FutureCallback.failed() is called for every in-flight request on the connection. HC5 does NOT automatically distinguish between streams ≤ Last-Stream-Id (processed by Stripe, potentially with committed charges) and streams > Last-Stream-Id (not received by Stripe, safe to retry). The developer receives a batch of failed() callbacks without information about which streams were and were not processed by Stripe:

// H2BillingService.java — UNSAFE: UUID.randomUUID() in failed() for all streams.
// When GOAWAY arrives, all in-flight billing request streams fire failed().
// Streams with IDs ≤ Last-Stream-Id: Stripe processed them; ch_A may be committed.
// Streams with IDs > Last-Stream-Id: Stripe never received them; safe to retry.
// Developer can't distinguish the two from failed() alone.
// UUID.randomUUID() retries all — ch_B for the ≤ Last-Stream-Id group.
import org.apache.hc.client5.http.impl.async.H2AsyncClientBuilder;
import org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient;

public class H2BillingService {
    private final CloseableHttpAsyncClient h2Client =
        H2AsyncClientBuilder.create().build();

    public void chargeCustomer(BillingCustomer customer, String billingPeriod) {
        // UUID_A on first call.
        String idempotencyKey = UUID.randomUUID().toString(); // UUID_A

        SimpleHttpRequest request = buildRequest(
            customer.id(), billingPeriod, idempotencyKey);

        h2Client.execute(request, new FutureCallback<SimpleHttpResponse>() {
            @Override public void completed(SimpleHttpResponse r) { /* success */ }

            @Override public void failed(Exception ex) {
                // GOAWAY triggers this for ALL in-flight streams on the connection.
                // The exception type is typically ConnectionClosedException
                // or H2StreamResetException. No Last-Stream-Id exposed here.
                // BUG: generate new UUID for retry — UUID_B.
                // For streams that Stripe processed (committed ch_A), this creates ch_B.
                // For streams Stripe never saw, this is the first and only charge —
                // UUID_B is fine, no duplication. But the developer doesn't know which is which.
                String retryKey = UUID.randomUUID().toString(); // UUID_B for all customers
                SimpleHttpRequest retryRequest = buildRequest(
                    customer.id(), billingPeriod, retryKey);
                h2Client.execute(retryRequest, noopCallback);
            }

            @Override public void cancelled() { /* no-op */ }
        });
    }
}

The specific failure scenario: 50 concurrent billing requests are in-flight on a single HTTP/2 connection. Stripe’s load balancer initiates a graceful restart and sends GOAWAY with Last-Stream-Id=42. Streams 1–42 (21 billing requests, since each billing POST uses one stream) were received and processed by Stripe. Some of those 21 may have had charges committed before the GOAWAY was sent. Streams 43–100 (29 billing requests) were not received by Stripe. All 50 callbacks fire failed(). The developer retries all 50 with fresh UUIDs. The 29 unprocessed customers receive their charge correctly (ch_A under UUID_B, since there was no ch_A from a prior UUID). The 21 that were processed may have ch_A committed; they receive ch_B from the retry with UUID_B. Up to 21 double charges from a single load balancer rotation.

Why HC5 does not expose Last-Stream-Id in failed()

HC5’s public FutureCallback interface carries only an Exception. For HTTP/2 stream failures, the exception is typically org.apache.hc.core5.http2.H2StreamResetException or org.apache.hc.core5.http.ConnectionClosedException. Neither carries the Last-Stream-Id from the GOAWAY frame. The stream ID of the request that failed is also not exposed in the public FutureCallback API — it is an internal HC5 transport detail. So from within a failed() callback, a developer cannot determine whether their specific request’s stream ID was ≤ or > the GOAWAY’s Last-Stream-Id.

The only safe strategy when GOAWAY could have affected some requests is to retry all of them with idempotency keys that are stable across the original attempt and the retry. If a stable content-hash key is used, the retry for a customer whose charge was committed by Stripe will receive the cached ch_A result (idempotency match). The retry for a customer whose stream was never received will create ch_A as the first charge. No duplicate charges in either case.

Subtler variant: HC5 automatic transparent H2 retry — HC5 migrates “not yet sent” streams to new connection automatically — developer misreads this as “all streams handled” and does not add idempotency key — committed streams retry without a key on the new connection — Stripe creates new charge

HC5’s HTTP/2 implementation performs automatic transparent connection retry for streams that have not been fully written to the wire when a GOAWAY is received. For those streams, HC5 opens a new HTTP/2 connection and re-submits the request without calling failed(). The developer may test their billing service under normal load, observe that streams are automatically handled across connection restarts (the automatic migration fires for streams not yet transmitted), and conclude that HC5 handles GOAWAY transparently. The conclusion is correct for not-yet-transmitted streams but incorrect for streams that were fully written to the wire and processed by Stripe before the GOAWAY:

// H2BillingService.java — UNSAFE: no Idempotency-Key header at all.
// Developer tested under light load where automatic stream migration handled GOAWAY.
// Under higher concurrency, some streams ARE transmitted before GOAWAY arrives.
// HC5 fires failed() for those streams. Developer added retry in failed() but no key
// on the original request — each attempt has no Idempotency-Key —
// Stripe treats each as a new charge — ch_A on first, ch_B on GOAWAY retry.
SimpleHttpRequest request = SimpleHttpRequest.create(Method.POST, stripeUrl);
request.setHeader("Authorization", "Bearer rk_live_xxx");
// BUG: no Idempotency-Key header. Developer tested with automatic stream migration
// and never saw a duplicate because stream migration re-uses the same request object
// (UUID_A preserved if a key existed). Without any key, Stripe has no cache entry
// and creates a new charge on every request it receives.
request.setBodyText(payload, ContentType.APPLICATION_FORM_URLENCODED);

HC5’s automatic stream migration preserves the original SimpleHttpRequest object, so if an Idempotency-Key header exists on the request, it is preserved during migration. The automatic migration is safe for idempotent keys. The bug is that the developer observed automatic migration working (correctly) and decided an idempotency key was unnecessary — then when a stream was processed before GOAWAY and HC5 called failed() instead of migrating it, the developer’s external retry built a new request without an idempotency key, creating a new charge.

Fix: use a stable content-hash idempotency key on every billing request — safe for both automatic stream migration and manual retry in failed()

// H2BillingService.java — FIXED.
public void chargeCustomer(BillingCustomer customer, String billingPeriod) {
    // Stable key computed once before the first execute() call.
    final String idempotencyKey = stableIdempotencyKey(customer.id(), billingPeriod);

    executeWithKey(customer, billingPeriod, idempotencyKey, 0);
}

private void executeWithKey(BillingCustomer customer, String billingPeriod,
                             String idempotencyKey, int attempt) {
    // Same idempotencyKey value on every call to executeWithKey() for this customer+period.
    // HC5 automatic stream migration: re-uses the same request object — UUID_A preserved.
    // HC5 failed() retry: passes the same idempotencyKey to the next executeWithKey() call.
    // Both paths converge on UUID_A at Stripe. No duplicate charge either way.
    SimpleHttpRequest request = SimpleHttpRequest.create(Method.POST,
        "https://api.stripe.com/v1/charges");
    request.setHeader("Authorization", "Bearer rk_live_xxx");
    request.setHeader("Idempotency-Key", idempotencyKey); // UUID_A, always
    request.setBodyText(
        "amount=" + customer.amountCents() + "¤cy=usd&customer=" + customer.id(),
        ContentType.APPLICATION_FORM_URLENCODED
    );

    h2Client.execute(request, new FutureCallback<SimpleHttpResponse>() {
        @Override public void completed(SimpleHttpResponse r) {
            System.out.println("Charged " + customer.id() + " via H2: " + r.getBody());
        }

        @Override public void failed(Exception ex) {
            // GOAWAY, connection reset, or other failure.
            // Re-call executeWithKey with the SAME idempotencyKey — UUID_A.
            // Stripe returns cached ch_A for customers already charged, creates ch_A for others.
            if (attempt < 2) {
                long delayMs = 500L * (attempt + 1);
                scheduler.schedule(
                    () -> executeWithKey(customer, billingPeriod, idempotencyKey, attempt + 1),
                    delayMs, TimeUnit.MILLISECONDS
                );
            } else {
                System.err.println("H2 billing failed after retries: " + customer.id());
            }
        }

        @Override public void cancelled() { /* no-op */ }
    });
}

This fix handles both the automatic-migration path and the manual-retry path uniformly. For the automatic-migration path: HC5 re-submits the original SimpleHttpRequest object with Idempotency-Key: UUID_A on the new connection — Stripe receives UUID_A as the first request for this customer, creates ch_A, and caches it. For the manual-retry path via failed(): executeWithKey() builds a new SimpleHttpRequest object but with the same idempotencyKey string — UUID_A. Stripe receives UUID_A, looks up the idempotency cache, and returns the cached ch_A result if the stream was already processed before the GOAWAY.

Pre-flight database guard and vault key spend cap as backstops

Stable content-hash idempotency keys eliminate duplicate charges caused by the three HC5 async failure modes above. But two additional safeguards are worth adding for production billing systems.

The first is a pre-flight database write before the first execute() call. Before dispatching any billing request, insert a row into a billing_runs table using INSERT INTO billing_runs (customer_id, billing_period, idempotency_key, status) VALUES (?, ?, ?, 'pending') ON CONFLICT (customer_id, billing_period) DO NOTHING. If the insert succeeds (row was new), proceed with the charge. If the insert fails (conflict — a row already exists for this customer and period), the charge was already initiated in a prior run — skip the execute() call and query the existing row’s status. This guard is authoritative across all JVM instances, Kubernetes pods, and replicas — it serializes the billing operation at the database level, independent of idempotency key logic at the Stripe API level. Even if a content-hash key collision occurred (two different (customer_id, billing_period) pairs colliding on the first 16 bytes of SHA-256 is astronomically unlikely, but the guard eliminates the consequence), the pre-flight write prevents a second execute().

The second is a per-billing-period vault key spend cap. Keybrake issues a vault_key_xxx scoped to a single billing period, with a USD cap set to expected_total × 1.10 (10% headroom for legitimate rounding and currency conversion). All HC5 async billing requests for that period pass through the Keybrake proxy using the vault key rather than the live Stripe key directly. The proxy enforces the cap and fires an alert when 80% of the cap is consumed. If a code bug causes more charges than expected — even with correct idempotency keys, a mis-scoped customer list or a wrong billing period string can result in more charges than the expected total — the cap absorbs the overrun and stops further charges before the full damage is done. The two safeguards together — pre-flight write and spend cap — provide defense-in-depth against billing bugs that survive the idempotency key layer.

Idempotency key placement table for HC5 async

Relationship to the HC5 sync client post

The Apache HttpClient 5 sync post covered three failure modes rooted in the synchronous execution chain: HttpRequestInterceptor.process() re-running on every retry attempt (UUID per interceptor invocation), a Kubernetes multi-pod TOCTOU scheduler race, and ConnectionRequestTimeoutException retry generating UUID_B on connection pool exhaustion. The async failure modes in this post are structurally distinct because the async model separates request construction (calling thread) from failure notification (FutureCallback.failed() on the I/O reactor thread). The separation makes UUID re-generation in failed() the natural mistake, whereas the sync post’s mistakes arise from the interceptor chain or from concurrent pod scheduling. Both posts share the same fix strategy — stable content-hash keys computed before the retry boundary — and the same backstops: pre-flight ON CONFLICT DO NOTHING and vault key spend cap.

For teams that use HC5 async for Stripe billing specifically because of its HTTP/2 support, the GOAWAY failure mode in this post is the most operationally significant. Load balancer rotations and API gateway deployments happen regularly in production infrastructure; a GOAWAY frame is the mechanism by which they are communicated to long-lived HTTP/2 clients. Every billing service using HC5’s H2AsyncClientBuilder against Stripe should audit whether it uses stable idempotency keys before the execute() call, whether failed() re-uses those keys on retry, and whether the pre-flight ON CONFLICT DO NOTHING guard is in place for the case where automatic stream migration and manual retry both fire for the same stream (a transient scenario that can occur during H2 connection pool reconfiguration).