Async HTTP Client and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

The Async HTTP Client library (org.asynchttpclient, formerly Ning, built on Netty) introduces three Stripe billing failure modes that are structurally distinct from those seen in Apache HttpClient 5, OkHttp, and Java 11’s built-in java.net.http.HttpClient. Three org.asynchttpclient-specific failure modes: IOExceptionFilter.filter() runs before each retry triggered by setMaxRequestRetry(n) — if the filter generates UUID.randomUUID() per invocation, the original request carries UUID_A and the first retry carries UUID_B, creating ch_B when ch_A already committed before the I/O exception; new RequestBuilder(originalRequest) in AsyncCompletionHandler.onThrowable() copies all headers including Idempotency-Key: UUID_A, and a subsequent addHeader() call appends UUID_B instead of replacing, producing a retry request with two Idempotency-Key values that Stripe processes as a new request; and both AsyncCompletionHandler.onThrowable() and ListenableFuture.toCompletableFuture().exceptionally() fire independently when a request fails — a developer who implements retry billing in both paths triggers two simultaneous retry billing calls on the first failure, each with a different UUID, creating ch_B and ch_C for a customer already charged ch_A.

This post covers all three failure modes with AsyncHttpClient 2.x/3.x code (IOExceptionFilter, RequestBuilder, AsyncCompletionHandler, ListenableFuture), the addHeader() vs setHeader() distinction in RequestBuilder, the onThrowable() and toCompletableFuture() interaction semantics, content-hash idempotency keys stable across all retry paths, pre-flight PostgreSQL ON CONFLICT DO NOTHING as the authoritative billing mutex — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For Netty pipeline handler patterns, see the Netty Pipeline and Stripe Integration post. For Apache HttpClient 5 request interceptor patterns, see the Apache HttpClient 5 and Stripe Integration post. For OkHttp application interceptor patterns, see the OkHttp and Retrofit and Stripe Integration post.

Failure mode 1: IOExceptionFilter.filter() generates UUID.randomUUID() per I/O exception — runs before each retry triggered by setMaxRequestRetry(n) — original request carries UUID_A — filter overwrites with UUID_B before first retry — ch_B when ch_A already committed

AsyncHttpClient’s setMaxRequestRetry(n) configuration provides built-in retry on IOException. When an I/O exception occurs (connection refused, connection reset, timeout during send), AsyncHttpClient invokes the registered IOExceptionFilter chain before deciding whether to replay the request. The filter receives a FilterContext containing the original request, the handler, and the exception, and can return a modified FilterContext — including a modified request — alongside a replayRequest(true) signal that tells AsyncHttpClient to retry with the new request object rather than the original.

A common pattern for centralising idempotency key management is to generate or refresh the key inside the filter, avoiding the need to set it at every call site:

// BillingService.java — UNSAFE: IOExceptionFilter generates UUID per invocation.
import org.asynchttpclient.*;
import org.asynchttpclient.filter.FilterContext;
import org.asynchttpclient.filter.FilterException;
import org.asynchttpclient.filter.IOExceptionFilter;
import java.util.UUID;

AsyncHttpClientConfig config = Dsl.config()
    .setMaxRequestRetry(2) // retry up to 2 times on IOException
    .addIOExceptionFilter(new IOExceptionFilter() {
        @Override
        public <T> FilterContext<T> filter(FilterContext<T> ctx) throws FilterException {
            if (ctx.getIOException() instanceof java.net.ConnectException) {
                // BUG: UUID.randomUUID() called per filter invocation (per I/O exception).
                // Original request already has Idempotency-Key: UUID_A.
                // This setHeader() replaces UUID_A with UUID_B on the first retry.
                // If Stripe committed ch_A before the TCP disconnect, retry with UUID_B creates ch_B.
                Request retryReq = new RequestBuilder(ctx.getRequest())
                    .setHeader("Idempotency-Key", UUID.randomUUID().toString()) // NEW UUID per retry
                    .build();
                return new FilterContext.FilterContextBuilder<>(ctx)
                    .request(retryReq)
                    .replayRequest(true)
                    .build();
            }
            return ctx;
        }
    })
    .build();

AsyncHttpClient client = Dsl.asyncHttpClient(config);

// Initial request — carries UUID_A set at call site.
Request req = new RequestBuilder("POST")
    .setUrl("https://api.stripe.com/v1/charges")
    .addHeader("Authorization", "Bearer " + stripeKey)
    .addHeader("Idempotency-Key", UUID.randomUUID().toString()) // UUID_A
    .addHeader("Content-Type", "application/x-www-form-urlencoded")
    .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
    .build();

ListenableFuture<Response> future = client.executeRequest(req);

The mechanics: client.executeRequest(req) sends the POST /v1/charges to Stripe. Stripe processes the request, creates ch_A (charge A), and then the TCP connection is terminated by a ConnectException before the HTTP response is delivered. AsyncHttpClient invokes IOExceptionFilter.filter(). The filter calls UUID.randomUUID().toString() — this produces UUID_B, a completely new, unrelated key — and calls setHeader("Idempotency-Key", UUID_B) on a new RequestBuilder copy of the original request. The filter returns replayRequest(true). AsyncHttpClient retries with the new request carrying UUID_B. Stripe sees a key it has never processed and treats it as a new charge request, creating ch_B. The customer is charged twice.

The developer’s intent was to “generate a fresh idempotency key for the retry” — which sounds correct if you believe that idempotency keys must be unique per request. But Stripe’s idempotency system works in the opposite direction: the same key must be used across all retries of the same logical operation, so that Stripe’s cache can return “you already created this charge” rather than creating a new one. A fresh UUID per retry is exactly what defeats that protection.

The subtler variant: IOExceptionFilter is the sole source of the idempotency key — developer assumes the filter “only runs on retries” — original request has no Idempotency-Key header — Stripe processes it as a non-idempotent charge — retry carries UUID_B — two independent charge requests

A second pattern treats IOExceptionFilter as a retry-only concern: “I don’t need an idempotency key on the first attempt because if it succeeds there is nothing to retry.” This reasoning sounds defensible — idempotency keys protect retries, and there is no retry on the happy path. But it is wrong in the failure case:

// BillingService.java — UNSAFE: original request has no Idempotency-Key.
// Developer adds the key only in IOExceptionFilter, thinking “I only need it for retries.”

// Original request: NO Idempotency-Key header.
Request req = new RequestBuilder("POST")
    .setUrl("https://api.stripe.com/v1/charges")
    .addHeader("Authorization", "Bearer " + stripeKey)
    .addHeader("Content-Type", "application/x-www-form-urlencoded")
    .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
    // NO addHeader("Idempotency-Key", ...) here
    .build();

// IOExceptionFilter adds the key — runs only on retry, not on the first attempt.
AsyncHttpClientConfig config = Dsl.config()
    .addIOExceptionFilter(new IOExceptionFilter() {
        @Override
        public <T> FilterContext<T> filter(FilterContext<T> ctx) throws FilterException {
            Request retryReq = new RequestBuilder(ctx.getRequest())
                .setHeader("Idempotency-Key", UUID.randomUUID().toString()) // UUID_B on retry
                .build();
            return new FilterContext.FilterContextBuilder<>(ctx)
                .request(retryReq)
                .replayRequest(true)
                .build();
        }
    })
    .build();

When the first request (no key) is sent, Stripe processes it as a non-idempotent charge and creates ch_A. Stripe does not record any idempotency key for ch_A — there is nothing in its cache to match against on retry. The IOException fires. IOExceptionFilter generates UUID_B and attaches it to the retry request. Stripe receives the retry with UUID_B — a key it has never seen — and creates ch_B. The customer is charged twice. Worse, the developer has no way to detect this via Stripe’s idempotency system, because the two charges have no idempotency relationship: ch_A was created without a key, and ch_B was created with UUID_B. They appear as two separate, unrelated charges in the audit log.

The rule Stripe documents is that an idempotency key should be set on the first attempt of any charge request, not only on retries. The purpose of setting it on the first attempt is to establish the key in Stripe’s cache before any network failure can occur. If the key is only set on retries, a failure between the first request (committed, no key) and the retry (UUID_B) produces two unrelated charges with no idempotency connection between them.

Fix: compute a stable content-hash key before the request; set it on the original request; have IOExceptionFilter preserve it from ctx.getRequest() rather than overwriting

// BillingService.java — FIXED.
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;

public class BillingService {
    private final AsyncHttpClient client;

    public BillingService() {
        AsyncHttpClientConfig config = Dsl.config()
            .setMaxRequestRetry(2)
            .addIOExceptionFilter(new IOExceptionFilter() {
                @Override
                public <T> FilterContext<T> filter(FilterContext<T> ctx) throws FilterException {
                    // Preserve the existing Idempotency-Key from the original request.
                    // Do NOT generate a new UUID here — the key was already set once,
                    // before the first attempt, and must remain the same on every retry.
                    String existingKey = ctx.getRequest()
                        .getHeaders().get("Idempotency-Key");
                    if (existingKey != null) {
                        // Key already correct — replay with the same request unchanged.
                        return new FilterContext.FilterContextBuilder<>(ctx)
                            .replayRequest(true)
                            .build();
                    }
                    // If somehow the key was missing, compute it from userData.
                    String key = (String) ctx.getRequest().getUserData();
                    if (key == null) return ctx; // cannot safely retry without a key
                    Request retryReq = new RequestBuilder(ctx.getRequest())
                        .setHeader("Idempotency-Key", key)
                        .build();
                    return new FilterContext.FilterContextBuilder<>(ctx)
                        .request(retryReq)
                        .replayRequest(true)
                        .build();
                }
            })
            .build();
        this.client = Dsl.asyncHttpClient(config);
    }

    public ListenableFuture<ChargeResponse> chargeCustomer(String customerId, String period) {
        // Compute stable key once from billing fields — same on every attempt, every JVM.
        String idempotencyKey = sha256(customerId + ":" + period + ":asynchttpclient-billing");

        Request req = new RequestBuilder("POST")
            .setUrl("https://api.stripe.com/v1/charges")
            .addHeader("Authorization", "Bearer " + stripeKey)
            .addHeader("Idempotency-Key", idempotencyKey) // stable key on the FIRST attempt
            .addHeader("Content-Type", "application/x-www-form-urlencoded")
            .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
            .setUserData(idempotencyKey) // also stored in userData as a secondary access path
            .build();

        return client.executeRequest(req, new AsyncCompletionHandler<ChargeResponse>() {
            @Override
            public ChargeResponse onCompleted(Response response) throws Exception {
                return parseCharge(response.getResponseBody());
            }
        });
    }

    private static String sha256(String input) {
        try {
            MessageDigest md = MessageDigest.getInstance("SHA-256");
            byte[] hash = md.digest(input.getBytes(StandardCharsets.UTF_8));
            StringBuilder sb = new StringBuilder();
            for (byte b : hash) sb.append(String.format("%02x", b));
            return sb.toString().substring(0, 32);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}

Three properties of this fix: (1) sha256(customerId + ":" + period + ":asynchttpclient-billing") is evaluated once before the request is built — the same value is produced for any given (customerId, period) pair on any JVM instance, on any retry, at any time; (2) the key is set as an Idempotency-Key header on the initial request, establishing it in Stripe’s cache before any network failure; (3) the IOExceptionFilter reads the existing header from ctx.getRequest().getHeaders().get("Idempotency-Key") and replays without modification — it never calls UUID.randomUUID() inside the filter. The setUserData(idempotencyKey) stores the key in the request’s userData field as a secondary access path for cases where headers might be modified by another filter in the chain.

Failure mode 2: new RequestBuilder(originalRequest) in AsyncCompletionHandler.onThrowable() inherits Idempotency-Key: UUID_A from the original request — .addHeader() appends UUID_B instead of replacing — retry carries two Idempotency-Key header values — Stripe’s behavior is undefined — potential ch_B

AsyncCompletionHandler.onThrowable() is the natural hook for application-level retry in AsyncHttpClient — called when the request fails with any Throwable that the library does not handle internally. A common pattern is to construct a new request from the failed request (to preserve the URL, headers, and body) and re-submit it via client.executeRequest(). AsyncHttpClient provides a copy constructor: new RequestBuilder(Request) that copies all properties — URL, method, headers, body, timeouts — from the original request into a new builder. The developer then calls .addHeader() to update the idempotency key for the retry:

// BillingService.java — UNSAFE: addHeader() appends to the inherited header.
public class BillingHandler extends AsyncCompletionHandler<ChargeResponse> {
    private final Request originalRequest;
    private final AsyncHttpClient client;
    private final int retries;

    public BillingHandler(Request originalRequest, AsyncHttpClient client, int retries) {
        this.originalRequest = originalRequest;
        this.client = client;
        this.retries = retries;
    }

    @Override
    public ChargeResponse onCompleted(Response response) throws Exception {
        return parseCharge(response.getResponseBody());
    }

    @Override
    public void onThrowable(Throwable t) {
        if (retries < 3 && isRetryable(t)) {
            // new RequestBuilder(originalRequest) copies ALL headers from originalRequest.
            // originalRequest already has Idempotency-Key: UUID_A.
            // .addHeader() APPENDS — does not replace.
            // Retry request now has TWO Idempotency-Key headers: UUID_A (inherited) and UUID_B (added).
            Request retryReq = new RequestBuilder(originalRequest)
                .addHeader("Idempotency-Key", UUID.randomUUID().toString()) // BUG: appends UUID_B
                .build();
            client.executeRequest(retryReq, new BillingHandler(retryReq, client, retries + 1));
        }
    }
}

The mechanics: the original request carries Idempotency-Key: UUID_A. Stripe processes it, creates ch_A, and then the connection is reset before the response is received. onThrowable() fires. new RequestBuilder(originalRequest) creates a builder with all of the original request’s headers, including Idempotency-Key: UUID_A. The developer calls .addHeader("Idempotency-Key", UUID.randomUUID().toString()) to attach UUID_B. RequestBuilder.addHeader() — like java.net.http.HttpRequest.Builder.header() — appends a new header entry with the given name, it does not replace an existing entry with the same name. The built retry request carries both Idempotency-Key: UUID_A and Idempotency-Key: UUID_B as two distinct header fields.

HTTP allows multiple header fields with the same field name, and RFC 9110 specifies that for most headers, multiple instances should be treated as equivalent to a single field with a comma-joined value. But Idempotency-Key is not a list-valued header in Stripe’s protocol — it is a single-value field intended to specify exactly one key per request. When Stripe’s API gateway receives two Idempotency-Key fields, the behavior depends on implementation: Stripe may take the first (UUID_A, returning the cached ch_A response — the safe outcome), take the last (UUID_B, treating it as a new request — ch_B), or return a 400 error for a malformed request. The 400 path is arguably safe (no duplicate charge, but the retry is non-fatal if the application handles it), but the last-value path is a silent duplicate charge. The safe behavior is not guaranteed by Stripe’s public documentation and may differ across API versions, regions, or edge gateways.

The subtler variant: even .setHeader() with UUID.randomUUID() is wrong — replacing UUID_A with UUID_B produces a single correct-looking header but still creates ch_B — the correct fix is to reuse UUID_A, not generate UUID_B via either method

The natural correction to addHeader() accumulation is to use setHeader() instead, which replaces any existing value for the given header name. This eliminates the two-header problem: the retry request will carry exactly one Idempotency-Key field. But if that field contains UUID.randomUUID().toString() — a new UUID generated at retry time — the header accumulation bug is gone but the idempotency key freshness bug remains:

// Still UNSAFE: setHeader() fixes the accumulation but not the new-UUID-per-retry problem.
Request retryReq = new RequestBuilder(originalRequest)
    .setHeader("Idempotency-Key", UUID.randomUUID().toString()) // UUID_B replaces UUID_A
    .build();
// Retry request has one Idempotency-Key: UUID_B.
// Stripe sees a key it has never processed and creates ch_B. Same failure, cleaner request.

The root cause of failure mode 2 is not addHeader() vs setHeader() — it is generating a new UUID for the retry at all. The correct behavior is to reuse the same key that was set on the original request. new RequestBuilder(originalRequest) already copies Idempotency-Key: UUID_A into the builder. The safest retry path is to not call either addHeader() or setHeader() for the idempotency key at all — let the inherited UUID_A remain. If the developer insists on an explicit set, the correct value to set is the original key, read back from the original request:

// Extract original key — do not generate a new one.
String originalKey = originalRequest.getHeaders().get("Idempotency-Key");

Request retryReq = new RequestBuilder(originalRequest)
    // Either: do nothing (inherited UUID_A is already correct)
    // Or: explicitly set the same original key (defensive, makes intent clear)
    .setHeader("Idempotency-Key", originalKey) // same UUID_A
    .build();

Fix: use a stable content-hash key so the “inherited from original” path and the “set explicitly on retry” path both produce the same value; never call UUID.randomUUID() inside a retry handler

// BillingService.java — FIXED.
public class BillingHandler extends AsyncCompletionHandler<ChargeResponse> {
    private final Request originalRequest;
    private final String idempotencyKey; // stable, pre-computed before first attempt
    private final AsyncHttpClient client;
    private final int retries;

    public BillingHandler(Request originalRequest, String idempotencyKey,
                          AsyncHttpClient client, int retries) {
        this.originalRequest = originalRequest;
        this.idempotencyKey = idempotencyKey;
        this.client = client;
        this.retries = retries;
    }

    @Override
    public ChargeResponse onCompleted(Response response) throws Exception {
        return parseCharge(response.getResponseBody());
    }

    @Override
    public void onThrowable(Throwable t) {
        if (retries < 3 && isRetryable(t)) {
            // new RequestBuilder(originalRequest) inherits Idempotency-Key: idempotencyKey.
            // We do not call addHeader() or setHeader() for it — the inherited value is correct.
            // The explicit idempotencyKey field is kept as documentation and for logging,
            // but the header value comes from the original request.
            Request retryReq = new RequestBuilder(originalRequest).build();
            client.executeRequest(retryReq,
                new BillingHandler(retryReq, idempotencyKey, client, retries + 1));
        }
    }
}

// Call site: compute key once, pass to handler.
public ListenableFuture<ChargeResponse> chargeCustomer(String customerId, String period) {
    String key = sha256(customerId + ":" + period + ":asynchttpclient-billing");

    Request req = new RequestBuilder("POST")
        .setUrl("https://api.stripe.com/v1/charges")
        .addHeader("Authorization", "Bearer " + stripeKey)
        .addHeader("Idempotency-Key", key) // stable key, not UUID.randomUUID()
        .addHeader("Content-Type", "application/x-www-form-urlencoded")
        .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
        .build();

    return client.executeRequest(req, new BillingHandler(req, key, client, 0));
}

With a stable content-hash key, the RequestBuilder(originalRequest) copy constructor behavior becomes a feature rather than a footgun: it copies UUID_A (which equals sha256(customerId:period:asynchttpclient-billing)[:32]) from the original request, and every retry inherits that same value without any additional addHeader() or setHeader() call. If the developer does add an explicit setHeader("Idempotency-Key", idempotencyKey) call on the retry builder, sha256 of the same inputs produces the same string — the replace is idempotent. The only path that fails is UUID.randomUUID() inside the retry handler, and that path is eliminated by computing the key before the first request.

Failure mode 3: AsyncCompletionHandler.onThrowable() and ListenableFuture.toCompletableFuture().exceptionally() are independent error-handling hooks that both fire when a request fails — developer implements retry billing in both paths — two simultaneous billing calls on the first failure — ch_B from onThrowable() and ch_C from exceptionally()

AsyncHttpClient’s ListenableFuture<T> extends java.util.concurrent.CompletionStage<T>, which means it has a toCompletableFuture() bridge method that returns a CompletableFuture<T> backed by the same underlying result. This opens up two parallel error-handling paths for the same request: the AsyncCompletionHandler.onThrowable() callback, which fires synchronously on the Netty event loop when the I/O layer encounters a Throwable, and the CompletableFuture exception pipeline (via exceptionally(), handle(), or whenComplete()), which fires when the CompletableFuture completes exceptionally. Both paths respond to the same underlying event — the request’s failure — and both fire independently:

// BillingService.java — UNSAFE: retry implemented in both onThrowable() and exceptionally().
public void chargeCustomer(String customerId, String period) {
    String key = sha256(customerId + ":" + period + ":asynchttpclient-billing");

    Request req = new RequestBuilder("POST")
        .setUrl("https://api.stripe.com/v1/charges")
        .addHeader("Authorization", "Bearer " + stripeKey)
        .addHeader("Idempotency-Key", key)
        .addHeader("Content-Type", "application/x-www-form-urlencoded")
        .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
        .build();

    // Developer A adds retry in onThrowable() to handle network exceptions.
    ListenableFuture<Response> future = client.executeRequest(req,
        new AsyncCompletionHandler<Response>() {
            @Override
            public Response onCompleted(Response response) throws Exception {
                return response;
            }

            @Override
            public void onThrowable(Throwable t) {
                if (isRetryable(t)) {
                    // BUG path 1: calls retryBilling() which generates UUID.randomUUID() per call.
                    retryBilling(customerId, period); // UUID_B → ch_B
                }
            }
        });

    // Developer B (or Developer A later) adds retry via CompletableFuture for a different
    // failure scenario — e.g., non-2xx Stripe errors surfaced as exceptions in onCompleted().
    future.toCompletableFuture()
        .exceptionally(e -> {
            if (isRetryable(e)) {
                // BUG path 2: also calls retryBilling() — fires on the SAME failure as onThrowable().
                retryBilling(customerId, period); // UUID_C → ch_C
            }
            return null;
        });
}

private void retryBilling(String customerId, String period) {
    // BUG: UUID.randomUUID() at method entry — new UUID per call.
    String newKey = UUID.randomUUID().toString(); // UUID_B on first call, UUID_C on second
    Request retryReq = new RequestBuilder("POST")
        .setUrl("https://api.stripe.com/v1/charges")
        .addHeader("Authorization", "Bearer " + stripeKey)
        .addHeader("Idempotency-Key", newKey)
        .addHeader("Content-Type", "application/x-www-form-urlencoded")
        .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
        .build();
    client.executeRequest(retryReq); // fire and forget retry
}

The mechanics: the initial request sends Idempotency-Key: UUID_A. Stripe creates ch_A. The TCP connection drops. AsyncHttpClient invokes onThrowable(ConnectException) — path 1 fires. Simultaneously, the ListenableFuture completes exceptionally, which propagates to the CompletableFuture and fires the exceptionally() callback — path 2 fires. Both paths call retryBilling(customerId, period), which calls UUID.randomUUID() at its entry. Path 1 gets UUID_B, submits a charge request with UUID_B, Stripe creates ch_B. Path 2 gets UUID_C (a different call to UUID.randomUUID() from a different execution context), submits a charge request with UUID_C, Stripe creates ch_C. On a single I/O exception, one customer is charged three times: ch_A from the original, ch_B from onThrowable(), and ch_C from exceptionally().

This failure mode is specific to AsyncHttpClient because it is the only mainstream Java HTTP client that simultaneously exposes both a callback-based handler API (AsyncCompletionHandler) and a CompletionStage-compatible future from the same executeRequest() call. Apache HttpClient 5 returns a Future but not a CompletionStage. Java’s HttpClient.sendAsync() returns a CompletableFuture with no separate handler callback. OkHttp’s Call.enqueue(Callback) has only the callback path. AsyncHttpClient’s dual-path API is a convenience feature for teams migrating from callback-based code to reactive composition — but it creates a trap where both paths respond to the same event.

The subtler variant: developer intends each path to cover a different failure category — onThrowable() for network exceptions, exceptionally() for application-level failures thrown from onCompleted() — but network exceptions propagate through both paths simultaneously

The intent behind implementing both error paths is sometimes a reasonable division of concerns: onThrowable() handles IOException and ConnectException at the network layer; exceptionally() handles failures thrown from inside onCompleted() (such as JSON parse errors or non-2xx status codes surfaced as exceptions). The developer models these as disjoint: “onThrowable() catches network failures, exceptionally() catches application failures from onCompleted, they never both fire for the same error.”

This model is wrong. When onThrowable(t) fires, the ListenableFuture is completed exceptionally with t. Because CompletableFuture (via the toCompletableFuture() bridge) is backed by the same result, it also completes exceptionally with t. The exceptionally() handler fires for any exceptional completion, including network exceptions that already triggered onThrowable(). The developer’s disjoint model fails for all IOException and ConnectException failures: both onThrowable() fires and the CompletableFuture propagates the exception, triggering exceptionally().

The only exceptions that do not propagate to exceptionally() would be those swallowed entirely by onThrowable() without completing the future exceptionally — but AsyncHttpClient completes the future based on the I/O result, not on what onThrowable() does with the exception. If onThrowable() submits a new retry request, it does not complete the original future successfully; the original future remains exceptionally completed, and exceptionally() still fires on the original failure.

Fix: use exactly one error-handling path — if toCompletableFuture() is used for downstream composition, make onThrowable() a no-op; if retry logic lives in onThrowable(), do not chain exceptionally() on the CF bridge for the same class of failures

// BillingService.java — FIXED: single error-handling path.
// Option A: all retry logic in AsyncCompletionHandler; toCompletableFuture() used read-only.
public ListenableFuture<ChargeResponse> chargeCustomer(String customerId, String period) {
    String key = sha256(customerId + ":" + period + ":asynchttpclient-billing");

    Request req = new RequestBuilder("POST")
        .setUrl("https://api.stripe.com/v1/charges")
        .addHeader("Authorization", "Bearer " + stripeKey)
        .addHeader("Idempotency-Key", key)
        .addHeader("Content-Type", "application/x-www-form-urlencoded")
        .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
        .build();

    return client.executeRequest(req, new AsyncCompletionHandler<ChargeResponse>() {
        private int attempts = 0;

        @Override
        public ChargeResponse onCompleted(Response response) throws Exception {
            if (response.getStatusCode() >= 500) {
                // Surface Stripe 5xx as an exception for the caller to handle.
                throw new StripeServerException(response.getStatusCode(), response.getResponseBody());
            }
            return parseCharge(response.getResponseBody());
        }

        @Override
        public void onThrowable(Throwable t) {
            if (attempts < 3 && isRetryable(t)) {
                attempts++;
                // Reuse the SAME request (same Idempotency-Key: key) — no new UUID.
                client.executeRequest(req, this); // re-use this handler for retry count tracking
            }
            // If not retryable or max retries exceeded: the future completes exceptionally.
            // The caller handles this via future.get() or toCompletableFuture().exceptionally().
        }
    });
}

// Caller: ONE path for exceptions — no separate exceptionally() adding retry.
ListenableFuture<ChargeResponse> future = billingService.chargeCustomer(customerId, period);
future.toCompletableFuture()
    .thenAccept(charge -> log.info("Charged: {}", charge.id()))
    .exceptionally(e -> {
        // Logging and escalation only — no retry here, retry is in onThrowable().
        log.error("Billing failed after retries for {}: {}", customerId, e.getMessage());
        alertOps(customerId, e);
        return null;
    });

// Option B: all composition via CompletableFuture; AsyncCompletionHandler is pass-through only.
public CompletableFuture<ChargeResponse> chargeCustomerCF(String customerId, String period) {
    String key = sha256(customerId + ":" + period + ":asynchttpclient-billing");

    Request req = buildRequest(key, customerId);

    // AsyncCompletionHandler is a simple pass-through — no retry logic, no onThrowable() override.
    return client.executeRequest(req) // no handler arg — uses default pass-through handler
        .toCompletableFuture()
        .thenApply(response -> parseCharge(response.getResponseBody()))
        .exceptionally(e -> {
            if (isRetryable(e)) {
                // Retry with same request object — same Idempotency-Key: key inherited.
                // chargeWithRetry() accepts a pre-built Request so no UUID.randomUUID() fires.
                return chargeWithRetry(req, 1).join();
            }
            throw new CompletionException(e);
        });
}

private CompletableFuture<ChargeResponse> chargeWithRetry(Request req, int attempt) {
    if (attempt >= 3) return CompletableFuture.failedFuture(new RuntimeException("Max retries"));
    return client.executeRequest(req) // same request, same Idempotency-Key on every retry
        .toCompletableFuture()
        .thenApply(r -> parseCharge(r.getResponseBody()))
        .exceptionally(e -> {
            if (isRetryable(e) && attempt < 3) return chargeWithRetry(req, attempt + 1).join();
            throw new CompletionException(e);
        });
}

Option A is appropriate when handler-level control is preferred (fine-grained retry policies, Netty thread awareness). Option B is appropriate for teams building reactive composition chains. The critical constraint in both is the same: retry must re-use the same pre-built Request object (which carries the stable Idempotency-Key: key), and the retry logic must live in exactly one location. Option B passes req directly to chargeWithRetry() rather than reconstructing it per attempt, so the key is never re-computed.

Spend cap via vault key: the financial backstop for all three failure modes

All three failure modes above produce duplicate Stripe charges through protocol-level bugs: the idempotency key changes between attempts (failure modes 1 and 3), or the retry request carries ambiguous or incorrect key state (failure mode 2). Stable content-hash keys and a single retry path are the correct fixes at the application layer. But there is a second category of risk: a retry misconfiguration, a runaway billing loop in an autonomous agent, or a staged deployment that runs old and new code simultaneously, where both versions retry the same customer’s billing with slightly different key derivation logic. A vault key with a per-billing-period spend cap provides a hard financial limit enforced outside the application code:

// Keybrake vault key for AsyncHttpClient billing service:
// - vendor: stripe
// - allowed endpoints: POST /v1/charges only
// - daily_usd_cap: expected_daily_billing_total * 1.10 (10% headroom)
// - expires_at: end of billing period + 1 hour

// The proxy sits between AsyncHttpClient and Stripe:
// client → proxy.keybrake.com/stripe/v1/charges → api.stripe.com/v1/charges

Request req = new RequestBuilder("POST")
    .setUrl("https://proxy.keybrake.com/stripe/v1/charges") // proxy endpoint
    .addHeader("Authorization", "Bearer vault_key_xxx") // vault key, not raw Stripe key
    .addHeader("Idempotency-Key", key)
    .addHeader("Content-Type", "application/x-www-form-urlencoded")
    .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
    .build();

With a spend cap set to 110% of the expected period total, a duplicate-charge bug from any of the three failure modes creates at most one duplicate before the proxy rejects further charges with a policy violation response. The audit log records every charge attempt with its idempotency key and policy verdict — the double-UUID pattern (UUID_A then UUID_B from failure modes 1 and 2, or three different UUIDs from failure mode 3) surfaces immediately in the per-period run report, distinct from the stable content-hash keys seen in correctly functioning billing runs.

Pre-flight database guard: the cluster-wide billing mutex for AsyncHttpClient fan-outs

AsyncHttpClient is designed for high-concurrency workloads — its Netty foundation handles thousands of concurrent connections efficiently. Billing services that use it to fan out charges across a customer list can submit hundreds of concurrent Stripe requests. In such fan-outs, multiple execution paths may reach the “has this customer been billed this period?” check at the same time, before any of them has committed a charge. An in-application check (“check a map of already-billed customers”) is not cluster-safe: two JVM instances, or two async threads within the same JVM, can both pass the check before either has received Stripe’s confirmation.

// BillingService.java — FIXED with pre-flight database guard.
public ListenableFuture<ChargeResult> chargeCustomerSafe(String customerId, String period,
                                                          DataSource db) {
    String key = sha256(customerId + ":" + period + ":asynchttpclient-billing");

    // Pre-flight: attempt to INSERT a billing record. UNIQUE constraint on (customer_id, billing_period)
    // ensures only one attempt per customer per period succeeds across the entire cluster.
    int inserted = db.getConnection().prepareStatement(
        "INSERT INTO billing_records (customer_id, billing_period, idempotency_key, status) " +
        "VALUES (?, ?, ?, 'pending') ON CONFLICT (customer_id, billing_period) DO NOTHING")
        .setString(1, customerId)
        .setString(2, period)
        .setString(3, key)
        .executeUpdate();

    if (inserted == 0) {
        // Another thread or JVM already claimed this customer for this period.
        return CompletableFuture.completedFuture(ChargeResult.skipped(customerId))
            .toListenableFuture(); // or wrap appropriately
    }

    Request req = new RequestBuilder("POST")
        .setUrl("https://proxy.keybrake.com/stripe/v1/charges")
        .addHeader("Authorization", "Bearer vault_key_xxx")
        .addHeader("Idempotency-Key", key)
        .addHeader("Content-Type", "application/x-www-form-urlencoded")
        .setBody("customer=" + customerId + "&amount=2999¤cy=usd")
        .build();

    return client.executeRequest(req, new AsyncCompletionHandler<ChargeResult>() {
        @Override
        public ChargeResult onCompleted(Response response) throws Exception {
            ChargeResponse charge = parseCharge(response.getResponseBody());
            db.getConnection().prepareStatement(
                "UPDATE billing_records SET status='committed', stripe_charge_id=? " +
                "WHERE customer_id=? AND billing_period=?")
                .setString(1, charge.id())
                .setString(2, customerId)
                .setString(3, period)
                .executeUpdate();
            return ChargeResult.charged(customerId, charge.id());
        }
    });
}

The pre-flight INSERT ... ON CONFLICT DO NOTHING uses a UNIQUE constraint on (customer_id, billing_period) as the authoritative billing mutex. Any concurrent execution — a second async request for the same customer fired by a Netty worker thread, a duplicate triggered by a racing CompletableFuture stage, or a retry submitted by onThrowable() for a request that actually succeeded — finds an existing row and returns 0 rows inserted, skipping the Stripe request entirely. This guard is independent of idempotency key correctness: even if the key changes between the original and a retry (failure modes 1 or 3), the database row inserted on the original attempt prevents the retry from reaching Stripe at all. The two fixes are complementary and provide independent safety layers.

Putting it together: three org.asynchttpclient failure modes and their fixes

AsyncHttpClient’s failure modes share a common origin with those in other Java HTTP clients — the HTTP library has no opinion on idempotency, and all idempotency discipline lives in application code — but the specific shapes of the failures are distinct from those in Apache HttpClient 5, OkHttp, and Java’s HttpClient:

• IOExceptionFilter is AsyncHttpClient’s hook for modifying requests before built-in retry — it runs per exception, not once per original request — generating UUID.randomUUID() inside the filter produces a new key per exception, replacing the original UUID_A with UUID_B before the first retry.
• The RequestBuilder(originalRequest) copy constructor copies all headers including the idempotency key, and addHeader() then appends a second key rather than replacing — the retry carries two Idempotency-Key values whose combined behavior against Stripe is undefined.
• ListenableFuture.toCompletableFuture() exposes the same request result through two independent exception-handling paths — onThrowable() and exceptionally() both fire on a network exception, and both calling executeRequest() with fresh UUIDs produces simultaneous duplicate charges.

The common thread across all three org.asynchttpclient failure modes is the same pattern seen in every Java HTTP client post in this series: the HTTP client library is agnostic to idempotency, and the failure arises from where application code evaluates UUID.randomUUID() relative to the retry boundary. What is specific to AsyncHttpClient is the shape: IOExceptionFilter’s per-exception execution model (distinct from Apache HttpClient 5’s HttpRequestInterceptor which runs once per original execution), RequestBuilder’s copy constructor (which makes header inheritance both a convenience and a footgun), and the dual error-handling paths exposed by the CompletionStage-compatible ListenableFuture (unique to AsyncHttpClient among mainstream Java HTTP clients). Content-hash idempotency keys derived from stable billing fields (sha256(customerId:billingPeriod:asynchttpclient-billing)[:32]) and a pre-flight database guard on (customer_id, billing_period) close all three failure modes: the key is the same whether computed for the first attempt or inherited by a retry copy, and the guard ensures the Stripe charge is not attempted for customers already billed regardless of which retry path fires.