Undertow Embedded Server and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Undertow used directly as a standalone embedded server — not as the servlet container backing a Spring Boot application — introduces three Stripe billing failure modes that are structurally distinct from every framework-layer pattern covered in earlier posts. The failure modes arise from Undertow’s low-level APIs: UndertowClient’s two-level ClientCallback chain builds a new ClientRequest object inside the connection callback and evaluates UUID.randomUUID() per callback invocation, so a retry wrapper that reconnects on failure creates ch_B with a fresh UUID while ch_A is already committed; HttpServerExchange.dispatch() hands a blocking billing task to an XNIO worker thread, and a manual retry loop inside the dispatched Runnable that calls UUID.randomUUID() at the top of each iteration generates a distinct idempotency key per attempt and creates ch_B on the second iteration if ch_A was committed before a socket timeout; and XnioIoThread.executeAfter() schedules a delayed retry Runnable without a ScheduledExecutorService dependency, and a recursive billing helper that calls UUID.randomUUID() at its entry point generates a new key on every executeAfter() callback invocation, creating ch_B the first time the delayed retry fires.

This post covers all three failure modes with Java code (Undertow 2.3.x, XNIO 3.8.x, UndertowClient, HttpServerExchange, XnioWorker), the UndertowClient callback chain lifecycle and why ClientRequest objects are typically built inside callbacks, the HttpServerExchange.dispatch() dispatch model and why blocking code must run in worker threads, the XnioIoThread.executeAfter() scheduled delay mechanism and why UUID.randomUUID() at method entry re-evaluates per scheduled callback, content-hash idempotency keys stable across all three retry patterns, Undertow’s AttachmentKey API for carrying stable state from the I/O thread through the dispatch boundary to worker threads and subsequent executeAfter() callbacks, parameter threading through recursive retry calls, pre-flight PostgreSQL ON CONFLICT DO NOTHING as the authoritative billing mutex, and per-billing-period vault keys as the financial backstop. The distinction from Spring Boot’s embedded Undertow adapter is important: when Spring Boot uses Undertow as its embedded container, the application uses Spring’s RestTemplate, WebClient, or Spring Retry for outbound HTTP — the Spring and Reactor failure modes covered in the Spring Retry and Spring WebFlux posts apply. This post covers standalone Undertow applications that use Undertow’s own APIs directly: UndertowClient for outbound HTTP, HttpHandler and HttpServerExchange for request handling, and the XNIO worker infrastructure for async scheduling.

Failure mode 1: UndertowClient ClientCallback<ClientConnection> retry reconnects with a new UUID.randomUUID() call inside completed() — initial attempt creates ch_A before IOException — retry callback creates ch_B via fresh UUID in request-building step

The Undertow project ships UndertowClient — a low-level async HTTP/1.1 client built on XNIO non-blocking I/O. It is distinct from the embedded server APIs and used when a developer wants outbound HTTP from within an Undertow application without pulling in OkHttp, Apache HttpClient, or java.net.http.HttpClient. The client’s API is callback-based: UndertowClient.getInstance().connect(uri, worker, ssl, pool, options, callback) invokes a ClientCallback<ClientConnection> when the TCP connection is established (or its failed(IOException) method when the connection cannot be established). Inside completed(ClientConnection conn), the developer calls conn.sendRequest(ClientRequest, ClientCallback<ClientExchange>) to initiate the HTTP request. The ClientCallback<ClientExchange> receives the exchange when the request channel is ready, at which point the developer writes the request body and sets up the response receiver.

A developer who builds retry logic around this callback chain will call UndertowClient.getInstance().connect() again on failure — either from the outer ClientCallback<ClientConnection>.failed() when the TCP connection fails, or by reconnecting after a ClientCallback<ClientExchange>.failed() when the exchange-level operation fails after the connection succeeded. If the ClientRequest is constructed inside the ClientCallback<ClientConnection>.completed() method — the natural location in callback-style code, where the request is built immediately before it is sent — any UUID.randomUUID() call in the request-building step evaluates fresh per callback invocation. Because each reconnect spawns a new callback and the callback builds a new ClientRequest, the idempotency key changes per retry:

// UNSAFE: UUID.randomUUID() inside ClientCallback.completed().
// UndertowClient.connect() is called again on failure, spawning a new callback.
// Each new completed() call builds a new ClientRequest with a fresh UUID.
// Initial attempt: UUID_A → Stripe creates ch_A before IOException from exchange callback.
// Retry (second connect()): UUID_B → Stripe creates ch_B.

import io.undertow.client.ClientCallback;
import io.undertow.client.ClientConnection;
import io.undertow.client.ClientExchange;
import io.undertow.client.ClientRequest;
import io.undertow.client.UndertowClient;
import io.undertow.util.HttpString;
import io.undertow.util.Methods;
import org.xnio.OptionMap;

public class StripeBillingClient {

    private final XnioWorker worker;
    private final XnioSsl ssl;
    private final ByteBufferPool pool;

    // UNSAFE: calls connectAndCharge() on first attempt, retries via failed() callback.
    public void chargeWithRetry(String customerId, int amountCents,
                                String billingPeriod, int maxAttempts) {
        connectAndCharge(customerId, amountCents, billingPeriod, maxAttempts, 0);
    }

    private void connectAndCharge(String customerId, int amountCents,
                                  String billingPeriod, int maxAttempts, int attempt) {
        try {
            URI stripeUri = new URI("https://api.stripe.com");
            UndertowClient.getInstance().connect(
                stripeUri, worker, ssl, pool, OptionMap.EMPTY,
                new ClientCallback() {

                    @Override
                    public void completed(ClientConnection conn) {
                        // UNSAFE: ClientRequest built here — UUID.randomUUID() per callback.
                        // On attempt 0: UUID_A set on Idempotency-Key header.
                        //   Stripe receives POST /v1/charges — commits ch_A.
                        //   SocketTimeoutException fires before response arrives.
                        //   IOException propagates through exchange callback.
                        // On attempt 1 (this callback fires again for the new connection):
                        //   UUID_B = UUID.randomUUID().toString()  ← fresh UUID
                        //   Stripe sees a new Idempotency-Key — creates ch_B  ← DUPLICATE
                        ClientRequest request = new ClientRequest()
                            .setMethod(Methods.POST)
                            .setPath("/v1/charges");

                        // UNSAFE: UUID.randomUUID() here re-evaluates per completed() invocation.
                        String idempotencyKey = java.util.UUID.randomUUID().toString();

                        request.getRequestHeaders()
                            .put(new HttpString("Idempotency-Key"), idempotencyKey)
                            .put(new HttpString("Authorization"),
                                 "Bearer " + System.getenv("STRIPE_SECRET_KEY"))
                            .put(new HttpString("Content-Type"),
                                 "application/x-www-form-urlencoded");

                        conn.sendRequest(request, new ClientCallback() {
                            @Override
                            public void completed(ClientExchange exchange) {
                                // Write form-encoded body: amount, currency, customer
                                String body = "amount=" + amountCents
                                    + "¤cy=usd&customer=" + customerId;
                                exchange.getRequestChannel().write(
                                    ByteBuffer.wrap(body.getBytes(StandardCharsets.UTF_8)),
                                    new IoCallback() {
                                        @Override
                                        public void onComplete(HttpServerExchange ex,
                                                               Sender sender) {
                                            exchange.getRequestChannel().shutdownWrites();
                                            // Read response...
                                        }
                                        @Override
                                        public void onException(HttpServerExchange ex,
                                                                Sender sender, IOException e) {
                                            // IOException after request body written.
                                            // Stripe may have committed ch_A.
                                            // Retry by reconnecting — new callback → UUID_B.
                                            if (attempt < maxAttempts - 1) {
                                                connectAndCharge(customerId, amountCents,
                                                    billingPeriod, maxAttempts, attempt + 1);
                                            }
                                        }
                                    }
                                );
                            }
                            @Override
                            public void failed(IOException e) {
                                if (attempt < maxAttempts - 1) {
                                    connectAndCharge(customerId, amountCents,
                                        billingPeriod, maxAttempts, attempt + 1);
                                }
                            }
                        });
                    }

                    @Override
                    public void failed(IOException e) {
                        if (attempt < maxAttempts - 1) {
                            connectAndCharge(customerId, amountCents,
                                billingPeriod, maxAttempts, attempt + 1);
                        }
                    }
                }
            );
        } catch (URISyntaxException e) {
            throw new RuntimeException(e);
        }
    }
}

There is a subtler variant involving the exchange-level callback. Some developers separate the retry logic: they retry the full connect() chain on connection-level failures (TCP refused, SSL handshake timeout), but handle exchange-level failures by reusing the existing ClientConnection object and calling conn.sendRequest() again with a newly built ClientRequest. The thinking is: “the connection is still open, I’ll just resend the request.” This is valid from an Undertow API perspective — an HTTP/1.1 keep-alive connection can service multiple sequential requests. But if the developer builds the ClientRequest inside a helper method that calls UUID.randomUUID(), the helper is called twice: once for the initial sendRequest() and once for the retry sendRequest(). The connection is reused but the idempotency key changes. This variant is harder to spot in review because the re-used connection obscures the fact that a new outbound HTTP request is being created with a new header value:

// UNSAFE subtler variant: connection is reused, but buildRequest() is called per sendRequest().
// buildRequest() calls UUID.randomUUID() — fresh UUID per sendRequest() invocation.
// Initial sendRequest() → UUID_A → Stripe creates ch_A before IOException.
// Retry sendRequest() on same connection → UUID_B → Stripe creates ch_B.

private ClientRequest buildRequest(String customerId, int amountCents) {
    ClientRequest req = new ClientRequest()
        .setMethod(Methods.POST)
        .setPath("/v1/charges");
    // UNSAFE: UUID.randomUUID() in helper — re-evaluates per buildRequest() call.
    req.getRequestHeaders()
        .put(new HttpString("Idempotency-Key"),
             java.util.UUID.randomUUID().toString())  // ← new UUID each invocation
        .put(new HttpString("Authorization"),
             "Bearer " + System.getenv("STRIPE_SECRET_KEY"))
        .put(new HttpString("Content-Type"),
             "application/x-www-form-urlencoded");
    return req;
}

// In the exchange-level retry:
conn.sendRequest(buildRequest(customerId, amountCents),  // UUID_A
    new ClientCallback() {
        @Override
        public void failed(IOException e) {
            conn.sendRequest(buildRequest(customerId, amountCents),  // UUID_B ← DUPLICATE
                retryExchangeCallback);
        }
        ...
    }
);

The fix computes the stable idempotency key once before any connect() call and closes over it in all nested callbacks. The key is derived from stable billing fields via a content hash rather than from UUID.randomUUID(). Because the key is computed before the first asynchronous step and captured as an effectively-final local variable, every callback invocation in the chain — connection callback, exchange callback, body write callback, and any retry path — reads the same value:

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.HexFormat;

// Stable content-hash key: sha256(customerId:billingPeriod:undertow-billing)[:32].
// Computed once before the first connect() call, captured as final local variable.
// All nested callbacks close over the same key — no re-evaluation per callback invocation.
public class StableKeyHelper {
    public static String billingKey(String customerId, String billingPeriod, String tag) {
        try {
            byte[] hash = MessageDigest.getInstance("SHA-256")
                .digest((customerId + ":" + billingPeriod + ":" + tag)
                    .getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash).substring(0, 32);
        } catch (Exception e) {
            throw new RuntimeException("SHA-256 unavailable", e);
        }
    }
}

// SAFE: idempotencyKey computed before connect() — same value in all callbacks.
public void chargeWithRetry(String customerId, int amountCents,
                            String billingPeriod, int maxAttempts) {
    // Compute key once. final — captured by value in every lambda/anonymous class below.
    final String idempotencyKey = StableKeyHelper.billingKey(
        customerId, billingPeriod, "undertow-billing"
    );
    connectAndCharge(customerId, amountCents, billingPeriod,
                     idempotencyKey, maxAttempts, 0);
}

private void connectAndCharge(String customerId, int amountCents, String billingPeriod,
                               String idempotencyKey, int maxAttempts, int attempt) {
    try {
        URI stripeUri = new URI("https://api.stripe.com");
        UndertowClient.getInstance().connect(
            stripeUri, worker, ssl, pool, OptionMap.EMPTY,
            new ClientCallback() {

                @Override
                public void completed(ClientConnection conn) {
                    // SAFE: ClientRequest reads idempotencyKey from outer scope.
                    // idempotencyKey was computed before the first connect() call.
                    // On attempt 0 and all retry attempts, the same key is used.
                    // Stripe returns ch_A from its idempotency cache on retry — no ch_B.
                    ClientRequest request = new ClientRequest()
                        .setMethod(Methods.POST)
                        .setPath("/v1/charges");

                    // SAFE: idempotencyKey is the same stable value on every attempt.
                    request.getRequestHeaders()
                        .put(new HttpString("Idempotency-Key"), idempotencyKey)
                        .put(new HttpString("Authorization"),
                             "Bearer " + System.getenv("STRIPE_SECRET_KEY"))
                        .put(new HttpString("Content-Type"),
                             "application/x-www-form-urlencoded");

                    conn.sendRequest(request, new ClientCallback() {
                        @Override
                        public void completed(ClientExchange exchange) {
                            String body = "amount=" + amountCents
                                + "¤cy=usd&customer=" + customerId;
                            exchange.getRequestChannel().write(
                                ByteBuffer.wrap(body.getBytes(StandardCharsets.UTF_8)),
                                new IoCallback() {
                                    @Override
                                    public void onComplete(HttpServerExchange ex, Sender s) {
                                        exchange.getRequestChannel().shutdownWrites();
                                    }
                                    @Override
                                    public void onException(HttpServerExchange ex,
                                                            Sender s, IOException e) {
                                        if (attempt < maxAttempts - 1) {
                                            // SAFE: same idempotencyKey passed through.
                                            connectAndCharge(customerId, amountCents,
                                                billingPeriod, idempotencyKey,
                                                maxAttempts, attempt + 1);
                                        }
                                    }
                                }
                            );
                        }
                        @Override
                        public void failed(IOException e) {
                            if (attempt < maxAttempts - 1) {
                                connectAndCharge(customerId, amountCents,
                                    billingPeriod, idempotencyKey,
                                    maxAttempts, attempt + 1);
                            }
                        }
                    });
                }

                @Override
                public void failed(IOException e) {
                    if (attempt < maxAttempts - 1) {
                        connectAndCharge(customerId, amountCents,
                            billingPeriod, idempotencyKey,
                            maxAttempts, attempt + 1);
                    }
                }
            }
        );
    } catch (URISyntaxException e) {
        throw new RuntimeException(e);
    }
}

The key property that makes this safe is that idempotencyKey is computed exactly once — before the first asynchronous step — and passed as a method parameter through every recursive connectAndCharge() call. All lambda and anonymous class bodies close over the same string value. Whether the billing succeeds on attempt 0 or attempt 2, the idempotency key Stripe receives is identical. On any retry where ch_A is already in Stripe’s ledger, Stripe returns the cached ch_A response rather than creating ch_B.

Failure mode 2: HttpServerExchange.dispatch() worker-thread retry loop calls UUID.randomUUID() per loop iteration — initial iteration creates ch_A before SocketTimeoutException — second iteration creates ch_B via fresh UUID

When embedding Undertow directly as a standalone HTTP server (not via Spring Boot’s embedded container integration), an application registers HttpHandler implementations that receive incoming HTTP requests. The handleRequest(HttpServerExchange exchange) method is invoked on the XNIO I/O thread. XNIO I/O threads handle non-blocking I/O events and must not perform any blocking operation — blocking the I/O thread stalls all other connections assigned to that thread’s event loop. Any operation that blocks (a synchronous Stripe HTTP call using java.net.http.HttpClient, OkHttpClient, or Apache CloseableHttpClient) must be dispatched to the XNIO worker thread pool.

The dispatch pattern in Undertow is: call exchange.isInIoThread() at the top of handleRequest(); if true, call exchange.dispatch(this) and return; Undertow puts the handler back onto the worker thread pool and calls handleRequest(exchange) again, this time from a worker thread where isInIoThread() returns false and blocking I/O is permitted. A developer who adds a manual retry loop inside the worker-thread portion of handleRequest() may place UUID.randomUUID() at the top of the loop body — the same position they would have placed it for a single-attempt handler before retry logic was added. Each iteration of the loop evaluates a new UUID, so the second iteration produces UUID_B:

// UNSAFE: UUID.randomUUID() inside the retry loop body.
// Second loop iteration generates UUID_B → Stripe creates ch_B.
// ch_A is already committed from the first iteration if a SocketTimeoutException fired
// after Stripe processed the request but before the response arrived.

import io.undertow.server.HttpHandler;
import io.undertow.server.HttpServerExchange;

public class BillingHandler implements HttpHandler {

    private final CloseableHttpClient httpClient;  // Apache HttpClient for blocking HTTP

    public BillingHandler(CloseableHttpClient httpClient) {
        this.httpClient = httpClient;
    }

    @Override
    public void handleRequest(HttpServerExchange exchange) throws Exception {
        if (exchange.isInIoThread()) {
            // Dispatch to worker thread — handleRequest() will be called again
            // on a XNIO worker thread where blocking I/O is permitted.
            exchange.dispatch(this);
            return;
        }

        // On worker thread now. Parse billing parameters from request.
        String customerId = getQueryParam(exchange, "customer_id");
        int amountCents = Integer.parseInt(getQueryParam(exchange, "amount"));
        String billingPeriod = getQueryParam(exchange, "billing_period");

        int maxAttempts = 3;
        ChargeResponse result = null;
        Exception lastException = null;

        for (int attempt = 0; attempt < maxAttempts; attempt++) {
            // UNSAFE: UUID computed at top of loop body — new UUID per iteration.
            // On attempt 0: UUID_A → Stripe commits ch_A before socket timeout.
            // SocketTimeoutException propagates, loop continues.
            // On attempt 1: UUID_B = UUID.randomUUID()  ← new UUID
            // Stripe creates ch_B alongside committed ch_A.
            String idempotencyKey = java.util.UUID.randomUUID().toString();

            try {
                HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
                post.setHeader("Authorization",
                               "Bearer " + System.getenv("STRIPE_SECRET_KEY"));
                post.setHeader("Idempotency-Key", idempotencyKey);
                post.setEntity(new UrlEncodedFormEntity(List.of(
                    new BasicNameValuePair("amount", String.valueOf(amountCents)),
                    new BasicNameValuePair("currency", "usd"),
                    new BasicNameValuePair("customer", customerId)
                )));

                try (CloseableHttpResponse resp = httpClient.execute(post)) {
                    int status = resp.getCode();
                    if (status == 200 || status == 201) {
                        result = parseChargeResponse(resp);
                        break;
                    } else if (status >= 500) {
                        // Transient server error — retry with fresh UUID (UNSAFE).
                        lastException = new IOException("Stripe " + status);
                        continue;
                    } else {
                        // Non-retryable error (4xx).
                        break;
                    }
                }
            } catch (SocketTimeoutException e) {
                // Timeout after Stripe may have committed the charge.
                // Loop continues to attempt + 1 with UUID_B.
                lastException = e;
            }
        }

        sendResponse(exchange, result, lastException);
    }
}

There is a subtler variant involving an attempt-index suffix appended to the UUID. A developer who understands that idempotency keys should be consistent across retries for the same logical charge may add a suffix to make each attempt unique — under the mistaken belief that Stripe requires a distinct key per attempt or that including the attempt count in the key helps with audit logging. The suffix produces a structurally distinct key per attempt by construction, guaranteeing ch_B on iteration 2 regardless of whether the base UUID is stable:

// UNSAFE subtler variant: UUID stable but attempt index appended — structurally distinct key
// per attempt by construction. UUID_A-attempt-0 on iteration 0 → Stripe creates ch_A.
// UUID_A-attempt-1 on iteration 1 → Stripe sees new key → creates ch_B.
// Developer intended this for audit logging. Stripe's idempotency cache key is the entire
// Idempotency-Key header value including the suffix — "uuid-attempt-0" and "uuid-attempt-1"
// are different keys and Stripe treats them as two separate charge requests.

// BEFORE the loop (one UUID per request — correct intent):
String baseKey = java.util.UUID.randomUUID().toString(); // stable per handler invocation

for (int attempt = 0; attempt < maxAttempts; attempt++) {
    // UNSAFE: attempt suffix makes this key structurally distinct per attempt.
    // Stripe sees "6a3f...c2d1-attempt-0" on attempt 0 and "6a3f...c2d1-attempt-1"
    // on attempt 1 — two different charge requests, two charges.
    String idempotencyKey = baseKey + "-attempt-" + attempt;

    // ... POST to Stripe with this key
}

The correct pattern uses Undertow’s AttachmentKey API to carry the stable idempotency key from before the dispatch boundary into the worker-thread handler. AttachmentKey is Undertow’s typed key-value store on the HttpServerExchange object — attachments survive the exchange.dispatch() call and are readable from the worker thread. The key is computed on the initial I/O-thread invocation (before exchange.dispatch(this)) and attached to the exchange. On the worker-thread re-invocation, the handler reads the key from the attachment and uses it for all retry iterations:

import io.undertow.util.AttachmentKey;

// SAFE: AttachmentKey carries stable idempotency key from I/O thread to worker thread.
// Key computed once before dispatch() — same value in all loop iterations.
public class BillingHandler implements HttpHandler {

    // AttachmentKey typed as String — one key per exchange instance.
    private static final AttachmentKey<String> IDEMPOTENCY_KEY_ATTACHMENT =
        AttachmentKey.create(String.class);

    private final CloseableHttpClient httpClient;

    public BillingHandler(CloseableHttpClient httpClient) {
        this.httpClient = httpClient;
    }

    @Override
    public void handleRequest(HttpServerExchange exchange) throws Exception {
        if (exchange.isInIoThread()) {
            // Parse parameters and compute stable key before dispatch.
            // The key is attached to the exchange — survives the dispatch() boundary.
            String customerId = getQueryParam(exchange, "customer_id");
            String billingPeriod = getQueryParam(exchange, "billing_period");

            // SAFE: key computed once, on I/O thread, before any retry attempt.
            String idempotencyKey = StableKeyHelper.billingKey(
                customerId, billingPeriod, "undertow-billing"
            );
            exchange.putAttachment(IDEMPOTENCY_KEY_ATTACHMENT, idempotencyKey);

            exchange.dispatch(this);
            return;
        }

        // On worker thread. Retrieve stable key from attachment.
        // This is the same value that was computed and attached on the I/O thread.
        String idempotencyKey = exchange.getAttachment(IDEMPOTENCY_KEY_ATTACHMENT);
        String customerId = getQueryParam(exchange, "customer_id");
        int amountCents = Integer.parseInt(getQueryParam(exchange, "amount"));

        int maxAttempts = 3;
        ChargeResponse result = null;
        Exception lastException = null;

        for (int attempt = 0; attempt < maxAttempts; attempt++) {
            // SAFE: idempotencyKey read from attachment — same value on all iterations.
            // If ch_A was committed on attempt 0 before SocketTimeoutException,
            // attempt 1 sends the same key and Stripe returns ch_A from its cache.
            // No ch_B created.
            try {
                HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
                post.setHeader("Authorization",
                               "Bearer " + System.getenv("STRIPE_SECRET_KEY"));
                post.setHeader("Idempotency-Key", idempotencyKey);
                post.setEntity(new UrlEncodedFormEntity(List.of(
                    new BasicNameValuePair("amount", String.valueOf(amountCents)),
                    new BasicNameValuePair("currency", "usd"),
                    new BasicNameValuePair("customer", customerId)
                )));

                try (CloseableHttpResponse resp = httpClient.execute(post)) {
                    int status = resp.getCode();
                    if (status == 200 || status == 201) {
                        result = parseChargeResponse(resp);
                        break;
                    } else if (status >= 500) {
                        lastException = new IOException("Stripe " + status);
                        // Wait before retry — but key stays the same.
                        Thread.sleep(1000L * (attempt + 1));
                        continue;
                    } else {
                        break;
                    }
                }
            } catch (SocketTimeoutException e) {
                lastException = e;
                // No sleep needed for timeout — Stripe has already committed or not.
                // The same key on the next attempt will surface the cached result.
            }
        }

        sendResponse(exchange, result, lastException);
    }
}

The AttachmentKey pattern is the idiomatic Undertow approach for passing typed state across the dispatch boundary. It avoids query-parameter re-parsing on the worker thread (parsing on the I/O thread is fine since it is a non-blocking string operation), and it guarantees that the idempotency key is computed exactly once per incoming HTTP request regardless of how many retry iterations the worker-thread handler executes. The stable key property holds because StableKeyHelper.billingKey() is deterministic: the same (customerId, billingPeriod, tag) input always produces the same 32-character hex output.

Failure mode 3: XnioIoThread.executeAfter() delay-based recursive retry calls UUID.randomUUID() at recursive method entry — initial call creates ch_A before IOException — first executeAfter() callback creates ch_B via fresh UUID

Undertow’s underlying I/O layer is XNIO (Extensible Non-blocking I/O). The XNIO worker infrastructure provides scheduling beyond what exchange.dispatch() offers: XnioIoThread (accessible via exchange.getIoThread()) implements XnioExecutor, which exposes executeAfter(Runnable task, long time, TimeUnit unit) for running a task after a specified delay on the I/O thread. In a minimal embedded Undertow application that wants exponential-backoff retry for outbound Stripe calls, a developer might reach for executeAfter() to schedule delayed retries without importing a ScheduledExecutorService or adding a dependency on a scheduler framework. The delay scheduling is done on the I/O thread (the executeAfter() registration), but the task execution is on the I/O thread as well — which creates a separate problem if the task is blocking. Setting that aside for the idempotency issue: a recursive billing helper method that accepts an attempt counter, calls UUID.randomUUID() at its entry, attempts the Stripe call, and on failure schedules itself via executeAfter() evaluates a new UUID on every scheduled invocation.

// UNSAFE: UUID.randomUUID() at top of recursive retry method.
// executeAfter() callback re-enters scheduleRetry() — UUID.randomUUID() re-evaluates.
// Initial call (attempt 0): UUID_A → Stripe creates ch_A before IOException.
// executeAfter() callback (attempt 1): UUID_B = UUID.randomUUID()  ← fresh UUID
// Stripe creates ch_B alongside committed ch_A.

import io.undertow.server.HttpHandler;
import io.undertow.server.HttpServerExchange;
import org.xnio.XnioExecutor;

public class BillingHandlerWithBackoff implements HttpHandler {

    private final OkHttpClient httpClient;

    @Override
    public void handleRequest(HttpServerExchange exchange) throws Exception {
        if (exchange.isInIoThread()) {
            exchange.dispatch(this);
            return;
        }

        String customerId = getQueryParam(exchange, "customer_id");
        int amountCents = Integer.parseInt(getQueryParam(exchange, "amount"));
        String billingPeriod = getQueryParam(exchange, "billing_period");

        // Start first attempt.
        scheduleRetry(exchange, customerId, amountCents, billingPeriod, 0, 3);
    }

    private void scheduleRetry(HttpServerExchange exchange,
                                String customerId, int amountCents,
                                String billingPeriod, int attempt, int maxAttempts) {
        // UNSAFE: UUID.randomUUID() at method entry — new UUID per scheduleRetry() call.
        // executeAfter() re-enters this method → UUID_B on attempt 1.
        String idempotencyKey = java.util.UUID.randomUUID().toString();

        try {
            Request request = new Request.Builder()
                .url("https://api.stripe.com/v1/charges")
                .header("Authorization", "Bearer " + System.getenv("STRIPE_SECRET_KEY"))
                .header("Idempotency-Key", idempotencyKey)
                .post(RequestBody.create(
                    "amount=" + amountCents + "¤cy=usd&customer=" + customerId,
                    MediaType.parse("application/x-www-form-urlencoded")
                ))
                .build();

            try (Response resp = httpClient.newCall(request).execute()) {
                if (resp.isSuccessful()) {
                    sendSuccess(exchange, resp.body().string());
                } else if (resp.code() >= 500 && attempt < maxAttempts - 1) {
                    long delay = (long) Math.pow(2, attempt) * 1000L;
                    // UNSAFE: re-enters scheduleRetry() → UUID.randomUUID() again → UUID_B.
                    exchange.getIoThread().executeAfter(
                        () -> scheduleRetry(exchange, customerId, amountCents,
                                           billingPeriod, attempt + 1, maxAttempts),
                        delay,
                        TimeUnit.MILLISECONDS
                    );
                } else {
                    sendError(exchange, resp.code());
                }
            }
        } catch (IOException e) {
            if (attempt < maxAttempts - 1) {
                long delay = (long) Math.pow(2, attempt) * 1000L;
                // UNSAFE: IOException fired after ch_A committed — Stripe committed the charge
                // but the response didn't arrive. executeAfter() re-enters scheduleRetry()
                // with attempt + 1. UUID_B = UUID.randomUUID() at method entry → ch_B.
                exchange.getIoThread().executeAfter(
                    () -> scheduleRetry(exchange, customerId, amountCents,
                                       billingPeriod, attempt + 1, maxAttempts),
                    delay,
                    TimeUnit.MILLISECONDS
                );
            } else {
                sendError(exchange, 503);
            }
        }
    }
}

There is a subtler variant involving an AtomicReference that the developer adds to “fix” the UUID issue. The developer recognizes that UUID.randomUUID() inside the recursive method re-evaluates per call and moves the UUID generation to an AtomicReference<String>. The intent is to generate the UUID once and store it in the reference. But the AtomicReference is declared as a local variable inside handleRequest(), and the scheduleRetry() method is called with the reference as a parameter. The AtomicReference.compareAndSet(null, UUID.randomUUID().toString()) pattern inside scheduleRetry() is supposed to set the key only once (when the reference is null). But AtomicReference.compareAndSet(null, ...) still evaluates the UUID.randomUUID().toString() argument before calling compareAndSet — in Java, method arguments are evaluated before the method is called. So even if compareAndSet returns false (key already set), a fresh UUID.randomUUID() was generated and discarded. Worse: if the developer uses the compareAndSet result to decide whether to continue (assuming a failed CAS means the billing already ran), the logic is inverted:

// UNSAFE subtler variant: AtomicReference.compareAndSet() evaluates UUID.randomUUID()
// argument before the CAS — UUID is generated even when compareAndSet returns false.
// This does NOT cause the duplicate charge directly (the stable reference value is used
// for the Stripe header), but reveals that the developer's mental model of lazy evaluation
// is wrong. A worse variant would be setting it directly: ref.set(UUID.randomUUID().toString())
// on every attempt.

// Inside scheduleRetry() — UNSAFE pattern:
private void scheduleRetry(HttpServerExchange exchange, AtomicReference<String> keyRef,
                            String customerId, int amountCents,
                            String billingPeriod, int attempt, int maxAttempts) {
    // UNSAFE: UUID.randomUUID().toString() is evaluated as an argument before compareAndSet()
    // is called. On attempt 0, compareAndSet succeeds (null → UUID_A). UUID_A is stored.
    // On attempt 1 (executeAfter() callback), compareAndSet fails (keyRef already UUID_A).
    // UUID_B was generated as the argument — it is discarded. The Stripe call uses UUID_A.
    // This accidentally works correctly for the idempotency key, but a developer who changes
    // keyRef.compareAndSet() to keyRef.set() (to "simplify") creates the duplicate charge:
    keyRef.compareAndSet(null, java.util.UUID.randomUUID().toString());
    // If developer replaces with:
    // keyRef.set(java.util.UUID.randomUUID().toString());  ← UUID_B set on retry → ch_B

    String idempotencyKey = keyRef.get();
    // ... use idempotencyKey for Stripe call
}

The fix threads the stable idempotency key as a parameter through the recursive scheduleRetry() calls, computed once before the first invocation. The key is computed in handleRequest() on the worker thread (after the I/O-thread dispatch), stored in an AttachmentKey for audit purposes, and passed as a method parameter to scheduleRetry(). All executeAfter() callbacks pass the same key value to the next recursive call. There is no mutation of the key across the retry chain:

// SAFE: idempotencyKey computed once in handleRequest() — passed as parameter through
// all recursive scheduleRetry() calls. executeAfter() lambda closes over the
// attempt-incremented counter but reads the key from the parameter — same value always.

@Override
public void handleRequest(HttpServerExchange exchange) throws Exception {
    if (exchange.isInIoThread()) {
        exchange.dispatch(this);
        return;
    }

    String customerId = getQueryParam(exchange, "customer_id");
    int amountCents = Integer.parseInt(getQueryParam(exchange, "amount"));
    String billingPeriod = getQueryParam(exchange, "billing_period");

    // SAFE: key computed once on worker thread before first scheduleRetry() call.
    // Deterministic hash — same (customerId, billingPeriod) always produces same key.
    final String idempotencyKey = StableKeyHelper.billingKey(
        customerId, billingPeriod, "undertow-billing"
    );

    scheduleRetry(exchange, customerId, amountCents, billingPeriod,
                  idempotencyKey, 0, 3);
}

private void scheduleRetry(HttpServerExchange exchange,
                            String customerId, int amountCents,
                            String billingPeriod,
                            String idempotencyKey,  // same value on every recursion
                            int attempt, int maxAttempts) {
    // SAFE: idempotencyKey is a parameter — same value on every call.
    // UUID.randomUUID() is NOT called here. Key is stable across all retries.
    Request request = new Request.Builder()
        .url("https://api.stripe.com/v1/charges")
        .header("Authorization", "Bearer " + System.getenv("STRIPE_SECRET_KEY"))
        .header("Idempotency-Key", idempotencyKey)
        .post(RequestBody.create(
            "amount=" + amountCents + "¤cy=usd&customer=" + customerId,
            MediaType.parse("application/x-www-form-urlencoded")
        ))
        .build();

    try {
        try (Response resp = httpClient.newCall(request).execute()) {
            if (resp.isSuccessful()) {
                sendSuccess(exchange, resp.body().string());
                return;
            }
            if (resp.code() >= 500 && attempt < maxAttempts - 1) {
                long delay = (long) Math.pow(2, attempt) * 1000L;
                // SAFE: idempotencyKey passed through — same value in next recursion.
                exchange.getIoThread().executeAfter(
                    () -> scheduleRetry(exchange, customerId, amountCents,
                                       billingPeriod, idempotencyKey,
                                       attempt + 1, maxAttempts),
                    delay,
                    TimeUnit.MILLISECONDS
                );
                return;
            }
            sendError(exchange, resp.code());
        }
    } catch (IOException e) {
        if (attempt < maxAttempts - 1) {
            long delay = (long) Math.pow(2, attempt) * 1000L;
            // SAFE: idempotencyKey unchanged. If ch_A was committed before IOException,
            // next attempt sends same key — Stripe returns ch_A from cache — no ch_B.
            exchange.getIoThread().executeAfter(
                () -> scheduleRetry(exchange, customerId, amountCents,
                                    billingPeriod, idempotencyKey,
                                    attempt + 1, maxAttempts),
                delay,
                TimeUnit.MILLISECONDS
            );
        } else {
            sendError(exchange, 503);
        }
    }
}

One operational note about executeAfter() in this context: the delayed retry task runs on the XNIO I/O thread, not on the worker thread pool. If scheduleRetry() makes a blocking HTTP call (via OkHttp, Apache HttpClient, or java.net.HttpURLConnection), the blocking call runs on the I/O thread during the executeAfter() execution. This blocks the I/O event loop for the duration of the HTTP call, which can stall other connections on the same XNIO worker thread. For production use, the executeAfter() callback should dispatch back to the worker thread pool using worker.execute(blockingRetryRunnable) before making the blocking Stripe call. The idempotency key correctness issue and the I/O-thread blocking issue are independent — the fix above addresses the idempotency key; the worker-thread dispatch should be added in production code as a separate concern.

Keybrake: spend caps as the financial backstop for all three failure modes

Stable content-hash idempotency keys, Undertow’s AttachmentKey for carrying state across dispatch boundaries, and parameter threading through recursive retry calls provide defense-in-depth at the application layer. They address each of the three failure modes under the specific conditions described. They do not protect against cases where a developer reverts the fix (adding UUID.randomUUID() back inside a callback, loop, or recursive method), future library upgrades that change callback invocation semantics, or failure modes that arise from Undertow configurations not covered by any single code review.

A per-billing-period vault key with a spend cap at expected_total_charges × 1.10 limits the blast radius independent of the application layer. The vault key is scoped to the Stripe create-charge endpoint and the current billing period. If any of the three failure modes produces ch_B alongside ch_A for even one customer, the spend cap absorbs the overage but blocks a runaway loop from charging every customer twice. The audit log records every charge by idempotency key, vault key, and policy verdict — post-run reconciliation against Stripe’s event stream becomes a lookup rather than a manual investigation.

The vault key also provides a backstop for a failure mode unique to standalone embedded Undertow applications in multi-instance deployments: each JVM instance runs its own Undertow.builder().addHttpListener().setHandler() server with its own billing handler. If a billing endpoint is triggered via an external message queue (Kafka, SQS, RabbitMQ) and the queue delivers the same billing event to all running instances before any instance acknowledges, each instance executes the billing handler independently. Even with stable content-hash idempotency keys, two instances that execute the billing handler for the same customer in the same billing period within the Stripe idempotency cache write window (roughly the first few hundred milliseconds of the first request) may both create their charges before either response arrives in the other’s cache. The pre-flight INSERT ... ON CONFLICT DO NOTHING guard on (customer_id, billing_period) is the cluster-wide serialization mechanism — only one instance wins the INSERT and proceeds to call Stripe; the others see zero rows inserted and skip. The spend cap adds a financial ceiling on top of this database guard.

Summary

The common thread across all three Undertow failure modes is the same pattern seen in the Netty and Armeria posts: low-level async frameworks with callback-based APIs naturally lead developers to compute per-request state inside callback bodies or at the entry points of methods that are invoked per callback. UUID.randomUUID() feels like “per-request” computation because it is called in a per-request code path. The critical distinction is between per logical billing request (the idempotency key must be stable across all network-level retries for the same logical charge) and per network request (where each callback or method entry corresponds to one network attempt). Content-hash keys derived from stable billing fields eliminate the ambiguity: the key is the same value whether the billing succeeds on the first callback invocation, the third executeAfter() callback, or the fifth loop iteration, because it is derived deterministically from inputs that do not change across retries.