Armeria and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Armeria’s RetryingClient decorator re-executes the decorator chain downstream of it on each retry attempt — a SimpleDecoratingHttpClient placed between RetryingClient and the base transport that calls UUID.randomUUID() to generate the Idempotency-Key header fires on the initial request and on every retry: the initial POST to Stripe creates ch_A before a transient ResponseTimeoutException, and the first retry’s decorator invocation produces a fresh UUID so Stripe creates ch_B. Three Armeria-specific Stripe billing failure modes: a SimpleDecoratingHttpClient inside RetryingClient computes UUID.randomUUID() per retry; an async billing path uses a recursive CompletableFuture retry callback that calls buildBillingRequest() with a fresh UUID per invocation; and Armeria’s per-JVM ScheduledExecutorService fires the billing job independently on every Kubernetes replica with no cross-pod coordination — three pods pass a concurrent database check, generate distinct UUID.randomUUID() values per customer, and create ch_A, ch_B, and ch_C per billing period.

This post covers all three failure modes with Java code, content-hash idempotency keys that are stable across SimpleDecoratingHttpClient re-invocations, recursive async retry callback re-entries, and multi-pod concurrent billing loops, pg_try_advisory_lock() for cross-pod scheduler serialization, pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For the AOP proxy retry pattern in Spring Boot (@Retryable), see the Spring Boot and Stripe Integration post. For the interceptor chain retry pattern in Dropwizard (Apache HttpClient), see the Dropwizard and Stripe Integration post. For async Akka HTTP retry failure modes, see the Akka HTTP and Stripe Integration post.

Failure mode 1: SimpleDecoratingHttpClient placed between RetryingClient and the base transport computes UUID.randomUUID() per execute() invocation — RetryingClient re-invokes the inner decorator chain on each retry — initial attempt created ch_A before ResponseTimeoutException — first retry’s decorator invocation creates ch_B

Armeria builds its HTTP client as a chain of decorators. Each decorator is a thin wrapper that intercepts HttpRequest / HttpResponse objects before they reach the next layer. RetryingClient is one such decorator: it wraps a delegate client and, when the delegate’s response indicates a retryable failure (5xx status, connection error, timeout), it discards the failed response and re-invokes the delegate with the same or a new request. Critically, every decorator registered inside RetryingClient — closer to the base transport than RetryingClient itself — is re-invoked on each retry attempt. A developer who registers a SimpleDecoratingHttpClient to automatically inject headers (a clean separation-of-concerns pattern) and places it inside the RetryingClient boundary sees the decorator fire on the initial request and on every retry. If the decorator computes UUID.randomUUID() per execute() call to generate a fresh idempotency key, each retry carries a different value:

// IdempotencyKeyDecorator.java
// UNSAFE: UUID.randomUUID() computed inside SimpleDecoratingHttpClient.execute().
// If this decorator is registered INSIDE RetryingClient in the decorator chain,
// RetryingClient re-invokes execute() on each retry — UUID.randomUUID() fires again.

import com.linecorp.armeria.client.DecoratingHttpClientFunction;
import com.linecorp.armeria.client.HttpClient;
import com.linecorp.armeria.client.SimpleDecoratingHttpClient;
import com.linecorp.armeria.common.HttpRequest;
import com.linecorp.armeria.common.HttpResponse;

import java.util.UUID;

public class IdempotencyKeyDecorator extends SimpleDecoratingHttpClient {

    public IdempotencyKeyDecorator(HttpClient delegate) {
        super(delegate);
    }

    @Override
    public HttpResponse execute(com.linecorp.armeria.client.ClientRequestContext ctx,
                                HttpRequest req) throws Exception {
        // UNSAFE: called on the initial request AND on every RetryingClient retry.
        // Initial attempt:  UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Retry attempt 1:  UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        // Retry attempt 2:  UUID = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"
        String idempotencyKey = UUID.randomUUID().toString();

        HttpRequest decorated = req.withHeaders(
                req.headers().toBuilder()
                   .set("Idempotency-Key", idempotencyKey)
                   .build());

        return delegate().execute(ctx, decorated);
    }

    public static DecoratingHttpClientFunction newDecorator() {
        return (delegate, ctx, req) -> new IdempotencyKeyDecorator(delegate).execute(ctx, req);
    }
}

// UNSAFE registration — IdempotencyKeyDecorator is INSIDE RetryingClient:
// RetryingClient wraps the client that already has IdempotencyKeyDecorator.
// Each retry by RetryingClient passes through IdempotencyKeyDecorator.execute() again.

WebClient client = WebClient.builder("https://api.stripe.com")
    .decorator(IdempotencyKeyDecorator.newDecorator())   // 1st: inner decorator
    .decorator(RetryingClient.newDecorator(             // 2nd: outer decorator — wraps inner
            RetryRule.builder()
                     .onServerErrorStatus()
                     .onException(ResponseTimeoutException.class)
                     .thenRetry()))
    .responseTimeoutMillis(15_000)
    .build();

The failure scenario: an agent calls billingService.chargeCustomer("cust_123", "2026-09", 9900L). Armeria dispatches the HttpRequest through the decorator chain. RetryingClient passes it to its delegate. IdempotencyKeyDecorator.execute() fires. UUID.randomUUID() returns "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e". The decorated request reaches the base HttpClient and is sent to Stripe.

Stripe receives the POST /v1/charges. The card is authorized and the charge object ch_A is committed to Stripe’s ledger. Stripe begins writing the HTTP response. Before the response bytes are fully flushed, Armeria’s responseTimeoutMillis fires a ResponseTimeoutException. RetryingClient catches it, consults the RetryRule (which includes onException(ResponseTimeoutException.class)), and schedules a retry. For the retry, RetryingClient re-invokes its delegate chain. IdempotencyKeyDecorator.execute() fires again. UUID.randomUUID() evaluates — a completely independent call returning "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d". The retry reaches Stripe with a different Idempotency-Key.

Stripe has ch_A cached against "3f7a9b2c...". The retry arrives with "b8d2e4f6..." — a key Stripe has not seen before. Stripe processes the retry as a fresh charge request. ch_B is created. Customer 123 is charged $99 twice for September 2026.

This failure mode is especially treacherous in Armeria because the decorator pattern is idiomatic Armeria style — the framework is explicitly designed around composable decorators. The IdempotencyKeyDecorator looks clean, is registered in one place, and correctly handles stateless headers like Authorization (where regenerating per invocation is fine). The UUID case is structurally different: it must carry the same value across all retry attempts for a single billing operation. The decorator model provides no mechanism to signal “this is a retry for request X” vs. “this is a fresh request” without reading from shared state anchored to the original request.

The subtler variant: WebClient.prepare().header("Idempotency-Key", UUID.randomUUID()) called inside a helper method invoked per retry attempt

Not all Armeria retry code uses the RetryingClient decorator. Some teams write manual retry loops — particularly for cases requiring custom backoff or circuit-breaker integration — and call WebClient.prepare() directly inside the retry iteration. If the prepare()...header(...)...execute() chain is called inside a helper method, and if UUID.randomUUID() is called at the point where the Idempotency-Key header is set, each call to the helper produces a new UUID:

// UNSAFE: buildBillingRequest() called per retry attempt inside a loop.
// WebClient.prepare().header(..., UUID.randomUUID()) evaluates UUID at call time.
// Each iteration's buildBillingRequest() invocation produces a new UUID.

private CompletableFuture<AggregatedHttpResponse> sendWithRetry(
        WebClient stripeClient, String customerId, String billingPeriod, long amountCents) {

    for (int attempt = 0; attempt < 3; attempt++) {
        try {
            // UNSAFE: buildBillingRequest() computes UUID.randomUUID() at method entry.
            // Called 3 times over 3 loop iterations — 3 distinct UUIDs.
            AggregatedHttpResponse response = stripeClient
                    .prepare()
                    .post("/v1/charges")
                    .header("Idempotency-Key", UUID.randomUUID().toString())  // UNSAFE
                    .header("Authorization", "Bearer " + stripeKey)
                    .content(MediaType.FORM_DATA, buildBody(customerId, amountCents))
                    .execute()
                    .aggregate()
                    .join();

            if (response.status().isSuccess()) return CompletableFuture.completedFuture(response);
        } catch (Exception e) {
            if (attempt == 2) throw e;
        }
    }
    throw new IllegalStateException("unreachable");
}

The loop calls UUID.randomUUID() on every iteration. If attempt 0 reaches Stripe and creates ch_A before the connection times out, attempt 1 re-enters the loop body, evaluates UUID.randomUUID() again, and sends a request with a different key. Stripe sees a new idempotency key and creates ch_B. This is the same failure mode as FM1 in a different surface: the retry boundary is the loop iteration, not the decorator invocation, but the effect is identical.

The fix for failure mode 1

The idempotency key must be computed once per billing operation — before the request enters any retry boundary — and carried through all retry attempts without modification. For the decorator pattern, compute the key in the calling code and pass it as a ClientRequestContext attribute that the decorator reads:

// Safe: stableKey computed BEFORE the request enters the decorator chain.
// IdempotencyKeyDecorator reads the key from ClientRequestContext — does not generate it.
// RetryingClient retries re-invoke the decorator, which reads the same attribute value.

import com.linecorp.armeria.client.ClientRequestContext;
import com.linecorp.armeria.client.SimpleDecoratingHttpClient;
import com.linecorp.armeria.common.AttributeKey;
import com.linecorp.armeria.common.HttpRequest;
import com.linecorp.armeria.common.HttpResponse;

public class IdempotencyKeyDecorator extends SimpleDecoratingHttpClient {

    static final AttributeKey<String> IDEMPOTENCY_KEY =
            AttributeKey.valueOf(IdempotencyKeyDecorator.class, "IDEMPOTENCY_KEY");

    public IdempotencyKeyDecorator(HttpClient delegate) {
        super(delegate);
    }

    @Override
    public HttpResponse execute(ClientRequestContext ctx, HttpRequest req) throws Exception {
        // Safe: reads key from context — does not call UUID.randomUUID().
        // Same ClientRequestContext is reused across RetryingClient retry attempts
        // (RetryingClient propagates the parent context's attributes to child contexts).
        String idempotencyKey = ctx.attr(IDEMPOTENCY_KEY);
        if (idempotencyKey == null) {
            // Fallback: key not pre-set, do not generate a random one.
            // Log a warning and reject rather than silently producing ch_B.
            throw new IllegalStateException(
                    "Stripe call missing pre-computed idempotency key in context");
        }

        HttpRequest decorated = req.withHeaders(
                req.headers().toBuilder()
                   .set("Idempotency-Key", idempotencyKey)
                   .build());

        return delegate().execute(ctx, decorated);
    }
}

// Safe calling code: compute stableKey BEFORE executing the request.
public class BillingService {

    private final WebClient stripeClient;

    public AggregatedHttpResponse chargeCustomer(
            String customerId, String billingPeriod, long amountCents) throws Exception {

        // Computed once per billing operation — deterministic, same on every retry.
        String idempotencyKey = stableKey(customerId, billingPeriod);

        // Pre-flight: claim the billing slot before sending to Stripe.
        boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
        if (!claimed) {
            return billingRepository.findExistingCharge(customerId, billingPeriod);
        }

        try (ClientRequestContext ctx = ClientRequestContext.of(
                HttpRequest.of(RequestHeaders.of(HttpMethod.POST, "/v1/charges")))) {
            ctx.setAttr(IdempotencyKeyDecorator.IDEMPOTENCY_KEY, idempotencyKey);

            return stripeClient
                    .prepare()
                    .post("/v1/charges")
                    .content(MediaType.FORM_DATA, buildBody(customerId, amountCents))
                    .execute()
                    .aggregate()
                    .join();
        }
    }

    // For simple cases without context propagation — pass key as header in calling code:
    public AggregatedHttpResponse chargeCustomerDirect(
            String customerId, String billingPeriod, long amountCents) throws Exception {

        String idempotencyKey = stableKey(customerId, billingPeriod);

        // Pre-flight guard.
        if (!billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)) {
            return billingRepository.findExistingCharge(customerId, billingPeriod);
        }

        // Key set before the RetryingClient boundary — same value on every retry.
        return stripeClient
                .prepare()
                .post("/v1/charges")
                .header("Idempotency-Key", idempotencyKey)  // stable, pre-computed
                .header("Authorization", "Bearer " + stripeKey)
                .content(MediaType.FORM_DATA, buildBody(customerId, amountCents))
                .execute()
                .aggregate()
                .join();
    }

    static String stableKey(String customerId, String billingPeriod) {
        try {
            var digest = java.security.MessageDigest.getInstance("SHA-256");
            var hash = digest.digest(
                    (customerId + ":" + billingPeriod + ":armeria-billing")
                            .getBytes(java.nio.charset.StandardCharsets.UTF_8));
            var sb = new StringBuilder(32);
            for (int i = 0; i < 16; i++) sb.append(String.format("%02x", hash[i]));
            return sb.toString();
        } catch (java.security.NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
    }
}

With the key computed before the execute() call, every RetryingClient retry passes the same header value to Stripe. Stripe caches the response for ch_A against the stable key and returns it on all subsequent retries without creating ch_B. The pre-flight claimSlot() write ensures durability: if the process crashes between the claimSlot write and the Stripe call, the next startup finds the slot claimed with the same stable key and resumes rather than starting a fresh charge.

Failure mode 2: recursive CompletableFuture retry callback calls buildBillingRequest() with UUID.randomUUID() at method entry — initial CompletableFuture created ch_A before ResponseTimeoutException — exceptionally() re-invokes buildBillingRequest() — fresh UUID causes ch_B

Armeria’s WebClient.execute() returns an HttpResponse that can be converted to a CompletableFuture<AggregatedHttpResponse> via .aggregate().toCompletableFuture(). Teams that need custom retry logic — beyond what RetryingClient provides out of the box — often implement it as a recursive CompletableFuture chain: the initial call is made, and on failure, .exceptionally() or .handle() schedules another attempt. A billing helper that builds the HttpRequest at call time creates a fresh request per invocation. If UUID.randomUUID() is called inside that helper, each retry invocation generates a new UUID:

// BillingService.java
// UNSAFE: buildAndSend() computes UUID.randomUUID() at method entry.
// Called by the initial CompletableFuture chain AND by the exceptionally() retry callback.
// Each call evaluates UUID.randomUUID() independently — initial attempt ch_A, retry ch_B.

import com.linecorp.armeria.client.WebClient;
import com.linecorp.armeria.common.AggregatedHttpResponse;
import com.linecorp.armeria.common.MediaType;

import java.util.UUID;
import java.util.concurrent.CompletableFuture;

public class BillingService {

    private final WebClient stripeClient;

    public CompletableFuture<ChargeResult> chargeWithRetry(
            String customerId, String billingPeriod, long amountCents) {

        // UNSAFE: buildAndSend() generates UUID.randomUUID() inside itself.
        // Called for the initial attempt and called again inside exceptionally().
        return buildAndSend(customerId, billingPeriod, amountCents)
                .exceptionally(ex -> {
                    // Retry on transient failures — but buildAndSend() calls UUID.randomUUID()
                    // at its entry, so this retry produces a NEW idempotency key.
                    // If the initial attempt created ch_A before the timeout, this creates ch_B.
                    if (isTransient(ex)) {
                        return buildAndSend(customerId, billingPeriod, amountCents).join();
                    }
                    throw new RuntimeException(ex);
                });
    }

    private CompletableFuture<ChargeResult> buildAndSend(
            String customerId, String billingPeriod, long amountCents) {

        // UNSAFE: UUID.randomUUID() called at method entry — fresh on every invocation.
        // Called once per retry attempt by chargeWithRetry's exceptionally() handler.
        String idempotencyKey = UUID.randomUUID().toString();

        return stripeClient
                .prepare()
                .post("/v1/charges")
                .header("Idempotency-Key", idempotencyKey)
                .header("Authorization", "Bearer " + stripeKey)
                .content(MediaType.FORM_DATA, buildBody(customerId, amountCents))
                .execute()
                .aggregate()
                .toCompletableFuture()
                .thenApply(this::parseResponse);
    }
}

The failure scenario: chargeWithRetry("cust_456", "2026-09", 9900L) is called. buildAndSend() executes. UUID.randomUUID() returns "4d8a2c1e-5b3f-4d7e-9c0a-1b2d3e4f5a6b". The POST /v1/charges reaches Stripe. Stripe authorizes the card and commits ch_A to its ledger. Armeria’s response timeout fires before the HTTP response arrives. The CompletableFuture completes exceptionally with a ResponseTimeoutException. The .exceptionally() handler fires. isTransient(ex) returns true. The handler calls buildAndSend() again. UUID.randomUUID() returns "9e7c3a5b-1d2f-4e8c-6a0b-3c4d5e6f7a8b" — a completely independent UUID. Stripe receives a new POST with a new idempotency key. ch_B is created. Customer 456 is charged $99 twice for September 2026.

This failure is subtle because the initial and retry calls to buildAndSend() look structurally identical. There is no visible difference between a first call and a retry call from inside the method body — both execute UUID.randomUUID() at the top. The coupling between “retry intent” and “key must be the same” is invisible unless you understand that UUID.randomUUID() is a per-invocation value, not a per-billing-operation constant.

The subtler variant: Mono.fromFuture with Mono.defer wrapping both UUID.randomUUID() and the Armeria WebClient call — .retryWhen() re-subscribes the deferred pipeline — UUID re-evaluated on each re-subscription — ch_B on retry 1

Teams integrating Armeria into a Reactor pipeline (common when Armeria is used alongside Spring WebFlux or Project Reactor) sometimes wrap the Armeria WebClient call inside a Mono.defer() to make the call lazy. If UUID.randomUUID() is also inside the defer block, the UUID and the Armeria call both re-evaluate on each Reactor retry re-subscription:

// UNSAFE: UUID.randomUUID() inside Mono.defer() — re-evaluated on every re-subscription.
// Mono.defer() defers factory evaluation to subscription time.
// retryWhen(Retry.backoff()) re-subscribes the Mono on each retry — re-runs the defer block.

import reactor.core.publisher.Mono;
import reactor.util.retry.Retry;
import java.time.Duration;
import java.util.UUID;

public Mono<ChargeResult> chargeReactive(
        String customerId, String billingPeriod, long amountCents) {

    return Mono.defer(() -> {
                // UNSAFE: called on initial subscription AND on every retryWhen re-subscription.
                // UUID.randomUUID() produces a new value on each defer factory invocation.
                String idempotencyKey = UUID.randomUUID().toString();

                return Mono.fromFuture(
                        stripeClient
                                .prepare()
                                .post("/v1/charges")
                                .header("Idempotency-Key", idempotencyKey)
                                .header("Authorization", "Bearer " + stripeKey)
                                .content(MediaType.FORM_DATA, buildBody(customerId, amountCents))
                                .execute()
                                .aggregate()
                                .toCompletableFuture()
                                .thenApply(this::parseResponse)
                );
            })
            .retryWhen(Retry.backoff(3, Duration.ofMillis(500))
                            .filter(this::isTransient));
}

Mono.defer() defers evaluation of the factory lambda to subscription time. retryWhen(Retry.backoff()) re-subscribes the Mono on each retryable error. Each re-subscription re-enters the defer factory lambda from scratch. UUID.randomUUID() inside the lambda executes afresh on every subscription — initial subscription produces UUID_0, first retry re-subscription produces UUID_1. If the initial subscription created ch_A, the first retry creates ch_B.

The fix for both variants is identical: compute the stable key outside the retry boundary and capture it as an effectively-final variable that the retry callback or defer factory reads without re-evaluating:

// Safe: stableKey computed ONCE before any retry boundary.
// Both the initial attempt and all retry attempts reference the same captured variable.

// Safe CompletableFuture pattern:
public CompletableFuture<ChargeResult> chargeWithRetry(
        String customerId, String billingPeriod, long amountCents) {

    // Computed once — the same value is used by every call to buildAndSend().
    final String idempotencyKey = stableKey(customerId, billingPeriod);

    // Pre-flight claim before any Stripe call.
    if (!billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)) {
        return CompletableFuture.completedFuture(
                billingRepository.findExistingCharge(customerId, billingPeriod));
    }

    return buildAndSend(idempotencyKey, customerId, amountCents)
            .exceptionally(ex -> {
                if (isTransient(ex)) {
                    // buildAndSend receives the pre-computed key — does not call UUID.randomUUID().
                    return buildAndSend(idempotencyKey, customerId, amountCents).join();
                }
                throw new RuntimeException(ex);
            });
}

private CompletableFuture<ChargeResult> buildAndSend(
        String idempotencyKey, String customerId, long amountCents) {
    // idempotencyKey is passed as a parameter — not computed inside this method.
    return stripeClient
            .prepare()
            .post("/v1/charges")
            .header("Idempotency-Key", idempotencyKey)
            .header("Authorization", "Bearer " + stripeKey)
            .content(MediaType.FORM_DATA, buildBody(customerId, amountCents))
            .execute()
            .aggregate()
            .toCompletableFuture()
            .thenApply(this::parseResponse);
}

// Safe Reactor pattern:
public Mono<ChargeResult> chargeReactive(
        String customerId, String billingPeriod, long amountCents) {

    // Computed OUTSIDE Mono.defer() — not re-evaluated on retryWhen re-subscriptions.
    final String idempotencyKey = stableKey(customerId, billingPeriod);

    if (!billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)) {
        return Mono.fromCallable(() ->
                billingRepository.findExistingCharge(customerId, billingPeriod));
    }

    return Mono.fromFuture(
                    stripeClient
                            .prepare()
                            .post("/v1/charges")
                            .header("Idempotency-Key", idempotencyKey)  // same on every retry
                            .header("Authorization", "Bearer " + stripeKey)
                            .content(MediaType.FORM_DATA, buildBody(customerId, amountCents))
                            .execute()
                            .aggregate()
                            .toCompletableFuture()
                            .thenApply(this::parseResponse)
            )
            .retryWhen(Retry.backoff(3, Duration.ofMillis(500))
                            .filter(this::isTransient));
}

The final String idempotencyKey = stableKey(...); line is the critical boundary. Everything after it — the exceptionally() callback, the Mono.fromFuture() factory, the retryWhen() re-subscriptions — captures the same reference. UUID.randomUUID() is never called inside any retry-boundary closure. Stripe receives the same idempotency key on the initial attempt and all retries and returns the cached ch_A result.

Failure mode 3: Armeria service on Kubernetes with replicas:3 and per-JVM ScheduledExecutorService — TOCTOU race on hasCompletedForPeriod() — three pods generate distinct UUID.randomUUID() per customer — ch_A, ch_B, and ch_C per customer per billing period

Armeria services running in Kubernetes Deployments with multiple replicas each start their own independent JVM. Any ScheduledExecutorService — including Armeria’s Server.config().blockingTaskExecutor() or a plain Executors.newSingleThreadScheduledExecutor() — is per-JVM with no cross-pod coordination. When a billing job is scheduled on this executor, every replica runs the job independently at the same scheduled time. If the billing job includes a guard check like hasCompletedForPeriod() but the check is not atomic with the billing claim, all three pods can pass the guard concurrently before any pod commits the billing-started record. All three then proceed to stream the customer list and call UUID.randomUUID() per customer independently — producing ch_A from pod 1, ch_B from pod 2, and ch_C from pod 3 for each of 500 customers per billing period:

// MonthlyBillingJob.java
// UNSAFE: scheduled on each pod's blockingTaskExecutor() independently.
// No cross-pod coordination — all three Kubernetes replicas fire at the same cron time.
// hasCompletedForPeriod() is checked by all three pods before any pod commits — TOCTOU race.

import com.linecorp.armeria.server.Server;
import java.util.UUID;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

public class MonthlyBillingJob {

    private final BillingRepository billingRepository;
    private final StripeClient stripeClient;

    public void schedule(ScheduledExecutorService executor, String billingPeriod) {
        // UNSAFE: this.schedule() is called at startup on every Kubernetes pod.
        // All three pods schedule the billing task independently.
        // All three will fire at t=0 with no cross-pod lock.
        executor.scheduleAtFixedRate(
                () -> runBilling(billingPeriod),
                computeInitialDelay(),
                30, TimeUnit.DAYS
        );
    }

    private void runBilling(String billingPeriod) {
        // UNSAFE: hasCompletedForPeriod() is a non-atomic SELECT.
        // All three pods call this SELECT at the same time before any pod commits
        // the billing-started record — all three return false — TOCTOU race.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) {
            return;  // Would stop the race — but all three pass this check first.
        }

        // All three pods reach here simultaneously.
        List<Customer> customers = billingRepository.listActive();
        for (Customer customer : customers) {
            // UNSAFE: UUID.randomUUID() per customer per pod.
            // Pod 1: customer "cust_123" → UUID_pod1_123 → ch_A
            // Pod 2: customer "cust_123" → UUID_pod2_123 → ch_B
            // Pod 3: customer "cust_123" → UUID_pod3_123 → ch_C
            String idempotencyKey = UUID.randomUUID().toString();
            stripeClient.charge(customer.id(), idempotencyKey, customer.monthlyAmountCents());
        }

        billingRepository.markCompleted(billingPeriod);
    }
}

The failure scenario: the monthly billing job fires at midnight UTC on the first of the month. All three Kubernetes pods call runBilling("2026-09") at approximately the same time. All three query hasCompletedForPeriod("2026-09") — a SELECT against the billing_periods table. The September 2026 row does not exist yet (no pod has committed it). All three return false. All three proceed to list the 500 active customers. All three iterate the list and call UUID.randomUUID() per customer. Pod 1 generates UUID_pod1_123 for customer 123 and creates ch_A. Pod 2 generates UUID_pod2_123 for customer 123 and creates ch_B. Pod 3 generates UUID_pod3_123 for customer 123 and creates ch_C. 1,500 charges are created for 500 customers across the three pods.

This failure is not unique to Armeria — it is a general property of any per-JVM scheduler without distributed coordination. It is common in Armeria applications because Armeria does not include a built-in distributed scheduler. Teams migrating from a single-instance deployment to a replicated Kubernetes Deployment often add the billing job to the startup code without rethinking the coordination model, and the failure only manifests at month-end when the job first fires in production under scale.

The subtler variant: Armeria service paired with Spring @Scheduled via armeria-spring-boot3-starter — @Scheduled fires on every replica independently — same TOCTOU race — ch_A, ch_B, ch_C per customer

Armeria is commonly embedded in Spring Boot applications via armeria-spring-boot3-starter to expose gRPC and HTTP/2 endpoints alongside existing Spring MVC routes. In this configuration, Spring’s scheduler is active alongside Armeria’s server. A @Scheduled(cron = "0 0 1 * * *") billing method on a Spring @Service bean fires on every replica independently. Spring’s ThreadPoolTaskScheduler is per-JVM with no cross-pod coordination, identically to Armeria’s own blockingTaskExecutor(). The same TOCTOU race applies: three replicas call hasCompletedForPeriod() concurrently before any replica commits, all three proceed, all three generate distinct UUIDs per customer, three charges per customer are created.

A related variant: Armeria services occasionally expose administrative endpoints such as POST /admin/billing/run-monthly for manually triggering billing runs. If an external monitoring system or CI job retries this trigger endpoint on timeout (e.g., curl --retry 3, Kubernetes httpGet health check trigger, or Prometheus AlertManager webhook with retries), and if the load balancer distributes retries across replicas, two or three replicas each receive the trigger, each call UUID.randomUUID() per customer, and three charges per customer are created. The trigger endpoint is idempotent for the caller but not for Stripe.

The fix for failure mode 3

The solution requires coordination at a scope that spans all Kubernetes pods: a distributed lock acquired before the billing loop begins, and a pre-flight database insert that makes the billing claim durable before any Stripe call is issued:

// Safe: pg_try_advisory_lock() as a cross-pod distributed mutex.
// Only the pod that acquires the lock runs the billing loop.
// All other pods skip billing for this period.

import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
public class MonthlyBillingJob {

    private final JdbcTemplate jdbc;
    private final BillingRepository billingRepository;
    private final StripeClient stripeClient;

    // Safe: hash the billing period to a stable advisory lock key.
    // Same billing period always maps to the same lock ID on every pod.
    private static long advisoryLockKey(String billingPeriod) {
        return Math.abs(("armeria-monthly-billing:" + billingPeriod).hashCode());
    }

    @Scheduled(cron = "0 0 1 * * *")
    public void runMonthlyBilling() {
        String billingPeriod = currentBillingPeriod();  // e.g., "2026-09"
        long lockKey = advisoryLockKey(billingPeriod);

        // pg_try_advisory_lock() is non-blocking — returns false if another pod holds it.
        // Session-level lock: auto-released when the database connection is returned to pool.
        Boolean acquired = jdbc.queryForObject(
                "SELECT pg_try_advisory_lock(?)", Boolean.class, lockKey);

        if (!Boolean.TRUE.equals(acquired)) {
            // Another pod acquired the lock — skip this pod's billing run.
            return;
        }

        try {
            runBillingUnderLock(billingPeriod);
        } finally {
            jdbc.execute("SELECT pg_advisory_unlock(" + lockKey + ")");
        }
    }

    @Transactional
    private void runBillingUnderLock(String billingPeriod) {
        // Double-check inside the lock — handles the case where pod A acquired the lock,
        // pod B had already completed billing before pod A even tried to acquire.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) {
            return;
        }

        List<Customer> customers = billingRepository.listActive();
        for (Customer customer : customers) {
            // Safe: stableKey derived from business identity — same on every pod, every retry.
            String idempotencyKey = BillingService.stableKey(customer.id(), billingPeriod);

            // Pre-flight per-customer INSERT — prevents double-charging this customer
            // even if the job runs concurrently on two pods (belt-and-suspenders after the lock).
            int inserted = jdbc.update(
                    "INSERT INTO billing_charges (customer_id, billing_period, idempotency_key) " +
                    "VALUES (?, ?, ?) ON CONFLICT (customer_id, billing_period) DO NOTHING",
                    customer.id(), billingPeriod, idempotencyKey);

            if (inserted == 0) {
                // Another pod already claimed this customer — skip.
                continue;
            }

            ChargeResult result = stripeClient.charge(
                    customer.id(), idempotencyKey, customer.monthlyAmountCents());
            billingRepository.recordCharge(customer.id(), billingPeriod, result.chargeId());
        }

        billingRepository.markCompleted(billingPeriod);
    }
}

The advisory lock is a session-level PostgreSQL lock keyed by a stable integer derived from the billing period. Because the key is computed the same way on all pods — Math.abs(("armeria-monthly-billing:" + billingPeriod).hashCode()) — only one pod can hold it at a time. All other pods that call pg_try_advisory_lock() with the same key while the first pod holds it receive false and skip their billing runs. The pre-flight ON CONFLICT DO NOTHING per-customer insert is a belt-and-suspenders backstop for the race between the advisory lock acquisition and the first customer insertion: if two pods somehow both pass the lock check (e.g., a lock timeout or database connection error released the lock early), the UNIQUE (customer_id, billing_period) constraint ensures only one pod’s insert wins per customer. The vault key capped at expected_total × 1.10 bounds the maximum spend achievable even if both layers of defense are bypassed.

Patterns that reliably cause these failures in Armeria

The three failure modes share a structural pattern: a value that must be stable across all retry attempts (UUID.randomUUID()) is computed at a scope that is smaller than the retry boundary. In FM1, the scope is the decorator execute() method, which is inside the RetryingClient boundary. In FM2, the scope is the buildBillingRequest() method call site, which is inside the exceptionally() callback or inside the Mono.defer() factory that re-evaluates per retry. In FM3, the scope is the per-JVM scheduler invocation, which is inside the pod-level execution boundary with no cross-pod coordination.

Patterns that reliably produce these failure modes in Armeria:

• UUID.randomUUID() or System.currentTimeMillis() inside a SimpleDecoratingHttpClient.execute() override registered inside RetryingClient in the decorator chain
• UUID.randomUUID() inside a WebClient.prepare()...header(...)...execute() call made inside a retry loop body or a retry callback (exceptionally(), handle())
• UUID.randomUUID() inside a Mono.defer() factory that wraps an Armeria WebClient call combined with .retryWhen()
• Billing logic in a ScheduledExecutorService task (from blockingTaskExecutor(), Executors.newScheduledThreadPool(), or Spring @Scheduled via armeria-spring-boot3-starter) without a distributed lock
• An administrative trigger endpoint at POST /admin/billing/run-monthly without an idempotency guard, exposed to a caller that retries across load-balanced replicas

Patterns that are safe in Armeria:

• stableKey(customerId, billingPeriod) computed before the first WebClient.prepare() call, passed to the .header() call directly in the calling code before any retry boundary
• RetryingClient configured with caller-supplied idempotency key set on the request before entering the decorator chain — decorator reads from request header, does not generate it
• Mono.fromFuture(...) built with the stable key captured as a final variable outside any defer() or retryWhen() boundary
• pg_try_advisory_lock() acquired before the billing loop with the lock key derived from the billing period string — not from hostname, pod ID, UUID, or timestamp

Summary

The pattern connecting all three: the idempotency key must be derived from the business intent of the billing operation — customer ID, billing period, vendor namespace — not from any ephemeral runtime value that re-evaluates between decorator invocations, async callback re-entries, or concurrent pod executions. UUID.randomUUID(), System.currentTimeMillis() at request-build time, a per-decorator-invocation UUID, and any per-pod or per-attempt value all produce different values each time the code path that computes them is executed. A key derived from sha256(customerId + ":" + billingPeriod + ":armeria-billing")[:32] produces the same 32-character hex string on every SimpleDecoratingHttpClient.execute() invocation, on every CompletableFuture retry callback re-entry, and on every Kubernetes pod — stable across all three failure modes. Backing it with a PostgreSQL-level UNIQUE (customer_id, billing_period) constraint and a pg_try_advisory_lock() distributed mutex moves the deduplication guarantee out of ephemeral per-JVM state and into durable shared storage that survives ResponseTimeoutExceptions, JVM restarts, and multi-pod concurrent billing loops.

The vault key spend cap adds a hard financial boundary as a last line of defence: a per-billing-period vault key issued with a max_amount set to expected_total × 1.10 caps the maximum spend that any combination of decorator retry re-invocations, async callback re-entries, and multi-pod billing races can produce. Once the cap is reached, Stripe rejects further charges with a 402 Payment Required on the vault key, containing the blast radius to a known maximum regardless of how many RetryingClient retry attempts, exceptionally() re-invocations, or concurrent billing pods are in flight.