Dropwizard and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Apache HttpClient’s request interceptor chain executes on every send attempt — including every retry — so a UUID.randomUUID() call inside an HttpRequestInterceptor that sets the Idempotency-Key header produces a new key on each attempt: the initial POST to Stripe creates ch_A before a transient 503, and the first retry’s interceptor produces a fresh UUID so Stripe creates ch_B. Three Dropwizard-specific Stripe billing failure modes: Apache HttpClient’s interceptor chain re-executes on every retry; retry wrappers like Failsafe that re-build the HttpPost inside the retry lambda generate a fresh UUID per attempt; and Dropwizard’s ManagedScheduledExecutorService fires the billing job independently on every Kubernetes replica with no cross-pod coordination — three pods pass a concurrent database check, generate distinct UUID.randomUUID() values per customer, and create ch_A, ch_B, and ch_C per billing period.

This post covers all three failure modes with Java code, content-hash idempotency keys that are stable across Apache HttpClient interceptor re-invocations, Failsafe retry lambda re-entries, and multi-pod concurrent billing loops, pg_try_advisory_lock() for cross-pod scheduler serialization, pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For the CDI interceptor retry pattern in Quarkus (SmallRye Fault Tolerance), see the Quarkus and Stripe Integration post. For the MicroProfile Fault Tolerance interceptor in Helidon, see the Helidon and Stripe Integration post. For reactive retry failure modes with Ktor, see the Ktor and Stripe Integration post.

Failure mode 1: Apache HttpClient HttpRequestInterceptor computes UUID.randomUUID() per request — the interceptor chain re-executes on every DefaultHttpRequestRetryHandler retry — the initial attempt created ch_A before the 503 — the first retry’s interceptor creates ch_B

Apache HttpClient executes its HttpRequestInterceptor chain immediately before each request is sent over the wire. This includes the very first send attempt and every subsequent retry attempt triggered by DefaultHttpRequestRetryHandler, StandardHttpRequestRetryHandler, or ServiceUnavailableRetryStrategy. Dropwizard’s JerseyClientBuilder and HttpClientBuilder expose addInterceptorLast() for registering these interceptors. A common pattern is to register a single interceptor that automatically injects headers like Authorization, Content-Type, and — for Stripe calls — Idempotency-Key. The intent is a clean separation of header concerns from billing logic. The problem is that UUID.randomUUID() called inside the interceptor’s process() method produces a completely independent value on every invocation of that method — once per send attempt:

// StripeInterceptor.java
// UNSAFE: UUID.randomUUID() computed inside HttpRequestInterceptor.process().
// Apache HttpClient calls process() on every send attempt including retries.
// The initial attempt and every DefaultHttpRequestRetryHandler retry each call
// process() independently — producing a new UUID per attempt.

import org.apache.http.HttpRequest;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.protocol.HttpContext;

public class StripeInterceptor implements HttpRequestInterceptor {

    @Override
    public void process(HttpRequest request, HttpContext context) {
        // UNSAFE: called on every send attempt including retries.
        // Attempt 0 (initial send): UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Attempt 1 (first retry): UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        // Attempt 2 (second retry): UUID = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"
        String idempotencyKey = UUID.randomUUID().toString();
        request.setHeader("Idempotency-Key", idempotencyKey);
        request.setHeader("Authorization", "Bearer " + stripeApiKey);
    }
}

// Registered in Dropwizard Application.run():
// httpClientBuilder.addInterceptorLast(new StripeInterceptor());

The failure scenario: an agent calls billingService.chargeCustomer("cust_123", "2026-08", 9900L). The Dropwizard HttpClient builds the HttpPost for POST /v1/charges and enters the send phase. Before the first send, the interceptor chain executes. StripeInterceptor.process() is called. UUID.randomUUID() returns "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e". The header is set. Apache HttpClient sends the request to Stripe.

Stripe receives the POST. The card is authorized and the charge object ch_A is committed to Stripe’s ledger. Before the HTTP response is flushed back to the client, Stripe’s infrastructure encounters a transient overload condition and closes the connection. Apache HttpClient’s DefaultHttpRequestRetryHandler intercepts the resulting NoHttpResponseException or SocketException and schedules a retry. For the retry, Apache HttpClient re-enters the send phase for the same HttpPost object. Before the retry send, the interceptor chain executes again. StripeInterceptor.process() is called again. UUID.randomUUID() evaluates — a completely independent call that returns "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d". The retry’s setHeader() call overwrites the header value on the same HttpPost object. The retry goes to Stripe with a different Idempotency-Key.

Stripe has ch_A cached against "3f7a9b2c...", not the new key. Stripe processes the retry as a fresh charge request. ch_B is created. Customer 123 is charged $99 twice for August 2026.

This failure is invisible in code review because the interceptor pattern appears to cleanly separate header injection from billing logic, the name StripeInterceptor sounds correct, and the interceptor correctly handles Authorization headers (which are stateless and fine to regenerate per request). The UUID case is structurally different — it must be the same value across all send attempts for a single billing operation — but the interceptor model provides no mechanism to distinguish “the initial send” from “a retry for the same request”.

The subtler variant: ServiceUnavailableRetryStrategy retries on 503 — retryRequest() returns true — Apache HttpClient re-enters the send phase — interceptors run again — ch_B

DefaultHttpRequestRetryHandler only retries on IOException (network errors: connection reset, no response, socket timeout during read). It does not retry on HTTP 5xx response codes. Dropwizard applications that need to retry on HTTP 503 typically configure ServiceUnavailableRetryStrategy alongside the standard retry handler. ServiceUnavailableRetryStrategy.retryRequest(HttpResponse response, int executionCount, HttpContext context) is called when the server returns a response. If it returns true, Apache HttpClient discards the response and re-enters the send phase — which means the interceptor chain runs again for the new send attempt:

// UNSAFE: ServiceUnavailableRetryStrategy triggers re-send on 503.
// Apache HttpClient re-enters the send phase for every ServiceUnavailableRetryStrategy retry.
// The interceptor chain runs on every send including ServiceUnavailableRetryStrategy retries.
// UUID.randomUUID() inside StripeInterceptor.process() produces ch_B.

HttpClient httpClient = HttpClients.custom()
    .addInterceptorLast(new StripeInterceptor())  // UUID.randomUUID() inside process()
    .setRetryHandler(new DefaultHttpRequestRetryHandler(3, true))
    .setServiceUnavailableRetryStrategy(new ServiceUnavailableRetryStrategy() {
        @Override
        public boolean retryRequest(HttpResponse response, int executionCount, HttpContext context) {
            int statusCode = response.getStatusLine().getStatusCode();
            // Retry on 503 up to 3 times with 1-second delay.
            // Each retry re-enters the send phase — interceptors run again.
            return statusCode == 503 && executionCount <= 3;
        }
        @Override
        public long getRetryInterval() { return 1000L; }
    })
    .build();

When Stripe returns HTTP 503 (not a network error), DefaultHttpRequestRetryHandler does not trigger. ServiceUnavailableRetryStrategy.retryRequest() returns true. Apache HttpClient re-enters the send phase, runs the interceptors, sets a new UUID in the header, and sends the request again. From Stripe’s perspective, this is a fresh request with a new idempotency key. ch_B is created even though the previous request’s ch_A was successfully committed to Stripe’s ledger before the 503 was returned.

The developer added ServiceUnavailableRetryStrategy to make the service more resilient. The strategy is correct for idempotent GET requests. For a POST /v1/charges with a per-send UUID, it converts a correctly handled 503 into a duplicate charge.

The fix for failure mode 1

The idempotency key must be computed once per billing operation, before Apache HttpClient enters any send phase, and passed into the HttpPost in a way that survives all interceptor invocations without modification. The interceptor must read the key from the request context, not generate it:

// Safe: stable key computed BEFORE building the HttpPost.
// The interceptor reads the key from HttpContext — it does not generate it.
// DefaultHttpRequestRetryHandler and ServiceUnavailableRetryStrategy retries
// pass the same HttpContext to the interceptor — the key is unchanged.

import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;

public class BillingService {

    private final CloseableHttpClient httpClient;

    public BillingService(CloseableHttpClient httpClient) {
        this.httpClient = httpClient;
    }

    public ChargeResult chargeCustomer(String customerId, String billingPeriod, long amountCents)
            throws Exception {
        // Computed once per billing operation before any send attempt.
        String idempotencyKey = stableKey(customerId, billingPeriod);

        // Pre-flight: claim the billing slot in the database before sending to Stripe.
        boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
        if (!claimed) {
            return billingRepository.findCharge(customerId, billingPeriod);
        }

        // Store the stable key in HttpContext so the interceptor can read it.
        HttpClientContext context = HttpClientContext.create();
        context.setAttribute("stripe.idempotency-key", idempotencyKey);

        HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
        post.setEntity(new StringEntity(buildChargeBody(customerId, amountCents)));
        post.setHeader("Content-Type", "application/x-www-form-urlencoded");
        // Key set on the request itself — interceptor reads from context,
        // does not overwrite this header.

        try (CloseableHttpResponse response = httpClient.execute(post, context)) {
            return parseResponse(response);
        }
    }

    static String stableKey(String customerId, String billingPeriod) {
        try {
            var digest = java.security.MessageDigest.getInstance("SHA-256");
            var hash = digest.digest(
                    (customerId + ":" + billingPeriod + ":dropwizard-billing")
                            .getBytes(java.nio.charset.StandardCharsets.UTF_8));
            var sb = new StringBuilder(32);
            for (int i = 0; i < 16; i++) sb.append(String.format("%02x", hash[i]));
            return sb.toString();
        } catch (java.security.NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
    }
}

// Safe interceptor: reads the idempotency key from HttpContext.
// Does not compute UUID.randomUUID() — no new value per retry.
public class StripeInterceptor implements HttpRequestInterceptor {
    @Override
    public void process(HttpRequest request, HttpContext context) {
        String idempotencyKey = (String) context.getAttribute("stripe.idempotency-key");
        if (idempotencyKey != null) {
            request.setHeader("Idempotency-Key", idempotencyKey);
        }
        request.setHeader("Authorization", "Bearer " + stripeApiKey);
    }
}

The HttpClientContext is passed to both the initial send and all DefaultHttpRequestRetryHandler / ServiceUnavailableRetryStrategy retries. context.getAttribute("stripe.idempotency-key") returns the same string on every interceptor invocation. The interceptor sets the same value on every send attempt. Stripe receives the same Idempotency-Key on the initial send and all retries and returns the cached ch_A result without creating ch_B.

Failure mode 2: Failsafe RetryPolicy wrapper re-builds HttpPost with UUID.randomUUID() inside the retry callable — each attempt executes the callable body from scratch — first attempt created ch_A before HttpRequestTimeoutException — second attempt’s new UUID causes ch_B

Because DefaultHttpRequestRetryHandler only retries on IOException and not on HTTP 5xx response codes, many Dropwizard applications wrap their Stripe HTTP calls in an external retry library — most commonly Failsafe or Resilience4j. The canonical Failsafe pattern wraps the Stripe call in a Callable or CheckedSupplier passed to Failsafe.with(retryPolicy).get(() -> ...). On each failure, Failsafe re-invokes the callable. If the callable builds the HttpPost internally and calls UUID.randomUUID() at construction time, each retry invocation builds a new HttpPost with a new UUID:

// BillingService.java
// UNSAFE: HttpPost built inside the Failsafe callable — UUID computed at callable-body time.
// Failsafe re-invokes the callable body on every retry — UUID.randomUUID() re-evaluates.

import dev.failsafe.Failsafe;
import dev.failsafe.RetryPolicy;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;

import java.time.Duration;
import java.util.UUID;

public class BillingService {

    private final CloseableHttpClient httpClient;

    private final RetryPolicy<ChargeResult> retryPolicy = RetryPolicy.<ChargeResult>builder()
            .handle(Exception.class)
            .withDelay(Duration.ofSeconds(1))
            .withMaxRetries(3)
            .build();

    public ChargeResult chargeCustomer(String customerId, String billingPeriod, long amountCents)
            throws Exception {
        return Failsafe.with(retryPolicy).get(() -> {
            // UNSAFE: UUID computed inside the Failsafe callable body.
            // Failsafe re-invokes this entire lambda on each retry attempt.
            // Attempt 0: UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
            // Attempt 1: UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
            // Attempt 2: UUID = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"
            String idempotencyKey = UUID.randomUUID().toString();

            HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
            post.setHeader("Idempotency-Key", idempotencyKey);
            post.setHeader("Authorization", "Bearer " + stripeApiKey);
            post.setHeader("Content-Type", "application/x-www-form-urlencoded");
            post.setEntity(new StringEntity(
                    "amount=" + amountCents + "&currency=usd&customer=" + customerId));

            try (var response = httpClient.execute(post)) {
                return parseResponse(response);
            }
        });
    }
}

The failure scenario: Failsafe invokes the callable body for attempt 0. UUID.randomUUID() executes at the top of the lambda body and returns "3f7a9b2c...". A new HttpPost is built with Idempotency-Key: 3f7a9b2c.... httpClient.execute(post) sends POST /v1/charges to Stripe. Stripe authorizes the card and commits ch_A. Before the HTTP response is fully transmitted back to the client, the connection times out (HttpRequestTimeoutException) or a transient network error fires. Failsafe catches the exception and schedules attempt 1.

For attempt 1, Failsafe re-invokes the lambda body from its first statement. UUID.randomUUID() executes again — a completely independent call, with no knowledge that attempt 0 used "3f7a9b2c...". It returns "b8d2e4f6...". A new HttpPost is built. httpClient.execute(post) sends the request to Stripe with the new key. Stripe does not have the new key in its idempotency cache. It processes the request as a new charge and creates ch_B. Customer 123 is charged $99 twice.

The same failure occurs with manual retry loops: a for (int attempt = 0; attempt < 3; attempt++) loop that calls buildChargeRequest(customerId, billingPeriod) on each iteration, where buildChargeRequest() calls UUID.randomUUID() at invocation time. Every call to buildChargeRequest() is a fresh method invocation that executes UUID.randomUUID() from scratch — attempt 0 uses UUID_0, attempt 1 uses UUID_1.

The subtler variant: HttpPost built outside the retry lambda (UUID stable) but setHeader("Idempotency-Key", UUID.randomUUID().toString()) called inside the lambda for per-attempt tracing — overwrites the stable hash key — ch_B

A developer who understands that the HttpPost must be built before the retry boundary sometimes makes a subtler mistake: building the HttpPost with the stable content-hash key outside the lambda, then adding a second setHeader() call inside the lambda for “per-attempt request ID tracing” that reuses the Idempotency-Key header name with a fresh UUID:

// UNSAFE variant: HttpPost built with stable key outside the lambda,
// but a setHeader() call inside the lambda overwrites the Idempotency-Key
// with UUID.randomUUID() on each attempt for "per-attempt tracing."

public ChargeResult chargeCustomer(String customerId, String billingPeriod, long amountCents)
        throws Exception {
    // Stable key computed correctly before the Failsafe lambda.
    String idempotencyKey = stableKey(customerId, billingPeriod);

    HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
    post.setHeader("Idempotency-Key", idempotencyKey);  // Stable — set here.
    post.setHeader("Authorization", "Bearer " + stripeApiKey);
    post.setHeader("Content-Type", "application/x-www-form-urlencoded");
    post.setEntity(new StringEntity(
            "amount=" + amountCents + "&currency=usd&customer=" + customerId));

    return Failsafe.with(retryPolicy).get(() -> {
        // UNSAFE: setHeader() with UUID.randomUUID() overwrites the stable key.
        // Apache HttpClient's HttpPost.setHeader() replaces the existing header value.
        // Attempt 0: Idempotency-Key overwritten with "3f7a9b2c..." — ch_A created
        // Attempt 1: Idempotency-Key overwritten with "b8d2e4f6..." — ch_B created
        post.setHeader("Idempotency-Key", UUID.randomUUID().toString());
        try (var response = httpClient.execute(post)) {
            return parseResponse(response);
        }
    });
}

HttpPost.setHeader(name, value) replaces any existing header with that name. The stable key set before the lambda is overwritten on every lambda invocation. The developer intended the per-attempt UUID to serve as a request tracing ID (analogous to a X-Request-Id), but reused the Idempotency-Key header name. The correct approach for per-attempt tracing is a separate header like X-Attempt-Id that is distinct from the Stripe idempotency key.

The fix for failure mode 2

The idempotency key must be computed once, before Failsafe enters the retry loop, and it must not be reassigned or overwritten inside the retry callable. The HttpPost may be re-created inside the callable for other reasons (e.g., Apache HttpClient marks consumed entities as non-repeatable and rejects retries on consumed request bodies), but the idempotency key must be captured in a final variable in the enclosing scope and referenced from inside the lambda without modification:

// Safe: stable key computed BEFORE the Failsafe lambda.
// The lambda captures idempotencyKey as a final-effectively-captured variable.
// Every Failsafe retry attempt uses the same key. No UUID.randomUUID() inside.

public ChargeResult chargeCustomer(String customerId, String billingPeriod, long amountCents)
        throws Exception {
    // Computed once before Failsafe enters the retry loop.
    final String idempotencyKey = stableKey(customerId, billingPeriod);

    // Pre-flight: claim billing slot before any Stripe call.
    boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
    if (!claimed) {
        return billingRepository.findCharge(customerId, billingPeriod);
    }

    return Failsafe.with(retryPolicy).get(() -> {
        // HttpPost may be re-created inside the lambda if the entity was consumed.
        // The idempotencyKey is captured from the enclosing scope — not re-computed.
        // Attempt 0: Idempotency-Key = sha256("cust_123:2026-08:dropwizard-billing")[:32]
        // Attempt 1: same value — Stripe returns cached ch_A result
        HttpPost post = new HttpPost("https://api.stripe.com/v1/charges");
        post.setHeader("Idempotency-Key", idempotencyKey);
        post.setHeader("Authorization", "Bearer " + stripeApiKey);
        post.setHeader("Content-Type", "application/x-www-form-urlencoded");
        post.setEntity(new StringEntity(
                "amount=" + amountCents + "&currency=usd&customer=" + customerId));
        // Separate per-attempt tracing header — does NOT reuse Idempotency-Key.
        post.setHeader("X-Attempt-Id", UUID.randomUUID().toString());

        try (var response = httpClient.execute(post)) {
            return parseResponse(response);
        }
    });
}

idempotencyKey is computed once before the lambda. Java captures it in the lambda’s closure as an effectively final variable. Every Failsafe attempt invokes the lambda body and reads the same idempotencyKey value — no re-computation, no new UUID. The per-attempt X-Attempt-Id header carries a fresh UUID for distributed tracing without affecting the Stripe idempotency key. The pre-flight claimSlot() with ON CONFLICT DO NOTHING ensures that even if two concurrent threads enter chargeCustomer() for the same customer and billing period simultaneously, only one proceeds to Stripe. The second caller finds the slot already claimed and returns the already-created charge.

Failure mode 3: Dropwizard ManagedScheduledExecutorService billing job runs on every Kubernetes replica independently — TOCTOU race on hasCompletedForPeriod() — all three pods pass the check before any pod commits — distinct UUID.randomUUID() per pod per customer — ch_A, ch_B, ch_C per customer per billing period

Dropwizard’s Environment.lifecycle().scheduledExecutorService(nameFormat) creates a ScheduledExecutorService managed by Dropwizard’s lifecycle manager. When the application starts, the executor is initialized. Scheduled tasks registered with scheduleAtFixedRate() or scheduleWithFixedDelay() begin running on that executor. The critical constraint: this executor is entirely per-JVM. Dropwizard has no mechanism for cross-pod scheduling coordination. With replicas: 3 in a Kubernetes Deployment, all three pods initialize their own ManagedScheduledExecutorService, each scheduling the billing task independently at the same startup-relative offset:

// BillingJob.java
// UNSAFE: ManagedScheduledExecutorService is per-JVM.
// With replicas: 3, three pods each initialize this executor and schedule the task.
// All three pods fire the billing task at the same time — no cross-pod coordination.

public class BillingApplication extends Application<BillingConfiguration> {

    @Override
    public void run(BillingConfiguration config, Environment environment) {
        BillingService billingService = new BillingService(config);

        ScheduledExecutorService executor = environment.lifecycle()
                .scheduledExecutorService("billing-scheduler-%d")
                .threads(1)
                .build();

        // UNSAFE: All three Kubernetes replicas schedule this task at startup.
        // All three pods fire runMonthlyBilling() at the same time.
        executor.scheduleAtFixedRate(
                () -> runMonthlyBilling(billingService),
                calculateInitialDelay(),  // Same offset on all pods
                30,
                java.util.concurrent.TimeUnit.DAYS
        );
    }

    void runMonthlyBilling(BillingService billingService) {
        String billingPeriod = currentBillingPeriod();
        // TOCTOU: all 3 pods call this check concurrently before any pod writes
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) return;

        billingRepository.streamActiveCustomers().forEach(customerId -> {
            // UNSAFE: UUID.randomUUID() per customer per pod.
            // Pod 1: cust_123 gets UUID_A → ch_A
            // Pod 2: cust_123 gets UUID_B → ch_B
            // Pod 3: cust_123 gets UUID_C → ch_C
            // Three charges for 500 customers = 1,500 charges total.
            String idempotencyKey = UUID.randomUUID().toString();
            billingService.chargeCustomer(customerId, billingPeriod, idempotencyKey);
        });

        billingRepository.markCompleted(billingPeriod);
    }
}

The failure scenario unfolds at the start of the billing period. All three pods have been running since the last deployment. At the configured 30-day offset from startup (or at the equivalent point in the cycle), all three pods fire runMonthlyBilling() within milliseconds of each other. Each pod calls billingRepository.hasCompletedForPeriod("2026-08"). Because no pod has yet written the billing-started record (or the billing-completed record), all three pods read false.

All three pass the check. All three begin streaming the 500 active customer IDs from the database. All three call chargeCustomer() per customer. Each pod independently calls UUID.randomUUID() for customer cust_123: pod 1 generates UUID_A, pod 2 generates UUID_B, pod 3 generates UUID_C. Each sends a POST /v1/charges to Stripe with a distinct idempotency key. Stripe creates ch_A (from pod 1), ch_B (from pod 2), and ch_C (from pod 3). Customer 123 is charged $99 three times for August 2026. This repeats for all 500 customers: 1,500 total charges.

The TOCTOU (time-of-check to time-of-use) race is inherent to the SELECT / INSERT pattern used by hasCompletedForPeriod() without database-level locking. Even if the billing-started record is written at the beginning of the loop (not the end), the concurrent check-and-write sequence is not atomic across pods. Three pods can all read false before any pod’s INSERT becomes visible to the others under READ COMMITTED isolation.

The subtler variant: dropwizard-jobs library with Quartz RAMJobStore (the default) — no cross-node trigger coordination — three pods fire independently

Many Dropwizard applications use the dropwizard-jobs library, which integrates Quartz Scheduler into the Dropwizard lifecycle. Quartz jobs are defined as classes annotated with @Every("30d") or @On("0 0 1 * * ?"). The critical detail: dropwizard-jobs uses Quartz’s default RAMJobStore, which stores job and trigger state in memory within a single JVM. Each pod has its own independent in-memory Quartz scheduler. There is no cross-pod coordination. All three pods fire the annotated job method at the configured schedule:

// UNSAFE: dropwizard-jobs with RAMJobStore (default) — all replicas fire independently.
// @On cron fires on every Kubernetes pod at the same UTC second.

import io.dropwizard.jobs.Job;
import io.dropwizard.jobs.annotations.On;
import org.quartz.JobExecutionContext;

@On("0 0 1 * * ?")  // 01:00 UTC on the 1st of each month
public class MonthlyBillingJob extends Job {

    @Override
    public void doJob(JobExecutionContext context) {
        String billingPeriod = currentBillingPeriod();
        // TOCTOU: all 3 pods call this check at 01:00:00 UTC simultaneously.
        // All 3 read false before any pod writes the billing-started record.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) return;

        billingRepository.streamActiveCustomers().forEach(customerId -> {
            // Three pods, three distinct UUIDs per customer, three charges per customer.
            String idempotencyKey = UUID.randomUUID().toString();
            billingService.chargeCustomer(customerId, billingPeriod, idempotencyKey);
        });
    }
}

At 01:00:00 UTC on the billing date, all three pods trigger the MonthlyBillingJob from their independent Quartz schedulers. The TOCTOU race and UUID duplication produce the same 1,500 charges for 500 customers. Switching Quartz to JDBCJobStore with clustering solves the scheduler coordination problem: Quartz uses a database-backed lock table to ensure only one node fires each trigger. But even with JDBCJobStore, a content-hash key and pre-flight ON CONFLICT DO NOTHING remain necessary backstops for edge cases where the cluster lock is not engaged (operator-triggered billing runs outside the scheduler, rolling deploy overlap windows, manual database maintenance runs).

The fix for failure mode 3

The fix has three layers: database-level distributed mutex to ensure only one pod runs the billing loop, content-hash idempotency key stable across all pods, and pre-flight ON CONFLICT DO NOTHING as the authoritative cluster-wide billing mutex:

// Safe: pg_try_advisory_lock() as cross-pod distributed mutex.
// Only the pod that acquires the lock runs the billing job.
// The other two pods log "lock not acquired" and exit immediately.
// Content-hash key ensures the same Idempotency-Key on every pod.
// Pre-flight ON CONFLICT DO NOTHING as authoritative cluster-wide backstop.

void runMonthlyBilling(BillingService billingService, DataSource dataSource) {
    String billingPeriod = currentBillingPeriod();
    long lockKey = Math.abs(("dropwizard-monthly-billing:" + billingPeriod).hashCode());

    try (Connection conn = dataSource.getConnection()) {
        // pg_try_advisory_lock() returns true if this pod acquired the lock.
        // Returns false immediately (non-blocking) if another pod holds it.
        // The lock is session-level: held until released or connection closed.
        boolean acquired;
        try (PreparedStatement ps = conn.prepareStatement(
                "SELECT pg_try_advisory_lock(?)")) {
            ps.setLong(1, lockKey);
            try (ResultSet rs = ps.executeQuery()) {
                rs.next();
                acquired = rs.getBoolean(1);
            }
        }

        if (!acquired) {
            log.info("Advisory lock for {} not acquired — another pod is billing", billingPeriod);
            return;
        }

        try {
            if (billingRepository.hasCompletedForPeriod(billingPeriod)) return;

            billingRepository.streamActiveCustomers().forEach(customerId -> {
                // Safe: content-hash key is identical on all pods.
                // Even if two pods somehow both acquire the lock,
                // the ON CONFLICT DO NOTHING pre-flight stops the second pod.
                String idempotencyKey = stableKey(customerId, billingPeriod);
                boolean claimed = billingRepository.claimSlot(
                        customerId, billingPeriod, idempotencyKey);
                if (claimed) {
                    billingService.chargeCustomer(customerId, billingPeriod, idempotencyKey);
                }
            });

            billingRepository.markCompleted(billingPeriod);
        } finally {
            // Release the session-level advisory lock explicitly.
            // Also released automatically if the connection is closed.
            try (PreparedStatement ps = conn.prepareStatement(
                    "SELECT pg_advisory_unlock(?)")) {
                ps.setLong(1, lockKey);
                ps.execute();
            }
        }
    } catch (SQLException e) {
        log.error("Billing job failed for period {}", billingPeriod, e);
    }
}

// billingRepository.claimSlot() implementation:
// INSERT INTO billing_slots (customer_id, billing_period, idempotency_key, created_at)
// VALUES (?, ?, ?, NOW())
// ON CONFLICT (customer_id, billing_period) DO NOTHING
// Returns true if the INSERT succeeded (this caller owns the slot).
// Returns false if another caller already claimed it (DO NOTHING fired).

// Quartz JDBCJobStore alternative for dropwizard-jobs:
// quartz.properties:
//   org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
//   org.quartz.jobStore.isClustered=true
//   org.quartz.jobStore.clusterCheckinInterval=20000
//   org.quartz.scheduler.instanceId=AUTO
// JDBCJobStore uses SELECT FOR UPDATE on QRTZ_LOCKS to ensure only one node
// fires each trigger — other nodes see the lock held and skip.
// pg_try_advisory_lock() remains as a backstop for operator-triggered runs.

pg_try_advisory_lock(lockKey) is a non-blocking PostgreSQL advisory lock. The first pod to call it with a given key acquires the lock and proceeds to billing. The second and third pods call it and immediately receive false — they log a skip message and return without running any billing code. The lock is held for the duration of the billing run and released in the finally block (or automatically when the database connection is returned to the pool or closed). The billing_slots table with UNIQUE (customer_id, billing_period) and the ON CONFLICT DO NOTHING pre-flight are a second layer of protection: even in the rare case where two pods concurrently acquire the advisory lock (e.g., after a PostgreSQL failover that resets advisory lock state), the database-level uniqueness constraint ensures only one charge per customer per billing period reaches Stripe.

For dropwizard-jobs, the Quartz JDBCJobStore configuration replaces the per-JVM RAMJobStore with a shared database-backed scheduler. Quartz uses its own locking protocol (SELECT FOR UPDATE on the QRTZ_LOCKS table) to ensure that only one Quartz node fires each trigger. The other two pods detect the lock held and do not fire the job. The pg_try_advisory_lock() check remains relevant for billing code paths outside the Quartz trigger — operator-triggered billing runs, API endpoints that initiate billing on demand, or startup billing checks that run independently of the Quartz schedule.

Summary

Three Dropwizard + Apache HttpClient failure modes, one root pattern: the idempotency key computed at runtime must not re-evaluate between any two code paths that represent the same billing operation. Apache HttpClient’s interceptor chain, Failsafe’s retry callable, and a per-JVM scheduler all introduce boundaries where the same logical billing operation crosses code that calls UUID.randomUUID() more than once.

The pattern connecting all three: the idempotency key must be derived from the business intent of the billing operation — customer ID, billing period, vendor namespace — not from any ephemeral runtime value that re-evaluates between interceptor invocations, Failsafe retry lambda re-entries, or concurrent pod executions. UUID.randomUUID(), System.currentTimeMillis() at method entry, a per-interceptor-call UUID, and any per-send-attempt value all produce different values each time the code path that computes them is executed. A key derived from sha256(customerId + ":" + billingPeriod + ":dropwizard-billing")[:32] produces the same 32-character hex string on every interceptor call, on every Failsafe retry lambda invocation, and on every Kubernetes pod — stable across all three failure modes. Backing it with a PostgreSQL-level UNIQUE (customer_id, billing_period) constraint and a pg_try_advisory_lock() distributed mutex moves the deduplication guarantee out of ephemeral per-JVM state and into durable shared storage that survives connection drops, rolling deploys, and multi-pod concurrent billing loops.

The vault key spend cap adds a hard financial boundary as a last line of defence: a per-billing-period vault key issued with a max_amount set to expected_total × 1.10 caps the maximum spend that any combination of interceptor retries, Failsafe retry attempts, and multi-pod billing races can produce. Once the cap is reached, Stripe rejects further charges with a 402 Payment Required on the vault key, containing the blast radius to a known maximum regardless of how many interceptor invocations, retry lambda re-entries, or concurrent billing pods are in flight.