Spring Boot and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Spring Retry’s @Retryable is an AOP proxy that calls MethodInvocation.proceed() on every retry attempt, re-executing the annotated method body from its first line — so UUID.randomUUID() at method entry evaluates fresh on every retry, the initial POST to Stripe creates ch_A before a transient error, and the first retry’s new UUID causes Stripe to create ch_B. Three Spring Boot-specific Stripe billing failure modes: @Retryable AOP re-invocation; RestTemplate’s ClientHttpRequestInterceptor computing a fresh UUID per execute() call, re-triggered by every @Retryable retry; and @Scheduled billing jobs running on all three Kubernetes replicas simultaneously with no cross-pod coordination — TOCTOU race on the database check, three independent UUID values per customer, ch_A, ch_B, and ch_C per billing period.

This post covers all three failure modes with Java code, content-hash idempotency keys that remain stable across @Retryable re-invocations and interceptor re-executions, ShedLock for @Scheduled distributed coordination, pg_try_advisory_lock() for cross-pod scheduler serialization, and pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — plus a per-billing-period spend-cap vault key as a hard financial backstop. For the reactive retry failure modes in Spring’s non-blocking layer, see the Spring WebFlux and Stripe Integration post. For the MicroProfile Fault Tolerance interceptor pattern in Quarkus, see the Quarkus and Stripe Integration post. For Ktor’s HttpRequestRetry plugin failure modes, see the Ktor and Stripe Integration post.

Failure mode 1: @Retryable AOP proxy calls proceed() on every retry — UUID.randomUUID() at method entry fires on every re-invocation — initial attempt created ch_A before ApiConnectionException — first retry creates ch_B

Spring Retry’s @Retryable annotation is implemented by the AnnotationAwareRetryOperationsInterceptor, which wraps the target bean in a CGLIB proxy. When the proxy intercepts a method call, it invokes the method via MethodInvocation.proceed(). If the method throws a retryable exception, the proxy waits for the configured backoff and then calls proceed() again to re-execute the method body from its beginning. Every statement in the annotated method body executes again on every retry. UUID.randomUUID() at the top of the method body is no different from any other Java expression — it evaluates fresh each time the method body starts:

// BillingService.java
// UNSAFE: @Retryable re-invokes the full method body on every retry attempt.
// UUID.randomUUID() at method entry fires on every AnnotationAwareRetryOperationsInterceptor
// proceed() call — a different UUID per retry means a different Idempotency-Key per attempt.

import com.stripe.exception.StripeException;
import com.stripe.model.Charge;
import com.stripe.net.RequestOptions;
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
import org.springframework.stereotype.Service;

import java.util.HashMap;
import java.util.Map;
import java.util.UUID;

@Service
public class BillingService {

    @Retryable(
        retryFor = {StripeException.class},
        maxAttempts = 3,
        backoff = @Backoff(delay = 1000, multiplier = 2.0)
    )
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents)
            throws StripeException {
        // UNSAFE: computed inside the @Retryable method body.
        // Spring AOP re-invokes this method from line 1 on every retry.
        // Attempt 0 (initial): UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Attempt 1 (first retry): UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        // Attempt 2 (second retry): UUID = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"
        String idempotencyKey = UUID.randomUUID().toString();

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();

        Map<String, Object> params = new HashMap<>();
        params.put("amount", amountCents);
        params.put("currency", "usd");
        params.put("customer", customerId);
        params.put("description", "Subscription " + billingPeriod);

        return Charge.create(params, options);
    }
}

The failure scenario: an agent or billing cron triggers billingService.chargeCustomer("cust_123", "2026-08", 9900L). The Spring AOP proxy intercepts the call and invokes proceed() for attempt 0. The method body executes from the top. UUID.randomUUID() returns "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e". The Stripe Java SDK sends POST /v1/charges with Idempotency-Key: 3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e to Stripe’s API.

Stripe receives the request. The card is authorized. The charge object ch_A is committed to Stripe’s ledger. Before Stripe’s API server flushes the HTTP response, a transient connectivity problem occurs — a connection reset mid-response, a read timeout, or an overloaded upstream proxy. The Stripe Java SDK catches the resulting SocketTimeoutException or ApiConnectionException and propagates it to the caller. Spring’s AnnotationAwareRetryOperationsInterceptor catches the ApiConnectionException (which is a subclass of StripeException), records attempt 0 as failed, waits 1,000 ms (the configured backoff delay), and calls proceed() again for attempt 1.

For attempt 1, the method body re-executes from its beginning. UUID.randomUUID() evaluates again — completely independently — and returns "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d". A new RequestOptions object is built with this new key. The SDK sends POST /v1/charges with Idempotency-Key: b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d to Stripe.

Stripe has ch_A committed against "3f7a9b2c...". The new request carries a key Stripe has never seen. Stripe processes it as a new charge request, authorizes the card again, and creates ch_B. Customer 123 is charged $99 twice for August 2026.

This failure is easy to miss in code review: @Retryable is intended to make the service more resilient, and adding it to a billing method looks like a straightforward reliability improvement. The UUID appears immediately inside the method body, not in a separate retry callback or interceptor, so there’s no obvious signal that it will re-evaluate. The test for this method typically mocks the Stripe SDK and asserts on the successful-path response, with no check on how many distinct idempotency keys were generated across retry attempts.

The subtler variant: @Retryable + @Transactional proxy chain — the transaction commits the billing record on the initial attempt — the retry’s new transaction hits a UNIQUE constraint violation — DataIntegrityViolationException fires and may itself be retried

A billing service that correctly writes a billing record to the database before calling Stripe is a step toward deduplication. But combining @Retryable and @Transactional on the same method creates a proxy chain with a non-obvious execution order that can produce a retry storm against the database:

// UNSAFE combination: @Retryable outer proxy wraps @Transactional inner proxy.
// The transaction commits the billing INSERT on the initial attempt.
// On retry, the retry proxy calls proceed() — the transaction proxy opens a new transaction.
// The new transaction tries to INSERT again — UNIQUE constraint violation fires.
// If @Retryable includes DataIntegrityViolationException, it retries the UNIQUE violation.

@Retryable(retryFor = {StripeException.class, DataIntegrityViolationException.class}, maxAttempts = 3)
@Transactional
public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents)
        throws StripeException {
    String idempotencyKey = UUID.randomUUID().toString();

    // Correct intent: write billing record before calling Stripe.
    // But each @Transactional re-invocation opens a fresh transaction.
    // The initial attempt commits this INSERT. The retry tries to INSERT again.
    billingRepository.insertBillingRecord(customerId, billingPeriod, idempotencyKey);

    // ...Stripe call here — may not even be reached on retry if INSERT already throws.
    RequestOptions options = RequestOptions.builder().setIdempotencyKey(idempotencyKey).build();
    return Charge.create(buildParams(customerId, amountCents), options);
}

Spring applies the @Retryable proxy outside the @Transactional proxy in the default proxy chain. When the retry proxy calls proceed() for attempt 1, it re-enters the transaction proxy, which opens a new transaction. The INSERT for (customerId="cust_123", billingPeriod="2026-08") is attempted again. The UNIQUE constraint on (customer_id, billing_period) fires a DataIntegrityViolationException. If @Retryable’s retryFor includes DataIntegrityViolationException (a common pattern for retry-all-exceptions), the retry proxy catches the new exception and schedules attempt 2 — which produces the same DataIntegrityViolationException again, consuming all remaining retry attempts against the database constraint rather than against a Stripe transient error. The correct fix is to separate the concerns: @Transactional pre-flight claim is not retried; only the Stripe call is retried with a stable key that was computed before any transaction opened.

The fix for failure mode 1

The idempotency key must be computed before the @Retryable-annotated method is called, and passed in as a parameter. An alternative structure is to keep the method annotated with @Retryable but eliminate UUID.randomUUID() from the method body and replace it with a deterministic computation from method parameters that produces the same value on every proceed() re-invocation:

// Safe: idempotency key derived from stable, deterministic inputs available as method parameters.
// stableKey() computes the same value regardless of how many times it is called with the same inputs.
// @Retryable proceeds() multiple times — stableKey() returns the same string on every invocation.

@Service
public class BillingService {

    @Retryable(
        retryFor = {StripeException.class},
        maxAttempts = 3,
        backoff = @Backoff(delay = 1000, multiplier = 2.0)
    )
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents)
            throws StripeException {
        // Deterministic — same value on every @Retryable proceed() for the same parameters.
        // Attempt 0: sha256("cust_123:2026-08:spring-billing") → "a3f7c9e2b1d4..."
        // Attempt 1: sha256("cust_123:2026-08:spring-billing") → same "a3f7c9e2b1d4..."
        // Attempt 2: sha256("cust_123:2026-08:spring-billing") → same "a3f7c9e2b1d4..."
        String idempotencyKey = stableKey(customerId, billingPeriod);

        // Pre-flight: claim billing slot before calling Stripe.
        // Uses ON CONFLICT DO NOTHING — safe to call on every @Retryable attempt.
        boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
        if (!claimed) {
            return billingRepository.findCharge(customerId, billingPeriod);
        }

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();

        Map<String, Object> params = new HashMap<>();
        params.put("amount", amountCents);
        params.put("currency", "usd");
        params.put("customer", customerId);

        return Charge.create(params, options);
    }

    static String stableKey(String customerId, String billingPeriod) {
        try {
            var digest = java.security.MessageDigest.getInstance("SHA-256");
            var hash = digest.digest(
                (customerId + ":" + billingPeriod + ":spring-billing")
                    .getBytes(java.nio.charset.StandardCharsets.UTF_8));
            var sb = new StringBuilder(32);
            for (int i = 0; i < 16; i++) sb.append(String.format("%02x", hash[i]));
            return sb.toString();
        } catch (java.security.NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
    }
}

stableKey("cust_123", "2026-08") computes sha256("cust_123:2026-08:spring-billing")[:32]. This is a pure function of its inputs. Every @Retryable proceed() invocation for the same parameters computes the same 32-character hex string. Stripe receives the same Idempotency-Key on every attempt and returns the cached ch_A result on all retry attempts after the initial one. The claimSlot() pre-flight with ON CONFLICT DO NOTHING is safe to call on every attempt — if the slot was already claimed on attempt 0, the call returns the existing charge without re-entering the Stripe path. The @Transactional proxy, if present, is applied inside claimSlot() at the repository layer rather than around the entire retry-annotated method, keeping the transaction scoped to the database operation and the retry scoped to the Stripe call.

Failure mode 2: RestTemplate ClientHttpRequestInterceptor computes UUID.randomUUID() per execute() call — @Retryable retry triggers a new restTemplate.exchange() — new execute() — interceptor re-runs — new UUID — ch_B

A common Spring Boot pattern delegates header injection to a ClientHttpRequestInterceptor registered with RestTemplate. This separates header concerns from business logic — the billing service calls restTemplate.exchange() with no header manipulation, and the interceptor handles Authorization, Content-Type, tracing headers, and — for Stripe calls — Idempotency-Key. The pattern looks clean, but the interceptor’s intercept() method is called by RestTemplate’s execute() on every request. When @Retryable retries the annotated method, the retry re-invokes restTemplate.exchange(), which re-calls execute(), which runs the interceptor chain again, which calls UUID.randomUUID() again:

// StripeInterceptor.java
// UNSAFE: UUID computed per execute() call inside ClientHttpRequestInterceptor.intercept().
// @Retryable retry triggers a new restTemplate.exchange() → new execute() → this fires again.

import org.springframework.http.HttpRequest;
import org.springframework.http.client.ClientHttpRequestExecution;
import org.springframework.http.client.ClientHttpRequestInterceptor;
import org.springframework.http.client.ClientHttpResponse;

import java.io.IOException;
import java.util.UUID;

public class StripeInterceptor implements ClientHttpRequestInterceptor {

    private final String stripeApiKey;

    public StripeInterceptor(String stripeApiKey) {
        this.stripeApiKey = stripeApiKey;
    }

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body,
                                        ClientHttpRequestExecution execution) throws IOException {
        // UNSAFE: called on every RestTemplate execute() invocation.
        // @Retryable re-invokes the annotated method → restTemplate.exchange() called again
        // → execute() called again → this intercept() called again → new UUID.
        // Attempt 0: "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Attempt 1 (@Retryable retry): "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        request.getHeaders().set("Idempotency-Key", UUID.randomUUID().toString());
        request.getHeaders().set("Authorization", "Bearer " + stripeApiKey);
        return execution.execute(request, body);
    }
}

// BillingService.java — looks clean: no UUID in the service layer.
// The interceptor handles it. But @Retryable makes this unsafe.

@Service
public class BillingService {

    private final RestTemplate stripeRestTemplate;  // RestTemplate with StripeInterceptor registered.

    @Retryable(retryFor = {RestClientException.class}, maxAttempts = 3, backoff = @Backoff(delay = 1000))
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents) {
        // No UUID here — delegated to StripeInterceptor. Looks correct. Is not.
        // @Retryable proxy re-invokes this method on retry.
        // re-invocation calls stripeRestTemplate.exchange() again.
        // exchange() calls execute() which runs the interceptor chain.
        // StripeInterceptor.intercept() fires again — UUID.randomUUID() → new key — ch_B.
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
        MultiValueMap<String, String> body = new LinkedMultiValueMap<>();
        body.add("amount", String.valueOf(amountCents));
        body.add("currency", "usd");
        body.add("customer", customerId);
        HttpEntity<MultiValueMap<String, String>> entity = new HttpEntity<>(body, headers);
        ResponseEntity<String> response = stripeRestTemplate.exchange(
            "https://api.stripe.com/v1/charges", HttpMethod.POST, entity, String.class);
        return parseCharge(response.getBody());
    }
}

The billing service looks correct: no UUID inside the service layer. The interceptor appears to centralize header management cleanly. The @Retryable annotation appears to add resilience. But the three features compose into a duplicate charge: @Retryable retries the method body, the method body calls restTemplate.exchange(), exchange() calls execute(), execute() runs the interceptor chain, and the interceptor recomputes the idempotency key. Attempt 0 uses UUID_0, Stripe creates ch_A. Attempt 1 uses UUID_1, Stripe creates ch_B.

The failure scenario: the initial request reaches Stripe, which commits ch_A. A ResourceAccessException (which wraps a network-level IOException) is thrown before the response is received. @Retryable catches it (since ResourceAccessException extends RestClientException), waits 1,000 ms, and calls proceed() for attempt 1. The method re-executes from the top, calls stripeRestTemplate.exchange(), which calls execute(), which calls StripeInterceptor.intercept(). The interceptor computes a fresh UUID. Stripe sees a new idempotency key it has never processed. ch_B is created.

The subtler variant: MockRestServiceServer unit tests pass — the mock does not enforce idempotency key uniqueness — production code silently generates a new UUID per retry

Unit tests that use MockRestServiceServer to simulate Stripe responses do not validate the Idempotency-Key header for uniqueness across retries. A test that configures the mock server to throw on the first call and succeed on the second call passes green — the retry succeeded — without checking whether attempt 0 and attempt 1 sent the same key:

// Unit test — passes green. Does not catch the idempotency key bug.
// MockRestServiceServer validates request count, not Idempotency-Key uniqueness.

@Test
void chargeCustomer_retriesOnNetworkError_succeeds() {
    mockServer.expect(once(), requestTo(STRIPE_URL))
        .andExpect(method(POST))
        // No check: .andExpect(header("Idempotency-Key", "same-value-on-both-attempts"))
        .andRespond(withException(new IOException("connection reset")));

    mockServer.expect(once(), requestTo(STRIPE_URL))
        .andExpect(method(POST))
        // This attempt has a DIFFERENT Idempotency-Key — MockRestServiceServer doesn't check.
        .andRespond(withSuccess(CHARGE_RESPONSE_JSON, MediaType.APPLICATION_JSON));

    Charge charge = billingService.chargeCustomer("cust_123", "2026-08", 9900L);
    assertNotNull(charge);

    // Test passes. In production, Stripe would have created ch_B on attempt 1.
    // MockRestServiceServer is not a Stripe server and does not enforce idempotency semantics.
    mockServer.verify();
}

The fix for the test: assert that both requests carry the same Idempotency-Key header. One approach is a custom RequestMatcher that captures the key from attempt 0 and asserts it matches attempt 1. Another is to use a real Stripe test-mode account with STRIPE_SECRET_KEY pointing to a test key — Stripe will return the cached ch_A result on the second request if the same key is used, and create a second charge if a new key is used, making the test failure observable.

The fix for failure mode 2

The interceptor must not generate the idempotency key. The key must be computed once per billing operation and stored in a context that the interceptor can read — a request attribute, a thread-local, or a header set by the calling service before entering exchange(). The cleanest approach for RestTemplate: compute the stable key in the service layer and set it as a header on the HttpEntity before calling exchange(). The interceptor reads the already-set header instead of generating a new value:

// Safe: stable key computed in the service layer before exchange().
// Interceptor reads and forwards the existing header — does not call UUID.randomUUID().

// StripeInterceptor.java (safe version)
public class StripeInterceptor implements ClientHttpRequestInterceptor {

    private final String stripeApiKey;

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body,
                                        ClientHttpRequestExecution execution) throws IOException {
        // Safe: reads existing Idempotency-Key set by the service layer.
        // Does not call UUID.randomUUID() — no new key per execute() call.
        if (!request.getHeaders().containsKey("Idempotency-Key")) {
            // Only set a fallback for non-billing endpoints. Billing endpoints must set their own.
            // For Stripe POST endpoints that need idempotency, the caller must provide the key.
            throw new IllegalStateException("Stripe POST request missing Idempotency-Key header");
        }
        request.getHeaders().set("Authorization", "Bearer " + stripeApiKey);
        return execution.execute(request, body);
    }
}

// BillingService.java (safe version)
@Service
public class BillingService {

    private final RestTemplate stripeRestTemplate;

    @Retryable(retryFor = {RestClientException.class}, maxAttempts = 3, backoff = @Backoff(delay = 1000))
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents) {
        // Stable key computed in the service layer — same value on every @Retryable proceed().
        String idempotencyKey = stableKey(customerId, billingPeriod);

        // Set the key on the request headers BEFORE exchange() — interceptor reads it, not generates it.
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
        headers.set("Idempotency-Key", idempotencyKey);

        MultiValueMap<String, String> body = new LinkedMultiValueMap<>();
        body.add("amount", String.valueOf(amountCents));
        body.add("currency", "usd");
        body.add("customer", customerId);

        HttpEntity<MultiValueMap<String, String>> entity = new HttpEntity<>(body, headers);
        ResponseEntity<String> response = stripeRestTemplate.exchange(
            "https://api.stripe.com/v1/charges", HttpMethod.POST, entity, String.class);
        return parseCharge(response.getBody());
    }
}

stableKey(customerId, billingPeriod) produces the same value on every @Retryable re-invocation for the same parameters. The interceptor throws an IllegalStateException if the header is missing — making it a mandatory contract rather than a silent fallback. The first @Retryable attempt and all subsequent retries send the same Idempotency-Key header to Stripe. Stripe returns the cached ch_A result on attempts 1 and 2 without creating ch_B or ch_C. The pre-flight claimSlot() with ON CONFLICT DO NOTHING ensures that concurrent calls for the same customer and billing period are serialized at the database layer, so only one call proceeds to Stripe even if two threads enter chargeCustomer() simultaneously.

Failure mode 3: Spring @Scheduled billing job runs on every Kubernetes replica independently — TOCTOU race on hasCompletedForPeriod() — all three pods pass the check before any pod commits — distinct UUID.randomUUID() per pod per customer — ch_A, ch_B, ch_C per customer per billing period

Spring’s @Scheduled annotation runs the annotated method on a TaskScheduler thread inside each JVM. There is no mechanism for cross-pod coordination. When a Spring Boot application is deployed with replicas: 3 in a Kubernetes Deployment, all three pods initialize their own TaskSchedulers and schedule the billing method independently. At the configured cron time, all three pods fire the billing method within milliseconds of each other:

// BillingJob.java
// UNSAFE: @Scheduled fires on every JVM — with replicas:3, three pods run this concurrently.
// TOCTOU race on hasCompletedForPeriod() — all three pods read false before any pod commits.

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import java.time.YearMonth;
import java.util.List;
import java.util.UUID;

@Component
public class BillingJob {

    private final CustomerRepository customerRepository;
    private final BillingRepository billingRepository;
    private final StripeClient stripeClient;

    // Fires at 02:00 UTC on the 1st of each month.
    // With replicas:3, all three pods fire this method at 02:00:00 UTC simultaneously.
    @Scheduled(cron = "0 0 2 1 * *")
    public void runMonthlyBilling() {
        String billingPeriod = YearMonth.now().toString();  // e.g., "2026-08"

        // TOCTOU: all three pods call this SELECT concurrently before any pod commits.
        // All three read false. All three proceed to billing.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) {
            return;
        }

        List<Customer> customers = customerRepository.findAllActive();  // 500 customers

        for (Customer customer : customers) {
            // UNSAFE: each pod generates its own UUID per customer.
            // Pod 1: UUID_A_1 for cust_1, UUID_A_2 for cust_2, ..., UUID_A_500 for cust_500 → ch_A_1 through ch_A_500
            // Pod 2: UUID_B_1 for cust_1, UUID_B_2 for cust_2, ..., UUID_B_500 for cust_500 → ch_B_1 through ch_B_500
            // Pod 3: UUID_C_1 for cust_1, ..., UUID_C_500 for cust_500 → ch_C_1 through ch_C_500
            // Result: 1,500 charges created for 500 customers — 3 charges per customer.
            String idempotencyKey = UUID.randomUUID().toString();
            stripeClient.charge(customer, idempotencyKey);
        }

        billingRepository.markCompleted(billingPeriod);
    }
}

The failure scenario: all three pods reach 02:00:00 UTC. Each pod’s Spring TaskScheduler fires runMonthlyBilling() on a scheduled thread. All three pods call billingRepository.hasCompletedForPeriod("2026-08") concurrently — all three read false from the database because no pod has yet committed a billing_completed record. This is a classic TOCTOU (time-of-check to time-of-use) race: the check and the subsequent work are not atomic across pods.

All three pods proceed past the guard and begin iterating over 500 customers. Pod 1 calls UUID.randomUUID() for customer 1 and gets UUID_A_1. Pod 2 calls UUID.randomUUID() for the same customer and gets UUID_B_1. Pod 3 gets UUID_C_1. All three pods send POST /v1/charges for customer 1 with three different idempotency keys. Stripe creates ch_A_1, ch_B_1, and ch_C_1 — three charges on the same customer’s card for the same billing period. Across 500 customers, this produces up to 1,500 charges (if all three pods complete their iteration without being interrupted).

The TOCTOU race is not prevented by adding synchronized to runMonthlyBilling(). synchronized serializes within a single JVM, not across three separate processes. With synchronized, each pod is guaranteed that only one of its own threads runs runMonthlyBilling() at a time — but all three pods’ single scheduled threads still run concurrently.

The subtler variant: @Scheduled triggers a Spring Batch job with in-memory MapJobRepository — no cross-pod JobRepository coordination — three independent job instances run simultaneously

Spring Batch adds a JobRepository abstraction to track job executions, prevent concurrent re-runs of the same job, and record job status. When a Spring Boot application includes spring-batch and an embedded H2 dependency (commonly added for local testing), Spring Boot autoconfigures an in-memory H2-backed JobRepository in non-production profiles. The in-memory H2 is per-JVM, not shared across pods:

// UNSAFE: @Scheduled triggers a Spring Batch job with in-memory JobRepository.
// Each pod has its own in-memory H2 instance with its own JobRepository.
// There is no cross-pod job execution record — all three pods run the job independently.

@Component
public class BillingJobScheduler {

    private final JobLauncher jobLauncher;
    private final Job monthlyBillingJob;

    @Scheduled(cron = "0 0 2 1 * *")
    public void scheduleBilling() throws Exception {
        String billingPeriod = YearMonth.now().toString();
        // Each pod launches its own job instance with its own in-memory JobRepository.
        // Spring Batch's "only one running job instance per parameters" check is per-JVM.
        // Pod 1 sees no running instance in its own H2 — launches job.
        // Pod 2 sees no running instance in its own H2 — launches job.
        // Pod 3 sees no running instance in its own H2 — launches job.
        JobParameters params = new JobParametersBuilder()
            .addString("billingPeriod", billingPeriod)
            .toJobParameters();
        jobLauncher.run(monthlyBillingJob, params);
    }
}

Even if the Spring Batch job’s ItemWriter uses a stable idempotency key derived from customer ID and billing period (fixing the UUID-per-pod problem), three pod-local job instances still create three sets of Stripe API calls. The stable key prevents Stripe from creating duplicate charges — attempts 2 and 3 return ch_A from the idempotency cache — but it creates three separate network round-trips per customer, three sets of Stripe API rate-limit consumption, and three writes to the billing audit table. For 500 customers at three pods, that is 1,500 Stripe API calls instead of 500.

The fix for failure mode 3

Cross-pod @Scheduled coordination requires a shared lock that all pods observe. There are two standard approaches in Spring Boot: ShedLock for annotation-based distributed lock on @Scheduled methods, and a PostgreSQL advisory lock acquired at the start of the billing method:

// Approach 1: ShedLock — distributed lock via shared JDBC table.
// Only the pod that acquires the lock runs the @Scheduled method.
// Other pods see the lock held and skip for the duration of lockAtMostFor.

// pom.xml:
// <dependency>
//   <groupId>net.javacrumbs.shedlock</groupId>
//   <artifactId>shedlock-spring</artifactId>
// </dependency>
// <dependency>
//   <groupId>net.javacrumbs.shedlock</groupId>
//   <artifactId>shedlock-provider-jdbc-template</artifactId>
// </dependency>

// SQL: CREATE TABLE shedlock (name VARCHAR(64) PRIMARY KEY, lock_until TIMESTAMP,
//      locked_at TIMESTAMP, locked_by VARCHAR(255));

@Component
public class BillingJob {

    @Scheduled(cron = "0 0 2 1 * *")
    @SchedulerLock(
        name = "monthlyBillingJob",
        lockAtLeastFor = "PT10M",   // Held for at least 10 minutes — prevents rapid re-fire.
        lockAtMostFor = "PT4H"      // Released after 4 hours if the pod crashes mid-run.
    )
    public void runMonthlyBilling() {
        String billingPeriod = YearMonth.now().toString();
        List<Customer> customers = customerRepository.findAllActive();
        for (Customer customer : customers) {
            // Safe: stableKey() — same value across all pods for the same customer and period.
            String idempotencyKey = stableKey(customer.getId(), billingPeriod);
            stripeClient.charge(customer, idempotencyKey);
        }
        billingRepository.markCompleted(billingPeriod);
    }
}

// ShedLock configuration:
@Bean
public LockProvider lockProvider(DataSource dataSource) {
    return new JdbcTemplateLockProvider(
        JdbcTemplateLockProvider.Configuration.builder()
            .withJdbcTemplate(new JdbcTemplate(dataSource))
            .usingDbTime()  // Use database time to avoid clock skew between pods.
            .build()
    );
}

// Approach 2: pg_try_advisory_lock() as a cross-pod distributed mutex.
// First pod to acquire the advisory lock proceeds. Other pods call pg_try_advisory_lock()
// on the same key and receive false — they log and return without billing.
// Advisory lock is automatically released when the database connection closes
// (session-level lock) or when the transaction commits (transaction-level lock).

@Component
public class BillingJob {

    private final JdbcTemplate jdbcTemplate;

    @Scheduled(cron = "0 0 2 1 * *")
    public void runMonthlyBilling() {
        String billingPeriod = YearMonth.now().toString();
        long lockKey = Math.abs(("spring-monthly-billing:" + billingPeriod).hashCode());

        // pg_try_advisory_lock() returns true for the one pod that acquires the lock.
        // All other pods receive false immediately — no blocking wait.
        Boolean acquired = jdbcTemplate.queryForObject(
            "SELECT pg_try_advisory_lock(?)", Boolean.class, lockKey);

        if (!acquired) {
            log.info("Billing lock held by another pod for {}. Skipping.", billingPeriod);
            return;
        }

        try {
            // Pre-flight: idempotent content-hash key + ON CONFLICT DO NOTHING
            // as authoritative cluster-wide billing mutex — backstop for rolling deploys
            // and any code paths that bypass the advisory lock.
            String billingPeriodFull = billingPeriod;
            List<Customer> customers = customerRepository.findAllActive();
            for (Customer customer : customers) {
                String idempotencyKey = stableKey(customer.getId(), billingPeriodFull);
                boolean claimed = billingRepository.claimSlot(
                    customer.getId(), billingPeriodFull, idempotencyKey);
                if (!claimed) continue;
                stripeClient.charge(customer, idempotencyKey);
            }
            billingRepository.markCompleted(billingPeriodFull);
        } finally {
            jdbcTemplate.execute("SELECT pg_advisory_unlock(" + lockKey + ")");
        }
    }
}

With ShedLock, the first pod to fire at 02:00:00 UTC acquires the ShedLock row via UPDATE shedlock SET lock_until = now() + lockAtMostFor WHERE name = 'monthlyBillingJob' AND lock_until < now(). The other two pods execute the same UPDATE and receive 0 rows affected (lock already held), skip the method body, and return. ShedLock’s lockAtMostFor ensures the lock is released if the pod that holds it crashes mid-billing run before calling @SchedulerLock’s release mechanism. With pg_try_advisory_lock(), the PostgreSQL server guarantees that at most one session holds an advisory lock on a given key at any instant — the server enforces the mutual exclusion, not the application. The content-hash key with ON CONFLICT DO NOTHING acts as the backstop: even if a rolling deploy starts a new pod during a billing run and the new pod escapes the advisory lock (because the lock was released by a pod that completed, before the new pod ran its scheduled check), the database constraint prevents double-billing for any customer already processed.

For Spring Batch, the fix is to configure a shared JobRepository backed by the production PostgreSQL database rather than an in-memory H2:

// Safe: shared JDBC JobRepository ensures cross-pod job uniqueness.
// Spring Batch uses SELECT FOR UPDATE on the BATCH_JOB_INSTANCE table to prevent
// concurrent launches of the same job with the same parameters across all pods.

@Configuration
@EnableBatchProcessing(dataSourceRef = "batchDataSource", transactionManagerRef = "batchTransactionManager")
public class BatchConfig {

    @Bean
    @Primary
    public DataSource batchDataSource(DataSource primaryDataSource) {
        // Use the production PostgreSQL datasource — shared across all pods.
        // Not H2 autoconfigured in-memory — that is per-JVM.
        return primaryDataSource;
    }
}
// Spring Batch 5+ with @EnableBatchProcessing on a PostgreSQL datasource
// creates a PlatformTransactionManager-backed JobRepository that uses
// BATCH_JOB_INSTANCE.JOB_NAME + BATCH_JOB_INSTANCE.JOB_KEY uniqueness
// to prevent concurrent launches of the same job instance across pods.

Gap analysis: other Spring Boot + Stripe failure modes

The three failure modes above represent the most common production paths. Several additional patterns are worth noting:

• @Retryable on a Feign client method — Spring Cloud OpenFeign generates a proxy for the Feign interface. If the interface method is annotated with @Retryable and the Feign RequestInterceptor adds Idempotency-Key: UUID.randomUUID(), each @Retryable retry triggers a new Feign request, which runs the RequestInterceptor, which generates a new UUID. Same failure as FM2 with RestTemplate, different HTTP client underneath.
• @Retryable on a CompletableFuture-returning method — @Retryable intercepts the method call and receives the CompletableFuture return value. It cannot observe exceptions that occur inside the future’s async computation. @Retryable only retries if the method itself (not the future) throws a retryable exception. This means retry logic must be inside the future’s exceptionally() or handle() stage — where UUID inside the retry lambda re-evaluates per retry, producing ch_B.
• @Async + @Retryable proxy interaction — both annotations create AOP proxies. @Async submits the method execution to an executor and returns a Future immediately. @Retryable’s retry logic expects to call proceed() synchronously and observe the exception on the call stack. The two proxies’ interaction is not straightforward: @Retryable sees the executor submission as a successful return (no exception), and does not trigger retries on async exceptions. Retry logic must be implemented explicitly inside the async method body.
• Spring Batch FaultTolerantStepBuilder.retry() on an ItemWriter that calls Stripe per item — Spring Batch’s item-level retry re-calls ItemWriter.write(singleItemChunk) for the failed item. If UUID.randomUUID() is inside write(), the retry call generates a new UUID for the item. Fix: compute stableKey(customer.getId(), billingPeriod) inside write() — safe because stableKey() is deterministic; the retry call computes the same key and Stripe returns the cached result.
• Resilience4j @CircuitBreaker + @Retry stacking — when both annotations are present on a billing method, the Resilience4j AOP interceptors stack. @Retry re-invokes the method body. @CircuitBreaker counts failures and may open the circuit. If UUID.randomUUID() is inside the method body, each @Retry re-invocation produces a new key. The circuit-breaker’s half-open test call — a single probe attempt after the circuit opens — also re-invokes the method body with a new UUID. Stripe creates a new charge on the half-open probe even if the circuit opened after ch_A was committed. Fix: same stableKey() derived from method parameters.

Summary

The pattern connecting all three: the idempotency key must be derived from the business intent of the billing operation — customer ID, billing period, vendor namespace — not from any ephemeral runtime value that re-evaluates between AOP proxy re-invocations, interceptor re-executions, or concurrent pod executions. UUID.randomUUID() at method entry, UUID.randomUUID() inside a ClientHttpRequestInterceptor, and UUID.randomUUID() in a per-JVM scheduled job all produce different values each time the code path that computes them is executed. A key derived from sha256(customerId + ":" + billingPeriod + ":spring-billing")[:32] produces the same 32-character hex string on every @Retryable proceed() invocation, on every RestTemplate interceptor execution for the same billing call, and on every Kubernetes pod for the same customer — stable across all three failure modes. Backing it with PostgreSQL-level UNIQUE (customer_id, billing_period) and a cross-pod distributed lock moves the deduplication guarantee out of ephemeral per-JVM state and into durable shared storage that survives rolling deploys, connection resets, and concurrent @Scheduled firings.

The vault key spend cap adds a hard financial boundary as a last line of defence: a per-billing-period vault key issued with a max_amount set to expected_total × 1.10 caps the maximum spend that any combination of @Retryable AOP re-invocations, interceptor re-executions, and multi-pod billing races can produce. Once the cap is reached, Stripe rejects further charges with a 402 Payment Required on the vault key, containing the blast radius to a known maximum regardless of how many AOP proxy retries, interceptor invocations, or concurrent billing pods are in flight.