Keybrake

Spring Retry and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

By Keybrake ·

Spring Retry’s @Retryable annotation wraps the annotated bean method in a Spring AOP proxy. When the method throws a retryable exception, the proxy re-invokes the underlying method from its first line on each retry attempt. UUID.randomUUID() inside the method body is a call expression that evaluates on every method invocation — not once before the proxy intercepts the call. The initial attempt creates ch_A before a StripeException wrapping a socket timeout; the first retry evaluates a new UUID.randomUUID(), causing Stripe to create ch_B. Three Spring Retry-specific Stripe billing failure modes: @Retryable re-invokes the annotated method body with fresh UUID.randomUUID() per retry attempt; RetryTemplate.execute() re-calls the RetryCallback.doWithRetry() lambda per attempt — subtler: including RetryContext.getRetryCount() in the idempotency key construction produces a structurally distinct key per attempt by design (count=0 on attempt 1, count=1 on attempt 2) — a guaranteed duplicate; and @Scheduled(fixedRate=86400000) billing method on a Kubernetes Deployment with replicas:3 fires on all three pods simultaneously — Spring Retry cannot coordinate across JVMs — each pod generates a distinct UUID.randomUUID() per customer — ch_A, ch_B, and ch_C per customer per billing period.

This post covers all three failure modes with Java code (Spring Retry 2.x, Spring Boot 3.x), content-hash stable keys passed as method parameters, RetryContext attribute storage for the programmatic RetryTemplate API, pg_try_advisory_lock() for cluster-wide @Scheduled serialization, and pre-flight PostgreSQL ON CONFLICT DO NOTHING as a durable billing mutex — plus per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For Spring Batch’s FaultTolerantStep and ItemWriter re-invocation patterns, see the Spring Integration and Spring Batch and Stripe Integration post. For Apache Camel’s onException().maximumRedeliveries() re-invocation, see the Apache Camel and Stripe Integration post. Spring Retry is the standalone annotation-based library distributed separately from Spring Batch; the AOP proxy mechanism and the duplicate-charge failure mode are distinct from what either of those covers.

Failure mode 1: @Retryable re-invokes the annotated method body on each retry attempt — UUID.randomUUID() inside the method body evaluates fresh per invocation — initial attempt creates ch_A before StripeException — first retry creates ch_B

Spring Retry’s @Retryable annotation relies on Spring AOP to intercept method calls on the annotated bean. When the annotated method throws an exception that matches the configured retryFor (or include) exception classes, the AOP interceptor catches it, waits for the configured backoff delay, and then calls the original method again — from its first line. The method body is executed from top to bottom on every attempt, including all variable initializations, all constructor calls, and all static method calls like UUID.randomUUID().

This is not a misconfiguration or a misuse of the annotation. It is how @Retryable is designed to behave: re-execute the method body on each retry attempt. The problem emerges when the method body generates the Stripe idempotency key using a non-deterministic call expression that produces a different value on each invocation:

// UNSAFE: UUID.randomUUID() inside a @Retryable method body.
// Spring Retry re-invokes the entire method body on each retry attempt.
// Each invocation of UUID.randomUUID() produces a fresh random UUID.

import com.stripe.model.Charge;
import com.stripe.net.RequestOptions;
import com.stripe.param.ChargeCreateParams;
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
import org.springframework.stereotype.Service;

import java.util.UUID;

@Service
public class BillingService {

    @Retryable(
        retryFor = { com.stripe.exception.StripeException.class },
        maxAttempts = 4,
        backoff = @Backoff(delay = 1000, multiplier = 2.0)
    )
    public Charge chargeCustomer(String customerId, long amountCents, String billingPeriod)
            throws Exception {

        // UNSAFE: UUID.randomUUID() is a call expression evaluated on every method
        // invocation. Spring Retry's AOP proxy re-invokes this method body on each
        // retry attempt. Each attempt generates a distinct UUID.
        //
        // Attempt 1: UUID = "7a3f1b2c-4d5e-6f7a-8b9c-0d1e2f3a4b5c" → ch_A
        // Attempt 2: UUID = "c9d8e7f6-5a4b-3c2d-1e0f-9a8b7c6d5e4f" → ch_B ← duplicate
        // Attempt 3: UUID = "b2a1c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d" → ch_C ← triplicate
        // Attempt 4: UUID = "91e0f1a2-b3c4-d5e6-f7a8-b9c0d1e2f3a4" → ch_D ← quadruplicate
        String idempotencyKey = UUID.randomUUID().toString();

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(amountCents)
            .setCurrency("usd")
            .setCustomer(customerId)
            .build();

        // POST /v1/charges — attempt 1: ch_A committed before SocketTimeoutException fires.
        // @Retryable AOP proxy catches StripeException, waits backoff delay,
        // calls chargeCustomer() again from first line.
        // Attempt 2: new UUID — Stripe sees different idempotency key — creates ch_B.
        return Charge.create(params, options);
    }
}

// Execution timeline for customer "cust_123", billingPeriod="2026-10":
// 10:00:00.000  Attempt 1: UUID_A → POST /v1/charges
// 10:00:29.998  Stripe commits ch_A (charges.created event fired)
// 10:00:30.000  SocketTimeoutException → StripeException caught by @Retryable proxy
// 10:00:31.000  Attempt 2: UUID_B → POST /v1/charges (entirely new idempotency key)
// 10:00:31.100  Stripe: new key → processes as new charge → ch_B committed
// Result: customer "cust_123" billed twice for October 2026.

The critical distinction: @Retryable’s AOP proxy intercepts the call site to chargeCustomer(), not the call site to UUID.randomUUID(). The caller of chargeCustomer() calls the method once; the AOP proxy is responsible for calling the underlying implementation multiple times on failure. The method implementation has no way to distinguish “attempt 1” from “attempt 2” from the caller’s perspective — the AOP proxy handles that logic transparently. What the method body does know is that it starts executing from line one every single time. UUID.randomUUID() on line one of the method body evaluates on every attempt.

Why moving UUID.randomUUID() above the method call does not help if the caller is also retried

A natural instinct is to move UUID.randomUUID() to the caller, before the call to the @Retryable method:

// CALLER — first attempt at a fix: generate UUID at the call site.
// This solves the case where the caller calls chargeCustomer() directly.
// It does NOT solve the case where the caller itself is inside a retry loop.

String idempotencyKey = UUID.randomUUID().toString(); // generated once at call site
billingService.chargeCustomer(customerId, amountCents, billingPeriod, idempotencyKey);

// WORKS if the caller is called exactly once.
// FAILS if the caller is:
//   - Another @Retryable method (the outer @Retryable re-invokes the entire
//     outer method body, including the UUID.randomUUID() line above the inner call)
//   - A @Scheduled method with retryTemplate.execute() wrapping the chargeCustomer() call
//   - A Kafka consumer listener with @RetryableTopic (re-invokes the listener method)
//   - A loop over customers where the loop body is wrapped in a retry template

Moving UUID.randomUUID() to the caller’s scope is a correct fix only when the caller is itself not retried. If the call to chargeCustomer() is inside another @Retryable method, a RetryTemplate.execute() callback, or any other construct that re-invokes its body on failure, the UUID is regenerated at the outer retry boundary even if it was “only generated once” per inner invocation. The fix must ensure that the idempotency key is a deterministic function of the billing request’s stable fields, not of when or how many times the code has been called.

The correct fix is a content-hash key computed from fields that uniquely and stably identify the billing operation — passed as a parameter into the @Retryable method so the method body reads a stable value rather than generating a new one:

// SAFE: stable content-hash key computed BEFORE the @Retryable method is called.
// The key is passed as a parameter. The method body reads the parameter on every
// retry invocation — the AOP proxy cannot change parameter values between retries.

import org.apache.commons.codec.digest.DigestUtils;
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
import org.springframework.stereotype.Service;

@Service
public class BillingService {

    @Retryable(
        retryFor = { com.stripe.exception.StripeException.class },
        maxAttempts = 4,
        backoff = @Backoff(delay = 1000, multiplier = 2.0)
    )
    public Charge chargeCustomer(String customerId, long amountCents,
                                  String billingPeriod, String idempotencyKey)
            throws Exception {

        // SAFE: idempotencyKey is a parameter — same value on every retry invocation.
        // AOP proxy calls this method again with the same arguments on each retry.
        // The parameter value does not change between attempts.
        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(amountCents)
            .setCurrency("usd")
            .setCustomer(customerId)
            .build();

        // Attempt 1: key="a3f1b2c4d5e6f7a8..." → ch_A committed
        // StripeException fires (socket timeout) → @Retryable proxy retries
        // Attempt 2: key="a3f1b2c4d5e6f7a8..." (same parameter) → Stripe cache hit → ch_A
        // Attempt 3: key="a3f1b2c4d5e6f7a8..." (same parameter) → Stripe cache hit → ch_A
        // No duplicate charge created regardless of how many times the proxy retries.
        return Charge.create(params, options);
    }
}

// Caller: compute stable key BEFORE calling the @Retryable method.
// The key must not include any non-deterministic element.
// Do not include: UUID.randomUUID(), System.currentTimeMillis(), attempt counter,
// hostname, JVM startup timestamp, or any value that changes between invocations.
@Service
public class BillingOrchestrator {

    private final BillingService billingService;

    public void runBillingRun(String billingPeriod, List<Customer> customers) throws Exception {
        for (Customer customer : customers) {
            // Compute stable key BEFORE the @Retryable call — deterministic hash of
            // fields that uniquely identify this billing operation.
            String stableKey = DigestUtils.sha256Hex(
                customer.getId() + ":" + billingPeriod + ":spring-retry-billing"
            ).substring(0, 32);

            // Pass the stable key as a parameter — @Retryable proxy will use
            // the same key on every retry attempt.
            billingService.chargeCustomer(
                customer.getStripeId(),
                customer.getAmountCents(),
                billingPeriod,
                stableKey
            );
        }
    }
}

// Key construction rule: sha256(customerId:billingPeriod:spring-retry-billing)[:32]
// — a pure function of stable billing fields.
// Same output for the same customer and billing period regardless of:
//   - How many times the @Retryable proxy has retried the method
//   - Which attempt number is currently executing
//   - What time the current attempt started
//   - Which pod is running the current attempt

Spring Retry’s AOP proxy passes the original method arguments to the retried invocation unchanged. If the caller computes a stable key and passes it as a parameter, the proxy ensures the method body receives the same key string on every attempt. The @Retryable annotation has no mechanism to transform parameters between retry attempts — it simply re-invokes the method with the same arguments. This parameter-passing approach is therefore reliable across all maxAttempts values and backoff configurations.

Failure mode 2: RetryTemplate.execute() re-calls the RetryCallback.doWithRetry() lambda per attempt — UUID.randomUUID() inside the lambda evaluates fresh per invocation — subtler: including RetryContext.getRetryCount() in the key construction produces a structurally distinct key per attempt by design

Spring Retry’s programmatic API, RetryTemplate, gives developers finer control over retry policy, backoff policy, and retry listeners. A typical usage wraps the Stripe API call in a RetryCallback lambda and passes it to RetryTemplate.execute(). The RetryCallback.doWithRetry(RetryContext) method is called on every attempt — including all retries — by the template’s internal retry loop. Any non-deterministic expression inside the lambda body, including UUID.randomUUID(), evaluates on every call to doWithRetry():

// UNSAFE: UUID.randomUUID() inside a RetryCallback lambda.
// RetryTemplate.execute() re-invokes doWithRetry() on every retry attempt.
// UUID.randomUUID() is a call expression inside the lambda body —
// it evaluates on each invocation of doWithRetry().

import org.springframework.retry.RetryCallback;
import org.springframework.retry.support.RetryTemplate;

@Service
public class BillingService {

    private final RetryTemplate retryTemplate;

    public BillingService(RetryTemplate retryTemplate) {
        this.retryTemplate = retryTemplate;
    }

    public Charge chargeCustomerWithRetry(String customerId, long amountCents,
                                           String billingPeriod) throws Exception {
        return retryTemplate.execute((RetryCallback<Charge, Exception>) context -> {
            // UNSAFE: UUID.randomUUID() is evaluated on every call to doWithRetry().
            // RetryTemplate calls doWithRetry() again on each retry attempt.
            // context.getRetryCount() is 0 on attempt 1, 1 on attempt 2, etc. —
            // it can be used to detect which attempt is running, but NOT to build
            // a stable idempotency key (count changes per attempt by definition).

            // Attempt 1: count=0, UUID = "7a3f1b2c-..." → ch_A committed (timeout fires)
            // Attempt 2: count=1, UUID = "c9d8e7f6-..." → ch_B ← duplicate
            // Attempt 3: count=2, UUID = "b2a1c3d4-..." → ch_C ← triplicate
            String idempotencyKey = UUID.randomUUID().toString(); // UNSAFE per invocation

            RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build();
            ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(amountCents)
                .setCurrency("usd")
                .setCustomer(customerId)
                .build();
            return Charge.create(params, options);
        });
    }
}

// RetryTemplate configuration (Spring Boot @Bean):
@Bean
public RetryTemplate retryTemplate() {
    return RetryTemplate.builder()
        .maxAttempts(4)
        .exponentialBackoff(1000, 2.0, 10000)
        .retryOn(com.stripe.exception.StripeException.class)
        .build();
}

Subtler variant: using RetryContext.getRetryCount() in the key construction produces a structurally distinct key per attempt by design

RetryContext is passed as a parameter to doWithRetry() on every call. It provides getRetryCount(), which returns 0 on the first attempt and increments by 1 on each subsequent attempt. A developer who is aware of the UUID.randomUUID() problem may reach for getRetryCount() as a stable identifier — reasoning that it is a deterministic value from the retry context rather than a random call. This reasoning is backwards: getRetryCount() is designed to be different on every attempt, making any key derived from it different on every attempt by construction:

// ALSO UNSAFE: RetryContext.getRetryCount() in the idempotency key.
// Developer intent: "use the retry context to build a stable key per attempt."
// Actual effect: getRetryCount() changes per attempt by design — 0, 1, 2, 3 —
// making the key different on every attempt — ch_A, ch_B, ch_C, ch_D.

return retryTemplate.execute((RetryCallback<Charge, Exception>) context -> {
    int attempt = context.getRetryCount(); // 0 on first, 1 on second, etc.

    // UNSAFE: key encodes the attempt number — different key per attempt.
    // Attempt 1 (count=0): sha256("cust_123:2026-10:0") → ch_A committed
    // Attempt 2 (count=1): sha256("cust_123:2026-10:1") → ch_B ← duplicate
    // Attempt 3 (count=2): sha256("cust_123:2026-10:2") → ch_C ← triplicate
    String idempotencyKey = DigestUtils.sha256Hex(
        customerId + ":" + billingPeriod + ":" + attempt
    ).substring(0, 32);

    // ...Stripe API call...
});

// This is structurally worse than UUID.randomUUID() in one sense:
// UUID.randomUUID() has negligible collision probability between two customers'
// keys (128 bits of entropy). The getRetryCount() approach has deterministic
// key variation — attempt 0 and attempt 1 will always produce different keys
// for the same customer and billing period. The duplicate charge is guaranteed
// to occur on every retry attempt, not with negligible probability.

The fix for the RetryTemplate pattern uses RetryContext’s attribute storage to carry the stable key across invocations — computed once, stored on the first call to doWithRetry() (when getRetryCount() is 0), and retrieved on subsequent calls:

// SAFE OPTION 1: compute stable key before RetryTemplate.execute(), capture as
// effectively-final local in the lambda. The lambda closes over the stable value.
// RetryTemplate re-invokes doWithRetry() with the same closed-over reference.

public Charge chargeCustomerWithRetry(String customerId, long amountCents,
                                       String billingPeriod) throws Exception {
    // Compute stable key BEFORE execute() — outside the retry loop entirely.
    // This is a pure function of stable billing fields — same output every time.
    final String stableKey = DigestUtils.sha256Hex(
        customerId + ":" + billingPeriod + ":spring-retry-billing"
    ).substring(0, 32);

    return retryTemplate.execute((RetryCallback<Charge, Exception>) context -> {
        // SAFE: stableKey is an effectively-final local captured from the enclosing scope.
        // Java lambda closure captures the reference at lambda creation time — before
        // RetryTemplate calls doWithRetry() for the first time.
        // All retry invocations of doWithRetry() see the same stableKey value.
        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(stableKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(amountCents)
            .setCurrency("usd")
            .setCustomer(customerId)
            .build();
        return Charge.create(params, options);
    });
}

// SAFE OPTION 2: store stable key in RetryContext attributes on first attempt,
// retrieve from attributes on subsequent attempts. Useful when the key must be
// computed inside the retry callback for architectural reasons (e.g., the
// stable fields are only available after the first callback invocation).

return retryTemplate.execute((RetryCallback<Charge, Exception>) context -> {
    // On attempt 0: compute and store. On attempts 1+: retrieve stored key.
    String stableKey;
    if (context.getRetryCount() == 0) {
        stableKey = DigestUtils.sha256Hex(
            customerId + ":" + billingPeriod + ":spring-retry-billing"
        ).substring(0, 32);
        context.setAttribute("stripeIdempotencyKey", stableKey);
    } else {
        stableKey = (String) context.getAttribute("stripeIdempotencyKey");
    }

    // Attempt 1 (count=0): stableKey computed, stored, used → ch_A committed (timeout)
    // Attempt 2 (count=1): stableKey retrieved from context attribute → same key → cache hit
    // Attempt 3 (count=2): stableKey retrieved from context attribute → same key → cache hit
    // No duplicate charge created.
    RequestOptions options = RequestOptions.builder()
        .setIdempotencyKey(stableKey)
        .build();
    ChargeCreateParams params = ChargeCreateParams.builder()
        .setAmount(amountCents)
        .setCurrency("usd")
        .setCustomer(customerId)
        .build();
    return Charge.create(params, options);
});

Option 1 (captured effectively-final local) is simpler and preferred when the stable key fields are available before the execute() call — which they almost always are, since the customer ID and billing period are known before the retry loop starts. Option 2 (context attribute storage) is useful when the key computation depends on a value that is only available after the first network call, though this pattern is rare for Stripe billing. The RetryContext object’s getAttribute() / setAttribute() pair is thread-safe within a single retry execution because Spring Retry runs all attempts for a given execute() call sequentially on the same thread (for synchronous retry).

One additional RetryTemplate pitfall: Spring Retry’s stateful retry feature (RetryTemplate with a RetryState object, used primarily for message-driven scenarios) changes how the retry context is stored between calls to execute() (persisted in a RetryContextCache keyed by the retry state key), but it does not change the fact that doWithRetry() is called again on each attempt. The stateful retry context is retrieved from the cache, making it possible to resume a retry sequence across JVM restarts or message redeliveries — but the lambda body still executes from top to bottom on every doWithRetry() call. UUID.randomUUID() inside the lambda still produces a new value on every attempt even in stateful retry mode.

Failure mode 3: @Retryable on a billing method called from @Scheduled(fixedRate=86400000) on Kubernetes replicas:3 — Spring Retry cannot coordinate across JVMs — all three pods enter the billing method simultaneously — each generates distinct UUID.randomUUID() per customer — ch_A, ch_B, and ch_C per customer per billing period

Spring’s @Scheduled(fixedRate=86400000) runs independently in each JVM. A Kubernetes Deployment with replicas:3 starts three separate Spring application contexts, each with its own TaskScheduler managing its own @Scheduled method executions. When the fixed-rate interval fires, all three pods execute the billing method within the same scheduling window. Spring Retry’s @Retryable annotation is an AOP interceptor scoped to a single JVM — it has no mechanism to detect or prevent concurrent execution on other pods.

The combination of @Scheduled multi-pod firing and @Retryable with UUID.randomUUID() in the method body creates a two-level duplicate charge problem: within a single pod, retries produce ch_A and ch_B for the same customer; across pods, the simultaneous first attempts produce ch_A, ch_B, and ch_C for the same customer from three different JVMs before any pod’s first attempt has completed:

// UNSAFE: @Scheduled billing method + @Retryable + UUID.randomUUID() on multi-pod deployment.
// @Scheduled fires independently on every Kubernetes pod — no cluster-wide coordination.
// @Retryable re-invokes the method body on exception within each pod.
// UUID.randomUUID() produces a distinct value per method invocation per pod.

@Service
public class ScheduledBillingService {

    @Scheduled(fixedRate = 86_400_000) // fires once every 24 hours in each JVM
    public void runDailyBilling() {
        String billingPeriod = LocalDate.now().toString().substring(0, 7); // "2026-10"
        List<Customer> customers = customerRepository.findAllActive();

        for (Customer customer : customers) {
            try {
                chargeSingleCustomer(customer, billingPeriod);
            } catch (Exception e) {
                log.error("Billing failed for customer {}", customer.getId(), e);
            }
        }
    }

    @Retryable(
        retryFor = { com.stripe.exception.StripeException.class },
        maxAttempts = 3,
        backoff = @Backoff(delay = 2000, multiplier = 2.0)
    )
    public Charge chargeSingleCustomer(Customer customer, String billingPeriod)
            throws Exception {

        // UNSAFE: UUID.randomUUID() inside @Retryable method body.
        // Pod 1 at 00:00:00.000: UUID_A1 → ch_A for cust_123
        // Pod 2 at 00:00:00.003: UUID_B1 → ch_B for cust_123 ← simultaneous duplicate
        // Pod 3 at 00:00:00.007: UUID_C1 → ch_C for cust_123 ← simultaneous duplicate
        // If Pod 1 also retries on exception: UUID_A2 → ch_D for cust_123 ← retry duplicate
        String idempotencyKey = UUID.randomUUID().toString();

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(customer.getAmountCents())
            .setCurrency("usd")
            .setCustomer(customer.getStripeId())
            .build();
        return Charge.create(params, options);
    }
}

// Execution timeline for customer "cust_123", billingPeriod="2026-10":
// All three pods start their @Scheduled task at approximately the same time
// (fixedRate starts from application startup — pods start within milliseconds
// of each other in a rolling deploy or from a common deployment time).
//
// 00:00:00.000  Pod 1: runDailyBilling() → chargeSingleCustomer() → UUID_A1 → POST /v1/charges
// 00:00:00.003  Pod 2: runDailyBilling() → chargeSingleCustomer() → UUID_B1 → POST /v1/charges
// 00:00:00.007  Pod 3: runDailyBilling() → chargeSingleCustomer() → UUID_C1 → POST /v1/charges
// 00:00:00.900  Stripe commits ch_A (Pod 1), ch_B (Pod 2), ch_C (Pod 3) — three charges
// Result: customer "cust_123" billed three times for October 2026.
//
// Even if UUID.randomUUID() is replaced with a content-hash key, all three pods
// generate the SAME stable key and make the SAME Stripe API call simultaneously.
// Stripe's idempotency cache prevents three charges from the same key, but only
// if all three requests arrive within Stripe's concurrent request deduplication
// window. Under Stripe's actual idempotency semantics, concurrent requests with
// the same key may all succeed and create multiple charges if the first request
// has not yet been committed to the idempotency cache when the second arrives.
// The pre-flight database guard is the correct solution for multi-pod coordination.

An important nuance: replacing UUID.randomUUID() with a content-hash stable key does not fully fix the multi-pod problem, even though it prevents retry-level duplicates within a single pod. When three pods make simultaneous POST /v1/charges requests with the same idempotency key within Stripe’s concurrent request window, Stripe may process all three before the first is committed to its idempotency cache. Stripe’s idempotency guarantee applies to sequential requests with the same key, not to concurrent requests arriving faster than the cache write can complete. The correct multi-pod fix is cluster-wide coordination at the application layer, before the Stripe API call is made.

Fix: pg_try_advisory_lock() at the @Scheduled method entry for cluster-wide serialization, combined with pre-flight ON CONFLICT DO NOTHING as a permanent guard

// SAFE: pg_try_advisory_lock() at @Scheduled method entry — only one pod runs billing.
// Pre-flight ON CONFLICT DO NOTHING as a permanent guard for delayed redeliveries.
// Stable content-hash key passed as parameter to @Retryable method.

@Service
public class ScheduledBillingService {

    private final DataSource dataSource;
    private final BillingService billingService;

    // Advisory lock key: stable integer derived from the billing function name.
    // All pods that connect to the same PostgreSQL instance will contend on
    // the same lock key. Only one connection can hold a given advisory lock.
    private static final long BILLING_LOCK_KEY = "daily-billing".hashCode() & 0x7FFFFFFFL;

    @Scheduled(fixedRate = 86_400_000)
    public void runDailyBilling() {
        String billingPeriod = LocalDate.now().toString().substring(0, 7);

        try (Connection conn = dataSource.getConnection()) {
            // pg_try_advisory_lock() returns true if the lock was acquired,
            // false if another pod already holds it (non-blocking).
            // The lock is automatically released when the connection is closed.
            try (PreparedStatement lockStmt = conn.prepareStatement(
                    "SELECT pg_try_advisory_lock(?)")) {
                lockStmt.setLong(1, BILLING_LOCK_KEY);
                ResultSet rs = lockStmt.executeQuery();
                rs.next();
                boolean lockAcquired = rs.getBoolean(1);

                if (!lockAcquired) {
                    // Another pod is already running billing for this period.
                    // This pod exits immediately without attempting any charges.
                    log.info("Billing lock held by another pod — skipping this firing.");
                    return;
                }
            }

            // Lock acquired — only this pod proceeds to billing.
            // All other pods returned above without reaching this point.
            try {
                runBillingUnderLock(billingPeriod, conn);
            } finally {
                // Explicit unlock before connection close (defense in depth —
                // PostgreSQL releases the lock automatically on connection close,
                // but explicit unlock is clearer and handles connection pooling).
                try (PreparedStatement unlockStmt = conn.prepareStatement(
                        "SELECT pg_advisory_unlock(?)")) {
                    unlockStmt.setLong(1, BILLING_LOCK_KEY);
                    unlockStmt.executeQuery();
                }
            }
        } catch (SQLException e) {
            log.error("Database error in billing scheduler", e);
        }
    }

    private void runBillingUnderLock(String billingPeriod, Connection conn) {
        List<Customer> customers = customerRepository.findAllActive();

        for (Customer customer : customers) {
            // Pre-flight guard: INSERT billing attempt — skip if already billed this period.
            // This guard fires even if the advisory lock is not used (defense in depth).
            // Survives crash-recovery, manual retries, and any lock expiry edge case.
            String stableKey = DigestUtils.sha256Hex(
                customer.getId() + ":" + billingPeriod + ":spring-retry-billing"
            ).substring(0, 32);

            boolean inserted = insertBillingAttempt(conn, customer.getId(), billingPeriod, stableKey);
            if (!inserted) {
                // Row already exists — this customer was billed for this period.
                // Skip to next customer. Do not call Stripe.
                continue;
            }

            try {
                // SAFE: pass stable key as parameter to @Retryable method.
                // @Retryable AOP proxy re-invokes chargeSingleCustomer() with the
                // same stableKey parameter on every retry attempt.
                billingService.chargeSingleCustomer(
                    customer.getStripeId(),
                    customer.getAmountCents(),
                    billingPeriod,
                    stableKey
                );
            } catch (Exception e) {
                log.error("Billing failed for customer {} after all retries",
                    customer.getId(), e);
                // Mark billing attempt as failed in the billing_attempts table
                // so it can be reprocessed by an operator or a separate recovery job.
                markBillingFailed(conn, customer.getId(), billingPeriod);
            }
        }
    }

    private boolean insertBillingAttempt(Connection conn, String customerId,
                                          String billingPeriod, String idempotencyKey) {
        try (PreparedStatement stmt = conn.prepareStatement(
                // ON CONFLICT DO NOTHING: if row exists, INSERT is a no-op.
                // Returns update count 0 (not inserted) or 1 (newly inserted).
                "INSERT INTO billing_attempts (customer_id, billing_period, idempotency_key, created_at) " +
                "VALUES (?, ?, ?, NOW()) " +
                "ON CONFLICT (customer_id, billing_period) DO NOTHING")) {
            stmt.setString(1, customerId);
            stmt.setString(2, billingPeriod);
            stmt.setString(3, idempotencyKey);
            int rows = stmt.executeUpdate();
            return rows == 1; // true if inserted, false if row already existed
        } catch (SQLException e) {
            log.error("Pre-flight billing_attempts insert failed", e);
            return false; // fail safe: skip this customer on database error
        }
    }
}

// Schema for billing_attempts table:
// CREATE TABLE billing_attempts (
//   customer_id     TEXT        NOT NULL,
//   billing_period  TEXT        NOT NULL,
//   idempotency_key TEXT        NOT NULL,
//   created_at      TIMESTAMPTZ NOT NULL,
//   status          TEXT        NOT NULL DEFAULT 'pending',
//   CONSTRAINT billing_attempts_pkey PRIMARY KEY (customer_id, billing_period)
// );
// UNIQUE (customer_id, billing_period) ensures ON CONFLICT DO NOTHING fires
// on duplicate (customer_id, billing_period) pairs regardless of idempotency_key value.

// @Retryable billing method — reads idempotency key from parameter:
@Service
public class BillingService {

    @Retryable(
        retryFor = { com.stripe.exception.StripeException.class },
        maxAttempts = 4,
        backoff = @Backoff(delay = 1000, multiplier = 2.0)
    )
    public Charge chargeSingleCustomer(String stripeCustomerId, long amountCents,
                                        String billingPeriod, String idempotencyKey)
            throws Exception {

        // SAFE: idempotencyKey is a stable parameter — same value on every retry.
        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(amountCents)
            .setCurrency("usd")
            .setCustomer(stripeCustomerId)
            .build();
        return Charge.create(params, options);
    }
}

The advisory lock approach serializes billing across the Kubernetes cluster at the database level. Only one pod acquires the lock; the other two pods detect the lock is held and exit the @Scheduled method immediately without attempting any charges. The lock is released when the connection is closed, which happens at the end of the try-with-resources block. If the pod holding the lock crashes mid-billing-run, PostgreSQL releases the advisory lock when the database connection closes (on pod termination), allowing the next pod’s @Scheduled firing to acquire the lock and resume. The pre-flight ON CONFLICT DO NOTHING guard ensures that customers already billed before the crash are skipped in the resumed run.

Alternative: Kubernetes replicas:1 for the billing worker pod

The most straightforward mitigation for the multi-pod @Scheduled problem is to deploy the billing scheduler on a separate Deployment with replicas:1. A single-pod scheduler has no concurrent execution problem by construction. The advisory lock is then an optional defense-in-depth measure rather than a primary correctness mechanism. The pre-flight ON CONFLICT DO NOTHING guard remains valuable as a permanent durable guard against delayed redeliveries and operator-triggered retries:

# Kubernetes Deployment for billing scheduler — single replica.
# The billing @Scheduled method fires exactly once per interval, in exactly one pod.
# No pg_try_advisory_lock() required for cluster-wide scheduling coordination.
# Pre-flight ON CONFLICT DO NOTHING remains as a durable guard against crash recovery.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-scheduler
  labels:
    app: billing-scheduler
spec:
  replicas: 1          # single instance — no concurrent @Scheduled execution
  selector:
    matchLabels:
      app: billing-scheduler
  template:
    metadata:
      labels:
        app: billing-scheduler
    spec:
      containers:
        - name: billing-scheduler
          image: billing-service:latest
          # This pod runs only the @Scheduled billing method.
          # The main API service runs as a separate Deployment with replicas:N.
          # Scheduling logic is isolated from the API serving layer.

# Main API service — multiple replicas, no @Scheduled billing annotation.
# @Scheduled on this Deployment would fire on all N replicas.
# All billing @Scheduled methods belong in the billing-scheduler Deployment only.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  replicas: 3          # high availability for API serving — no @Scheduled billing here

Separating the billing scheduler from the API service into a dedicated single-replica deployment also has architectural benefits beyond duplicate charge prevention: it allows independent scaling (the API scales based on request volume; the billing scheduler does not need to scale), independent failure domains (an API deployment issue does not interrupt billing), and clearer operational ownership (the billing scheduler’s pod logs, resource limits, and rollout strategy are managed separately).

Why Stripe’s 24-hour idempotency cache is not the last line of defense

Stripe’s idempotency cache matches requests by key within a 24-hour sliding window. A request with key K submitted at 10:00:00 UTC will cache-hit for subsequent requests with the same key K submitted before 10:00:00 UTC the following day. After 24 hours, the key expires from Stripe’s cache, and a new request with the same key is treated as a fresh charge.

For monthly billing, a crash-recovery scenario where the Spring application restarts more than 24 hours after the original billing attempt will bypass Stripe’s idempotency cache entirely. The stable content-hash key matches nothing in the cache, and Stripe creates a new charge. The pre-flight ON CONFLICT DO NOTHING guard is immune to Stripe’s cache TTL because it checks your own database, which retains billing records indefinitely. The advisory lock is also immune: it prevents the billing method from reaching the Stripe API call at all when a billing run is already in progress or completed for the billing period.

Guard Prevents TTL Scope
pg_try_advisory_lock() Concurrent billing runs (multi-pod @Scheduled race) Until connection closes or explicit unlock Cluster-wide (same PostgreSQL instance)
ON CONFLICT DO NOTHING pre-flight Duplicate billing attempts (all causes including crash recovery) Permanent (row retained in table) Cluster-wide (same PostgreSQL instance)
Stable content-hash idempotency key parameter Duplicate Stripe charges from @Retryable retries 24 hours (Stripe cache) Stripe’s API layer
Vault key spend cap at expected_total × 1.10 Financial damage from any combination of the above failures Per billing period (configurable) Proxy layer (Keybrake)

Implementation checklist for Spring Retry and Stripe billing

Put a spend cap on your Spring Retry billing service

Keybrake issues scoped vault keys for Stripe, Twilio, and Resend — with per-vendor daily spend caps, allowlisted endpoints, and a one-click kill switch. A Spring service with @Retryable(maxAttempts=4) or a @Scheduled billing method on a three-replica Kubernetes deployment gets a hard financial ceiling even when the idempotency logic has a bug in a retry path you never tested.