Spring Boot Actuator and Stripe Integration: How a Custom @WriteOperation Endpoint Called Twice by an HTTP Retry Generates UUID_B While ch_A Stays Committed on Stripe, ApplicationStartedEvent Recovery Calls Stripe with a Fresh UUID Misreading PENDING as “Charge Never Committed”, and @SchedulerLock TTL Expiry Lets a Second Node Re-process the Batch Window with New UUIDs Alongside Already-Committed Charges

Spring Boot Actuator introduces three deployment patterns that each create a distinct Stripe idempotency failure mode: a custom management endpoint that can be replayed by HTTP clients, a startup recovery hook that misinterprets database state as evidence of Stripe call status, and a distributed scheduler lock whose TTL semantics differ from developer expectations. Each failure produces a second Stripe charge (ch_B) for a customer who has already been charged (ch_A).

This post covers three failure modes that are mechanically distinct from all prior posts in this series (which addressed Spring Kafka Streams EOS, Spring Kafka EOS, Micronaut Kafka EOS, Spring Data R2DBC reactive patterns, Quarkus SmallRye Fault Tolerance, and others). The common thread across all three Actuator modes is that Spring Boot Actuator infrastructure components — the @Endpoint dispatch mechanism, the application lifecycle event system, and the @Scheduled / ShedLock integration — each create execution paths that can invoke your Stripe charge code more than once, and none of them provide built-in protection against the resulting double charge.

Mode 1: Custom @Endpoint @WriteOperation — HTTP client retry on lost response calls the endpoint twice, generating UUID_B while ch_A stays committed on Stripe

The setup

The developer wants an admin-controllable way to trigger a one-off charge for a specific customer — perhaps to collect an overdue invoice or to process a manual order. Rather than building a separate controller, they use Spring Boot Actuator’s @Endpoint abstraction, which exposes the operation over the management HTTP port (default 8080 or a separate port if management.server.port is set) with automatic Spring Security integration and JMX exposure:

@Component
@Endpoint(id = "billing")
public class BillingActuatorEndpoint {

    private final BillingService billingService;

    public BillingActuatorEndpoint(BillingService billingService) {
        this.billingService = billingService;
    }

    @WriteOperation
    public Map<String, Object> chargeCustomer(
            @Selector String customerId,
            String orderId,
            long amountCents) {

        // UUID generated fresh per invocation — this is the bug
        String idempotencyKey = UUID.randomUUID().toString();

        ChargeResult result = billingService.chargeCustomer(
            customerId, orderId, amountCents, idempotencyKey);

        return Map.of(
            "chargeId", result.getChargeId(),
            "status", result.getStatus(),
            "idempotencyKey", idempotencyKey
        );
    }
}

The BillingService calls Stripe synchronously and persists the result:

@Service
public class BillingService {

    private final PaymentRepository paymentRepository;

    @Transactional
    public ChargeResult chargeCustomer(
            String customerId, String orderId,
            long amountCents, String idempotencyKey) {

        // Call Stripe
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(amountCents)
            .setCurrency("usd")
            .setCustomer(customerId)
            .putMetadata("orderId", orderId)
            .build();

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)  // UUID from caller
            .build();

        Charge charge = Charge.create(params, options); // HTTP call to Stripe

        // Persist
        PaymentRecord record = new PaymentRecord(
            orderId, customerId, charge.getId(), amountCents, idempotencyKey);
        paymentRepository.save(record);

        return new ChargeResult(charge.getId(), charge.getStatus());
    }
}

Why this creates a double charge

The endpoint is invoked from an internal administration script — a Python script, a Temporal workflow, or a CI/CD job — that calls POST http://admin.internal:8080/actuator/billing/cust_123 with a JSON body. The calling HTTP client is configured with a timeout and a retry policy. This is where the failure begins.

Consider the following sequence of events:

1. HTTP request arrives at the Actuator endpoint handler.
2. chargeCustomer generates UUID_A = "3f7c...".
3. billingService.chargeCustomer() calls Stripe with Idempotency-Key: 3f7c....
4. Stripe processes the charge and responds with HTTP 200: {"id": "ch_A", "status": "succeeded"}.
5. The application receives Stripe’s response and starts executing paymentRepository.save().
6. The database write is slow (secondary index contention, lock wait, or connection pool exhaustion) — the entire HTTP request has now taken longer than the client’s configured timeout (e.g., 5 seconds).
7. The HTTP client throws SocketTimeoutException or ReadTimeoutException — from the client’s perspective, the request timed out.
8. The client’s retry policy fires: retry once on SocketTimeoutException.
9. A second HTTP request arrives at the Actuator endpoint handler.
10. chargeCustomer generates UUID_B = "9a2e..." — a completely different value.
11. billingService.chargeCustomer() calls Stripe with Idempotency-Key: 9a2e....
12. Stripe has no record of 9a2e... in its idempotency cache — it creates a new charge ch_B.
13. The customer has now been charged twice: ch_A (from the first request) and ch_B (from the second request).

The same failure can occur without a timeout if the load balancer or reverse proxy in front of the management port retries the request on connection reset, if the TCP connection drops after the application sends the response (so the client never receives the 200 and retries), or if the caller is a Temporal workflow activity that treats any exception as a retryable error and re-executes the activity on a different worker.

The developer misconception

“My HTTP client only retries on 5xx responses. The first request returned 200 (which it did — I can see ch_A in the Stripe dashboard). The client will not retry a successful 200 response.”

This is true when the client receives the 200 response. But in the failure scenario described above, the client never received the 200 — the TCP connection timed out before the response arrived. From the client’s perspective, the request timed out: it threw SocketTimeoutException. SocketTimeoutException is not the same as “server returned 5xx”. It is an I/O exception raised before any HTTP status code was read. The client’s retry-on-5xx policy does not even apply — the retry fires on the read timeout, regardless of what the server ultimately returned.

A subtler version of the same misconception: “The endpoint is protected by Spring Security — only authenticated requests can reach it. A second request would need the same bearer token, which the attacker doesn’t have.” Spring Security correctly prevents unauthorized access, but the retry comes from the same authorized caller. Authentication protects against malicious callers; it does not protect against the legitimate caller retrying a request whose response was lost in transit.

The fix

The Stripe Idempotency-Key must be derived from the content of the operation — specifically, from the fields that uniquely identify the charge — not from a random value generated at request time. A SHA-256 hash of (customerId, orderId, amountCents) produces a key that is identical for every invocation that represents the same logical charge:

@WriteOperation
public Map<String, Object> chargeCustomer(
        @Selector String customerId,
        String orderId,
        long amountCents) {

    // Content-hash key — identical on every invocation for the same charge
    String idempotencyKey = computeContentHash(customerId, orderId, amountCents);

    ChargeResult result = billingService.chargeCustomer(
        customerId, orderId, amountCents, idempotencyKey);

    return Map.of(
        "chargeId", result.getChargeId(),
        "status", result.getStatus(),
        "idempotencyKey", idempotencyKey
    );
}

private String computeContentHash(String customerId, String orderId, long amountCents) {
    String input = customerId + "|" + orderId + "|" + amountCents;
    MessageDigest digest = MessageDigest.getInstance("SHA-256");
    byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8));
    return HexFormat.of().formatHex(hash);
}

With a content-hash key, the second HTTP request from the retry reaches the endpoint, computes the same key as the first request (because customerId, orderId, and amountCents are identical), calls Stripe with Idempotency-Key: 3f7c..., and Stripe returns the original ch_A from its 24-hour idempotency cache. No second charge is created.

One subtlety: if the amountCents value is legitimately different on the retry (e.g., an admin corrects the amount before clicking retry in the UI), the content-hash key will be different, and Stripe will create a new charge. This is the correct behavior — the operator explicitly changed the charge amount. The content-hash key distinguishes “same charge retried due to network failure” from “intentionally different charge”.

If the amount can legitimately change between retries and you want to prevent any double charge, add an explicit request-level idempotency token as a required parameter to the @WriteOperation — the caller provides a UUID that it generates once and reuses on every retry of the same logical operation:

@WriteOperation
public Map<String, Object> chargeCustomer(
        @Selector String customerId,
        String orderId,
        long amountCents,
        String requestIdempotencyKey) {  // caller-supplied, stable across retries

    ChargeResult result = billingService.chargeCustomer(
        customerId, orderId, amountCents, requestIdempotencyKey);

    return Map.of(
        "chargeId", result.getChargeId(),
        "status", result.getStatus()
    );
}

Mode 2: ApplicationStartedEvent recovery — startup listener calls Stripe with a fresh UUID.randomUUID() misreading PENDING as “Stripe never called”, creating ch_B alongside already-committed ch_A

The setup

The developer wants to recover from application crashes gracefully. The standard approach is to write a PENDING row to the database before calling Stripe, and then update it to COMPLETED after Stripe responds. On startup, the recovery hook scans for any rows stuck in PENDING state and retries them. This pattern is correct in structure but fatally incorrect in implementation when the retry uses a new UUID.randomUUID() as the Stripe idempotency key:

@Service
@Transactional
public class BillingService {

    private final PaymentAttemptRepository paymentAttemptRepository;

    public ChargeResult chargeCustomer(String customerId, String orderId, long amountCents) {
        // Step 1: write PENDING row before calling Stripe
        PaymentAttempt attempt = new PaymentAttempt();
        attempt.setCustomerId(customerId);
        attempt.setOrderId(orderId);
        attempt.setAmountCents(amountCents);
        attempt.setStatus(PaymentStatus.PENDING);
        attempt.setIdempotencyKey(UUID.randomUUID().toString()); // BUG: random, not content-hash
        paymentAttemptRepository.save(attempt); // committed immediately (REQUIRES_NEW)

        // Step 2: call Stripe
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(amountCents)
            .setCurrency("usd")
            .setCustomer(customerId)
            .build();

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(attempt.getIdempotencyKey())
            .build();

        Charge charge = Charge.create(params, options);

        // Step 3: update to COMPLETED — JVM crash here leaves row PENDING
        attempt.setStatus(PaymentStatus.COMPLETED);
        attempt.setChargeId(charge.getId());
        paymentAttemptRepository.save(attempt);

        return new ChargeResult(charge.getId(), charge.getStatus());
    }
}

The startup recovery listener:

@Component
public class BillingRecoveryListener {

    private final PaymentAttemptRepository paymentAttemptRepository;
    private final BillingService billingService;

    @EventListener(ApplicationStartedEvent.class)
    public void recoverPendingCharges(ApplicationStartedEvent event) {
        List<PaymentAttempt> pending =
            paymentAttemptRepository.findByStatus(PaymentStatus.PENDING);

        for (PaymentAttempt attempt : pending) {
            log.info("Recovering PENDING charge for orderId={}", attempt.getOrderId());
            // BUG: calls the chargeCustomer method which generates a NEW random UUID
            billingService.chargeCustomer(
                attempt.getCustomerId(),
                attempt.getOrderId(),
                attempt.getAmountCents()
            );
        }
    }
}

The crash scenario and why it creates a double charge

Consider the following sequence that produces a PENDING row for a charge that Stripe has already committed:

1. chargeCustomer("cust_1", "order_1", 5000) is called.
2. PaymentAttempt written to the database with idempotencyKey = "uuid-A", status = PENDING. The row is committed immediately in a REQUIRES_NEW transaction.
3. Stripe is called with Idempotency-Key: uuid-A.
4. Stripe processes the charge: ch_A created, HTTP 200 returned.
5. The application receives Stripe’s 200 response and prepares to execute Step 3 (attempt.setStatus(COMPLETED)).
6. JVM crash — power failure, OOM kill, kill -9, or a hardware fault — occurs between receiving Stripe’s HTTP 200 and executing paymentAttemptRepository.save(attempt) to mark the row COMPLETED.
7. The payment_attempts row for order_1 remains permanently in PENDING state in the database.
8. Application restarts.
9. ApplicationStartedEvent fires.
10. BillingRecoveryListener.recoverPendingCharges() finds the PENDING row for order_1.
11. The listener calls billingService.chargeCustomer("cust_1", "order_1", 5000).
12. chargeCustomer writes a new PaymentAttempt row (second row!) with idempotencyKey = "uuid-B".
13. Stripe is called with Idempotency-Key: uuid-B. Stripe has no record of uuid-B in its idempotency cache — uuid-A was the key used for the original call, not uuid-B.
14. Stripe creates ch_B. Customer is now charged twice: ch_A (from before the crash) and ch_B (from the recovery).

The developer misconception

“A PENDING row proves the Stripe call was not committed. The PENDING state is written before Stripe is called — if Stripe had responded with HTTP 200, the code would have continued and updated the row to COMPLETED. A PENDING row means the charge did not commit on Stripe.”

This reasoning would be correct if the Stripe HTTP response and the database COMPLETED update were atomic — if receiving Stripe’s response and writing COMPLETED either both happened or neither happened. They are not atomic. There is a window between “Stripe HTTP response received by the JVM” and “paymentAttemptRepository.save(COMPLETED) committed to the database”. That window can be nanoseconds (negligible in the absence of crashes) or it can be open forever (if the JVM crashes after the HTTP response is buffered in memory but before the database write executes).

The PENDING row tells you exactly one thing: paymentAttemptRepository.save(COMPLETED) did not execute during the original invocation. It does not tell you whether the Stripe HTTP call completed. The Stripe HTTP call might have: (a) not been sent yet (crash before the HTTP call), (b) been sent but no response received (crash during the HTTP call while the network was in flight), or (c) completed successfully and ch_A committed on Stripe (crash after receiving the response but before writing COMPLETED). All three scenarios produce an identical database state: PENDING.

The common developer intuition is that scenario (c) is rare — “how often does a crash happen in that tiny window?” In production, JVM crashes are rare but they are not zero. Over months or years of operation, a service that processes thousands of charges per day will almost certainly hit this window at least once. When it does, the recovery loop creates a duplicate charge.

The fix

Store the idempotency key in the PENDING row before calling Stripe, and derive it as a content hash of the stable business inputs rather than a fresh UUID.randomUUID(). On recovery, reuse the key from the PENDING row:

@Service
public class BillingService {

    private final PaymentAttemptRepository paymentAttemptRepository;

    @Transactional(propagation = Propagation.REQUIRES_NEW)
    public PaymentAttempt createPendingAttempt(
            String customerId, String orderId, long amountCents) {

        // Content-hash key — identical for any invocation of the same logical charge
        String idempotencyKey = computeContentHash(customerId, orderId, amountCents);

        // Check for existing attempt (concurrent protection)
        return paymentAttemptRepository
            .findByIdempotencyKey(idempotencyKey)
            .orElseGet(() -> {
                PaymentAttempt attempt = new PaymentAttempt();
                attempt.setCustomerId(customerId);
                attempt.setOrderId(orderId);
                attempt.setAmountCents(amountCents);
                attempt.setStatus(PaymentStatus.PENDING);
                attempt.setIdempotencyKey(idempotencyKey);
                return paymentAttemptRepository.save(attempt);
            });
    }

    public ChargeResult chargeCustomer(
            String customerId, String orderId, long amountCents) {

        PaymentAttempt attempt = createPendingAttempt(customerId, orderId, amountCents);

        if (attempt.getStatus() == PaymentStatus.COMPLETED) {
            // Already charged — return the stored result
            return new ChargeResult(attempt.getChargeId(), "succeeded");
        }

        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(amountCents)
            .setCurrency("usd")
            .setCustomer(customerId)
            .build();

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(attempt.getIdempotencyKey()) // content-hash, same as original
            .build();

        Charge charge = Charge.create(params, options); // returns ch_A from cache on recovery

        attempt.setStatus(PaymentStatus.COMPLETED);
        attempt.setChargeId(charge.getId());
        paymentAttemptRepository.save(attempt);

        return new ChargeResult(charge.getId(), charge.getStatus());
    }

    private String computeContentHash(
            String customerId, String orderId, long amountCents) {
        String input = customerId + "|" + orderId + "|" + amountCents;
        try {
            MessageDigest digest = MessageDigest.getInstance("SHA-256");
            byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException("SHA-256 not available", e);
        }
    }
}

The recovery listener now simply calls the same chargeCustomer method, which reads the stored idempotency key from the PENDING row and passes it to Stripe. Stripe recognizes the key (from its 24-hour cache) and returns ch_A without creating a new charge:

@Component
public class BillingRecoveryListener {

    private final PaymentAttemptRepository paymentAttemptRepository;
    private final BillingService billingService;

    @EventListener(ApplicationStartedEvent.class)
    public void recoverPendingCharges(ApplicationStartedEvent event) {
        List<PaymentAttempt> pending =
            paymentAttemptRepository.findByStatus(PaymentStatus.PENDING);

        for (PaymentAttempt attempt : pending) {
            log.info("Recovering PENDING charge orderId={} key={}",
                attempt.getOrderId(), attempt.getIdempotencyKey());

            // chargeCustomer reads the PENDING row, finds the stored content-hash key,
            // and passes it to Stripe — Stripe returns ch_A from its 24h cache
            billingService.chargeCustomer(
                attempt.getCustomerId(),
                attempt.getOrderId(),
                attempt.getAmountCents()
            );
        }
    }
}

The critical invariant: the idempotency key stored in the PENDING row and the idempotency key sent to Stripe must be the same value. If the recovery creates a new random key, Stripe’s cache cannot match it to the original request — the cache key is the Idempotency-Key header value, not the orderId or any business identifier. The fix ensures that the key stored in the database row before the first Stripe call is the same key that will be used on every subsequent Stripe call for the same logical charge, regardless of which application instance is making the call or how much time has passed since the original attempt (up to Stripe’s 24-hour cache TTL).

A note on the 24-hour expiry: Stripe’s idempotency cache retains entries for 24 hours from the first request. If the ApplicationStartedEvent recovery fires more than 24 hours after the original Stripe call (e.g., the application was down for more than a day due to an extended incident), the cache entry for ch_A’s key will have expired, and Stripe will create a new charge even with the correct content-hash key. For scenarios where recovery may be delayed beyond 24 hours, first check whether a charge already exists for the customer and order via Charge.list() before calling Charge.create().

Mode 3: @SchedulerLock lockAtMostFor TTL expires before the batch completes — second node acquires the lock and re-processes the same batch window with UUID.randomUUID(), creating ch_B for customers already charged by the first node

The setup

The developer runs a monthly billing batch as a @Scheduled task on multiple application instances. To prevent concurrent execution, they use ShedLock’s @SchedulerLock annotation, which uses a shared lock table in the application database to ensure only one instance runs the task at a time. ShedLock is commonly used in Spring Boot applications alongside Spring Boot Actuator because both are standard production-readiness components.

@Component
public class MonthlyBillingBatch {

    private final CustomerRepository customerRepository;
    private final BillingService billingService;

    @Scheduled(cron = "0 0 1 * * ?") // 1 AM on the 1st of each month
    @SchedulerLock(
        name = "monthlyBilling",
        lockAtLeastFor = "PT5M",
        lockAtMostFor = "PT10M"   // BUG: must be longer than worst-case batch runtime
    )
    public void runMonthlyBilling() {
        LocalDate billingMonth = LocalDate.now().withDayOfMonth(1).minusMonths(1);

        List<Customer> customers =
            customerRepository.findBillableForMonth(billingMonth);

        log.info("Starting monthly billing batch for {} customers", customers.size());

        for (Customer customer : customers) {
            // BUG: UUID.randomUUID() — different value if this loop re-runs on another node
            String idempotencyKey = UUID.randomUUID().toString();

            try {
                billingService.chargeCustomer(
                    customer.getId(),
                    billingMonth.toString() + "-" + customer.getId(),
                    customer.getMonthlyAmountCents(),
                    idempotencyKey
                );
            } catch (Exception e) {
                log.error("Failed to charge customer={}", customer.getId(), e);
            }
        }

        log.info("Monthly billing batch completed");
    }
}

How ShedLock’s lockAtMostFor works and why it causes a double charge

ShedLock implements its lock using a row in a shedlock table with three columns: name (the lock name, e.g., "monthlyBilling"), lock_until (a timestamp indicating when the lock expires), and locked_at/locked_by (metadata about which instance holds it). When a @SchedulerLock task starts on Instance A:

1. ShedLock’s DefaultLockingTaskExecutor inserts (or updates) the row: lock_until = NOW() + lockAtMostFor, locked_by = "instance-a".
2. The task runs on Instance A.
3. When the task finishes, ShedLock updates the row: lock_until = NOW() + lockAtLeastFor (to prevent immediate re-execution on the same node).
4. Other nodes attempting to acquire the lock check whether lock_until < NOW() — if the lock is expired, they can acquire it.

The crucial property: lockAtMostFor is a hard wall-clock limit, set once at task start. ShedLock does not “extend the lock while the task is alive” or “check whether the lock holder is still running before expiring the lock”. After lockAtMostFor elapses, the lock_until timestamp is in the past, and the lock is available to any node — regardless of whether Instance A is still running the task.

For a billing batch that processes 10,000 customers and takes 15 minutes, a lockAtMostFor = "PT10M" configuration creates the following failure sequence:

1. 01:00:00 — Instance A acquires the lock. lock_until = 01:10:00.
2. 01:00:00–01:10:00 — Instance A charges customers 1–6,000. Each charge uses a unique UUID.randomUUID() as the Stripe key. Stripe commits ch_A for each customer. The billingService’s @Transactional method commits the COMPLETED rows to the database after each charge (or in small batches).
3. 01:10:00 — lock_until timestamp expires. Instance A is still running (processing customers 6,001–10,000) but the lock row in the database says the lock is expired.
4. 01:10:05 — Instance B’s @Scheduled task fires (slightly delayed). ShedLock checks the lock: lock_until = 01:10:00 < NOW() = 01:10:05 — the lock is expired. Instance B acquires the lock. lock_until = 01:20:05.
5. 01:10:05 — Instance B calls customerRepository.findBillableForMonth(billingMonth). This returns all customers where billing for the month has not been recorded. But how does the query determine “not been recorded”?

The answer depends on whether Instance A has committed its COMPLETED rows. In most implementations, the billing batch commits after each customer (or after each small batch of customers). By 01:10:00, customers 1–6,000 likely have COMPLETED rows in the database — Instance B’s query will skip them and process only customers 6,001–10,000. In this best-case scenario, there is no double charge (Instance B picks up where Instance A left off). But this best-case assumes:

• The query accurately identifies “already charged this month” using the committed data from Instance A.
• Instance A does not also try to charge customers 6,001–10,000 simultaneously (both nodes are processing the same tail of the customer list simultaneously).
• The in-flight charges on Instance A (customer 6,001 being charged right now) are not also being initiated by Instance B (because both nodes see customer 6,001 as uncharged when they both query at 01:10:05).

The double charge occurs at the boundary: Instance A starts charging customer 6,001 at 01:10:03 (two seconds before the lock expires). Instance A has inserted a PENDING row but not yet committed the COMPLETED row. At 01:10:05, Instance B queries for unbilled customers and finds customer 6,001 (the PENDING row is not a COMPLETED row). Instance B also starts charging customer 6,001 with a new UUID.randomUUID(). Both instances call Stripe with different idempotency keys — ch_A (Instance A) and ch_B (Instance B) are both committed on Stripe.

In the worst case — where findBillableForMonth is defined as “customers where there is no COMPLETED row” and Instance A commits in one large transaction at the end of the batch — Instance B sees all 10,000 customers as unbilled at 01:10:05 and re-charges all 6,000 that Instance A has already charged with Stripe.

The developer misconception

“lockAtMostFor is a safety net for crashed nodes — its purpose is to prevent a dead node’s lock from blocking all other nodes forever. A live node running the task holds the lock for as long as the task takes and releases it when the task finishes. lockAtMostFor will only fire if the node dies mid-task without releasing the lock. Setting it to 10 minutes means a crashed node can block others for at most 10 minutes — it does not mean a healthy node’s task is interrupted after 10 minutes.”

This is the most common misunderstanding of ShedLock’s lockAtMostFor parameter. Unlike a lease renewal mechanism (where the lock holder periodically extends the lock_until timestamp while it is alive), ShedLock sets lock_until once at task start and never updates it while the task is running. The lock expires at a fixed wall-clock time, not “when the task finishes plus a safety margin”. ShedLock’s documentation does note this, but developers who arrive at the parameter from the mental model of “max lock duration for a crashed node” miss the implication: a live node’s lock expires too, if the task takes longer than lockAtMostFor.

ShedLock does have a lease-renewal mechanism available through LockExtender (available since ShedLock 4.x), but it requires explicit code in the task to extend the lock. The @SchedulerLock annotation alone does not enable lease renewal.

The fix

Three complementary fixes, applied together:

Fix 1: Set lockAtMostFor to a value comfortably above the worst-case batch runtime. If the batch typically takes 15 minutes and sometimes takes 20 minutes under load, set lockAtMostFor = "PT45M". The purpose of lockAtMostFor is to bound how long a crashed node can hold the lock, not to limit how long the task runs. Set it generously:

@Scheduled(cron = "0 0 1 * * ?")
@SchedulerLock(
    name = "monthlyBilling",
    lockAtLeastFor = "PT20M",  // minimum gap between runs, prevents immediate re-run
    lockAtMostFor = "PT45M"    // crash-protection TTL — must exceed worst-case runtime
)
public void runMonthlyBilling() { ... }

Fix 2: Use a content-hash idempotency key instead of UUID.randomUUID(). Even if the lock expires and a second node starts processing, a content-hash key derived from (customerId, billingMonth) will be identical on both nodes for the same customer. Stripe’s cache returns ch_A when the second node calls Stripe for the same customer:

for (Customer customer : customers) {
    // Content-hash — identical on any node for the same customer + month
    String idempotencyKey = computeContentHash(
        customer.getId(), billingMonth.toString());

    billingService.chargeCustomer(
        customer.getId(),
        billingMonth.toString() + "-" + customer.getId(),
        customer.getMonthlyAmountCents(),
        idempotencyKey
    );
}

Fix 3: Use LockExtender for long-running batches to extend the lock while the task is alive. ShedLock’s LockExtender allows the task to explicitly extend lock_until during execution, preventing lock expiry for a healthy node:

@Autowired LockExtender lockExtender;

@Scheduled(cron = "0 0 1 * * ?")
@SchedulerLock(
    name = "monthlyBilling",
    lockAtLeastFor = "PT5M",
    lockAtMostFor = "PT15M"   // initial TTL; will be extended as the batch runs
)
public void runMonthlyBilling() {
    LocalDate billingMonth = LocalDate.now().withDayOfMonth(1).minusMonths(1);
    List<Customer> customers =
        customerRepository.findBillableForMonth(billingMonth);

    int processed = 0;
    for (Customer customer : customers) {
        String idempotencyKey = computeContentHash(
            customer.getId(), billingMonth.toString());

        billingService.chargeCustomer(
            customer.getId(),
            billingMonth.toString() + "-" + customer.getId(),
            customer.getMonthlyAmountCents(),
            idempotencyKey
        );

        processed++;

        // Extend the lock every 500 customers to prevent expiry mid-batch
        if (processed % 500 == 0) {
            lockExtender.extendActiveLock(Duration.ofMinutes(10));
            log.info("Extended lock after {} customers", processed);
        }
    }
}

Fix 1 alone prevents the lock from expiring before the batch finishes (assuming the worst-case runtime is correctly estimated). Fix 2 alone limits the blast radius if the lock does expire — concurrent processing on two nodes produces idempotent Stripe calls. Fix 3 makes the lock expiry adaptive to actual runtime. For production monthly billing, all three fixes should be applied together.

Integration test patterns for all three modes

All three failure modes can be reproduced in integration tests using @SpringBootTest, WireMock for Stripe, and Testcontainers for the database. The key insight is that each mode requires a different test structure:

@SpringBootTest
@Testcontainers
class ActuatorStripeIdempotencyTest {

    @Container
    static PostgreSQLContainer<?> postgres =
        new PostgreSQLContainer<>("postgres:16");

    static WireMockServer wireMock =
        new WireMockServer(WireMockConfiguration.options().dynamicPort());

    @Autowired BillingActuatorEndpoint billingEndpoint;
    @Autowired BillingRecoveryListener recoveryListener;
    @Autowired PaymentAttemptRepository paymentAttemptRepository;

    @BeforeAll
    static void startWireMock() {
        wireMock.start();
        wireMock.stubFor(WireMock.post(WireMock.urlEqualTo("/v1/charges"))
            .willReturn(WireMock.okJson(
                "{\"id\":\"ch_A\",\"status\":\"succeeded\"}")));
    }

    @Test
    void writeOperation_httpRetry_contentHashKey_stripeCalledOncePerLogicalCharge() {
        // Simulate two HTTP calls to the @WriteOperation endpoint
        // with identical parameters (HTTP retry scenario)
        billingEndpoint.chargeCustomer("cust_1", "order_1", 5000L);
        billingEndpoint.chargeCustomer("cust_1", "order_1", 5000L);

        // Assert: Stripe was called twice (once per endpoint invocation)
        // but with the SAME Idempotency-Key header both times (content-hash)
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            WireMock.postRequestedFor(WireMock.urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSize(2);

        String key1 = stripeRequests.get(0).getHeader("Idempotency-Key");
        String key2 = stripeRequests.get(1).getHeader("Idempotency-Key");
        assertThat(key1)
            .describedAs("HTTP retry must use same content-hash key")
            .isEqualTo(key2);
    }

    @Test
    void applicationStartedEvent_pendingRecovery_reuseStoredKey_noDoubleCharge() {
        // Simulate crash: write PENDING row with content-hash key,
        // do NOT update to COMPLETED (simulates crash after Stripe succeeded)
        String idempotencyKey = computeContentHash("cust_2", "order_2", 7500L);

        PaymentAttempt crashedAttempt = new PaymentAttempt();
        crashedAttempt.setCustomerId("cust_2");
        crashedAttempt.setOrderId("order_2");
        crashedAttempt.setAmountCents(7500L);
        crashedAttempt.setStatus(PaymentStatus.PENDING);
        crashedAttempt.setIdempotencyKey(idempotencyKey); // stored before Stripe call
        paymentAttemptRepository.save(crashedAttempt);

        // Run recovery
        recoveryListener.recoverPendingCharges(null);

        // Assert: Stripe received the request with the SAME key as the stored PENDING key
        List<LoggedRequest> stripeRequests = wireMock.findAll(
            WireMock.postRequestedFor(WireMock.urlEqualTo("/v1/charges"))
                .withHeader("Idempotency-Key", WireMock.equalTo(idempotencyKey)));
        assertThat(stripeRequests).hasSize(1)
            .describedAs("Recovery must reuse the stored content-hash key — not generate UUID_B");

        // Assert: no second attempt with a different key
        List<LoggedRequest> allStripeRequests = wireMock.findAll(
            WireMock.postRequestedFor(WireMock.urlEqualTo("/v1/charges")));
        assertThat(allStripeRequests).hasSize(1);
    }

    @Test
    void schedulerLock_concurrent_contentHashKey_bothNodesProduceSameIdempotencyKey() {
        // Simulate two nodes running the billing batch concurrently
        // for the same customer (lock TTL expiry scenario)
        String customerId = "cust_3";
        String billingMonth = "2026-09";
        long amountCents = 9900L;

        String keyFromNode1 = computeContentHash(customerId, billingMonth);
        String keyFromNode2 = computeContentHash(customerId, billingMonth);

        // Assert: both nodes compute the same key
        assertThat(keyFromNode1)
            .describedAs("Content-hash keys must be identical on both nodes")
            .isEqualTo(keyFromNode2);

        // Simulate both nodes calling Stripe with the same key
        wireMock.stubFor(WireMock.post(WireMock.urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", WireMock.equalTo(keyFromNode1))
            .willReturn(WireMock.okJson(
                "{\"id\":\"ch_A\",\"status\":\"succeeded\"}")));

        // First node charges
        ChargeResult result1 = billingService.chargeCustomer(
            customerId, billingMonth + "-" + customerId, amountCents, keyFromNode1);

        // Second node charges (simulating lock expiry concurrent execution)
        ChargeResult result2 = billingService.chargeCustomer(
            customerId, billingMonth + "-" + customerId, amountCents, keyFromNode2);

        // Both return ch_A — Stripe's idempotency cache deduplicated
        assertThat(result1.getChargeId()).isEqualTo("ch_A");
        assertThat(result2.getChargeId()).isEqualTo("ch_A");
    }
}

The test for Mode 1 directly invokes the @WriteOperation method twice with identical parameters, verifying that the content-hash key is stable across invocations. In a real HTTP retry scenario, the second invocation arrives via a separate HTTP request — simulating this in a unit/integration test by calling the method twice is equivalent from a Stripe idempotency perspective, because Stripe’s cache operates on the Idempotency-Key header value, not on HTTP session state.

The test for Mode 2 directly inserts a PENDING row (simulating a crash-surviving state) and then invokes the recovery listener. The assertion verifies that the Stripe call uses the key stored in the PENDING row — not a fresh random key.

The test for Mode 3 verifies that two nodes computing the content-hash key for the same customer and billing month produce identical keys, and that Stripe returns ch_A for both calls. This test does not require running two actual Spring Boot instances — it suffices to show that the key derivation function is deterministic across JVM instances.

How the three failure modes differ from the rest of this series

The prior posts in this series have focused on failure modes caused by message redelivery (Kafka consumer seek-and-re-poll, Streams REPLACE_THREAD, TaskMigratedException), reactive re-subscription (Project Reactor .retryWhen() re-subscribing through @Transactional proxy, Flux.merge() restart, Mono.cache() per-call scope), and AOP proxy re-entry (@Retryable calling MethodInvocation.proceed() through nested interceptors). The three Spring Boot Actuator modes in this post differ in an important way: they are caused by infrastructure-level re-execution at a higher abstraction layer than the application’s own retry logic.

• Mode 1 is caused by the HTTP transport layer, not by any application retry annotation. The application code has no @Retryable — the re-execution is initiated by an external client before the application even knows the first request succeeded.
• Mode 2 is caused by the Spring Boot lifecycle event system. The re-execution is a deliberate design decision (recover on startup), but the execution model of the lifecycle event does not provide any protection against calling Stripe twice for the same charge.
• Mode 3 is caused by the semantics of a distributed lock whose TTL behavior differs from developer expectations. The developer intended to prevent concurrent execution — the failure occurs because the prevention mechanism (ShedLock) has a hard TTL that the developer did not account for.

All three share the root cause that is common across the entire series: the Stripe Idempotency-Key header is computed from a value that changes on re-execution (UUID.randomUUID()), rather than from stable business inputs. The fix is always the same: compute the key as a deterministic hash of stable inputs so that any execution of the same logical charge sends the same key to Stripe.

Comparison table

The invariant that prevents all three failures

Every failure in this series — across Kafka consumers, reactive streams, AOP proxies, and now Spring Boot Actuator infrastructure — reduces to the same broken assumption: that the Stripe Idempotency-Key header value is stable across executions of the same logical charge. UUID.randomUUID() is not stable — it generates a unique value on every JVM call, which means every execution of the charge code path sends a different key to Stripe, and Stripe’s idempotency cache cannot protect against the duplicate.

The invariant that prevents all three failures:

For a given logical charge (a specific customer, order, and amount), every execution of the Stripe API call — whether it is the first attempt, a retry, a crash recovery, a concurrent execution on a second node, or an HTTP client retry from a caller — must send the same Idempotency-Key header to Stripe.

A content-hash key derived from the stable business inputs (customerId, orderId, amountCents) satisfies this invariant for all three Spring Boot Actuator failure modes. The hash function is deterministic: the same inputs always produce the same output, regardless of which JVM instance computes it, when it computes it, or how many times the computation is re-executed. Stripe’s 24-hour idempotency cache uses the key value as the deduplication token — it does not require any coordination between application instances to function correctly. The content-hash key is the application’s half of the contract with Stripe’s idempotency system; UUID.randomUUID() is a unilateral violation of that contract on every execution after the first.