Axon Framework and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Axon Framework’s CQRS and event-sourcing architecture introduces Stripe billing failure modes that are structurally different from monolithic retry loops or reactive-stream re-subscription patterns. Three places where duplicate charges appear: RetryScheduler re-dispatching a command through CommandBus to a @CommandHandler that generates UUID.randomUUID() on each method invocation; a Saga @SagaEventHandler where the Tracking Event Processor replays a billing event after a pod restart because the handler threw before the Saga’s billingInitiated flag was persisted; and Spring @Scheduled billing triggers firing on every Kubernetes replica simultaneously, each dispatching a ChargeBillingCommand with a distinct UUID to the same @CommandHandler.

This post covers all three failure modes with Java code (Axon Framework 4.x, Spring Boot 3.x), content-hash stable keys carried in the command object rather than generated inside the handler, the Saga billingInitiated guard pattern and how Axon’s SagaStore persistence interacts with Tracking Event Processor token commits, ShedLock for distributed @Scheduled coordination across JVM replicas, and pre-flight PostgreSQL ON CONFLICT DO NOTHING as a durable billing mutex — plus per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For Spring Retry’s @Retryable AOP proxy re-invocation pattern, see the Spring Retry and Stripe Integration post. For Apache Camel’s onException().maximumRedeliveries() redelivery pattern, see the Apache Camel and Stripe Integration post. The Axon failure modes share a common root — UUID.randomUUID() called at handler invocation time rather than at command-construction time or Saga-initialization time — but the trigger mechanisms differ at each layer of the framework.

Failure mode 1: RetryScheduler re-dispatches the command through CommandBus — @CommandHandler generates UUID.randomUUID() on each method invocation — initial dispatch creates ch_A before StripeException — retry dispatch creates ch_B

Axon Framework’s CommandGateway can be configured with a RetryScheduler that intercepts command dispatch failures and re-sends the original command object to the CommandBus. The IntervalRetryScheduler (the built-in implementation) re-dispatches the command after a configurable interval, up to a maximum number of attempts. The same CommandMessage object is re-sent; the command handler receives the same field values on every attempt.

The failure arises when the Stripe idempotency key is generated inside the @CommandHandler method body rather than inside the command object itself. In that pattern, UUID.randomUUID() is a call expression evaluated each time the handler method is invoked — once on the initial dispatch and once per retry dispatch. The first invocation generates UUID_A and commits ch_A to Stripe before a socket timeout raises StripeException. The RetryScheduler re-sends the command; the @CommandHandler is invoked again, generates UUID_B, and Stripe creates ch_B as a duplicate charge:

// UNSAFE: UUID.randomUUID() generated inside @CommandHandler.
// RetryScheduler re-dispatches the same ChargeBillingCommand object on failure.
// @CommandHandler is a Java method — invoked fresh on each command dispatch.
// UUID.randomUUID() is a call expression evaluated per method invocation.

import com.stripe.model.Charge;
import com.stripe.net.RequestOptions;
import com.stripe.param.ChargeCreateParams;
import org.axonframework.commandhandling.CommandHandler;
import org.springframework.stereotype.Component;

import java.util.UUID;

@Component
public class BillingAggregate {

    // @CommandHandler is a method — Axon invokes it once per command dispatch.
    // RetryScheduler re-dispatches: this method is called again with the same
    // command object but UUID.randomUUID() evaluates fresh per invocation.
    //
    // Dispatch 1 (initial attempt):
    //   UUID_A = "7a3f1b2c-4d5e-6f7a-8b9c-0d1e2f3a4b5c"
    //   POST /v1/charges → ch_A committed before socket timeout fires
    //   StripeException(SocketTimeoutException) thrown → CommandBus receives failure
    //
    // Dispatch 2 (RetryScheduler retry attempt 1):
    //   UUID_B = "c9d8e7f6-5a4b-3c2d-1e0f-9a8b7c6d5e4f"  ← different UUID
    //   POST /v1/charges → Stripe sees new idempotency key → creates ch_B ← duplicate
    //
    // Dispatch 3 (RetryScheduler retry attempt 2):
    //   UUID_C = "b2a1c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d"  ← yet another UUID
    //   POST /v1/charges → Stripe creates ch_C ← triplicate
    @CommandHandler
    public void handle(ChargeBillingCommand command) throws Exception {
        // UNSAFE: UUID generated at handler invocation time, not at command construction time.
        String idempotencyKey = UUID.randomUUID().toString(); // evaluates fresh per invocation

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(command.getAmountCents())
            .setCurrency("usd")
            .setCustomer(command.getCustomerId())
            .build();

        Charge charge = Charge.create(params, options);
        // RetryScheduler never sees ch_A — it re-dispatches because StripeException was thrown
        // after ch_A was committed on Stripe's side but before the return here.
    }
}

// ChargeBillingCommand carries customerId and amountCents but NOT an idempotencyKey field.
// The missing field is what forces UUID generation into the handler.
public class ChargeBillingCommand {
    private final String customerId;
    private final long amountCents;
    private final String billingPeriod;
    // No idempotencyKey field — generator is in the handler — BUG
    // ...
}

// RetryScheduler configuration:
import org.axonframework.commandhandling.gateway.CommandGateway;
import org.axonframework.commandhandling.gateway.DefaultCommandGateway;
import org.axonframework.commandhandling.gateway.IntervalRetryScheduler;

IntervalRetryScheduler retryScheduler = IntervalRetryScheduler.builder()
    .retryExecutor(Executors.newSingleThreadScheduledExecutor())
    .maxRetryCount(3)
    .retryInterval(1000)
    .build();

// CommandGateway with retry — re-dispatches on RuntimeException including StripeException.
CommandGateway gateway = DefaultCommandGateway.builder()
    .commandBus(commandBus)
    .retryScheduler(retryScheduler)
    .build();

// Timeline for customer "cust_123", billingPeriod="2026-11":
// 10:00:00.000  gateway.send(command) — Dispatch 1
// 10:00:00.001  @CommandHandler invoked — UUID_A generated
// 10:00:00.002  POST /v1/charges (idempotencyKey=UUID_A)
// 10:00:29.999  Stripe: ch_A committed (charges.created event fired internally)
// 10:00:30.001  Socket read timeout — StripeException thrown from Charge.create()
// 10:00:30.001  CommandBus receives failure — IntervalRetryScheduler triggered
// 10:00:31.001  RetryScheduler re-dispatches same ChargeBillingCommand
// 10:00:31.002  @CommandHandler invoked again — UUID_B generated (different)
// 10:00:31.100  POST /v1/charges (idempotencyKey=UUID_B) — Stripe creates ch_B
// Result: customer "cust_123" billed twice for November 2026.

The fix is to make the Stripe idempotency key a field of the command object, computed by the caller before the first gateway.send() call. RetryScheduler re-dispatches the same ChargeBillingCommand object; the @CommandHandler reads the idempotencyKey field — the same value on every dispatch — and passes it to Stripe. Stripe’s idempotency cache returns ch_A on all retries with the same key:

// SAFE: stable content-hash key computed at command construction time.
// The key is a field of ChargeBillingCommand — set once before gateway.send().
// @CommandHandler reads the field value — same on every dispatch including retries.
// RetryScheduler re-dispatches the same command object — field value unchanged.

import org.apache.commons.codec.digest.DigestUtils;

public class ChargeBillingCommand {
    private final String customerId;
    private final long amountCents;
    private final String billingPeriod;
    private final String idempotencyKey; // carried in the command object

    public ChargeBillingCommand(String customerId, long amountCents, String billingPeriod) {
        this.customerId = customerId;
        this.amountCents = amountCents;
        this.billingPeriod = billingPeriod;
        // SAFE: computed once at construction time — never re-evaluated.
        // sha256(customerId:billingPeriod:axon-billing)[:32] is deterministic:
        // same inputs → same output, regardless of which attempt number this is,
        // which thread constructs the command, or what time it is constructed.
        this.idempotencyKey = DigestUtils.sha256Hex(
            customerId + ":" + billingPeriod + ":axon-billing"
        ).substring(0, 32);
    }

    public String getIdempotencyKey() { return idempotencyKey; }
    // ... other getters
}

// @CommandHandler reads the field — never calls UUID.randomUUID().
@CommandHandler
public void handle(ChargeBillingCommand command) throws Exception {
    // SAFE: command.getIdempotencyKey() returns the pre-computed stable key.
    // RetryScheduler dispatches the same ChargeBillingCommand object — same key.
    // Dispatch 1: key="a3f1b2c4d5e6f7a8b9c0d1e2f3a4b5c6" → POST /v1/charges → ch_A
    // Dispatch 2: key="a3f1b2c4d5e6f7a8b9c0d1e2f3a4b5c6" → Stripe idempotency cache → ch_A
    // Dispatch 3: key="a3f1b2c4d5e6f7a8b9c0d1e2f3a4b5c6" → Stripe idempotency cache → ch_A
    RequestOptions options = RequestOptions.builder()
        .setIdempotencyKey(command.getIdempotencyKey())
        .build();
    ChargeCreateParams params = ChargeCreateParams.builder()
        .setAmount(command.getAmountCents())
        .setCurrency("usd")
        .setCustomer(command.getCustomerId())
        .build();
    Charge charge = Charge.create(params, options);
    // If Stripe timed out after committing ch_A, next retry returns ch_A from cache.
    // No duplicate charge regardless of retry count.
}

// Add a pre-flight guard in the aggregate state for post-24h scenarios
// where Stripe's idempotency cache window has expired:
@Aggregate
public class CustomerBillingAggregate {

    @AggregateIdentifier
    private String customerId;

    // Persisted in EventStore — survives pod restarts.
    private final Set<String> billedPeriods = new HashSet<>();

    @CommandHandler
    public void handle(ChargeBillingCommand command) throws Exception {
        // Guard: reject duplicate billing attempts for periods already completed.
        if (billedPeriods.contains(command.getBillingPeriod())) {
            return; // already billed — idempotent no-op
        }
        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(command.getIdempotencyKey())
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(command.getAmountCents())
            .setCurrency("usd")
            .setCustomer(command.getCustomerId())
            .build();
        Charge charge = Charge.create(params, options);
        // Publish domain event — @EventSourcingHandler updates billedPeriods set.
        apply(new CustomerBilledEvent(command.getCustomerId(),
                                      command.getBillingPeriod(),
                                      charge.getId()));
    }

    @EventSourcingHandler
    public void on(CustomerBilledEvent event) {
        billedPeriods.add(event.getBillingPeriod()); // persisted to EventStore
    }
}

Why placing the key inside the command handler is the structural root cause, not just a code smell

In Axon’s CQRS model, a command is a value object: an immutable description of an intent, carrying all information the handler needs to execute the operation. A command that says “charge customer X an amount Y for period Z” but leaves the idempotency key to be generated inside the handler is a leaky abstraction — the handler is no longer a pure function of its inputs. The same ChargeBillingCommand produces different Stripe calls depending on when and how many times the handler method is invoked. Carrying the idempotency key as a command field restores the pure-function property: given the same command object, the handler always makes the same Stripe call. Retry, event replay, and projection rebuild all become safe because the command’s identity is fixed at construction time.

The content-hash construction rule is: sha256(customerId + ":" + billingPeriod + ":axon-billing")[:32]. This must not include UUID.randomUUID(), System.currentTimeMillis(), System.nanoTime(), the command object’s memory address or identity hash, a retry counter, a pod hostname, or any other value that changes between invocations. The key should be a deterministic function of the stable billing fields — the values that uniquely identify “this charge for this customer for this period”.

Failure mode 2: Saga @SagaEventHandler on a billing event — Tracking Event Processor replays the event after pod restart because token not committed — billingInitiated flag not persisted before the throw — UUID.randomUUID() fresh on replay — ch_B while ch_A already committed

Axon Framework’s Saga pattern is commonly used for multi-step business processes where billing is triggered by a domain event. A MonthlyBillingDueEvent is published by a scheduler or aggregate; a Saga listens with @SagaEventHandler and handles the billing step. The Tracking Event Processor (TEP) that drives this Saga commits an event processing token — a position marker in the event store — after the handler completes successfully. If the handler throws, the token is not committed, and the event is replayed on the next processor start.

The failure pattern is: the @SagaEventHandler calls Stripe (creating ch_A), then attempts to write a local billing record to PostgreSQL. The PostgreSQL write times out. The handler throws. The TEP does not commit the token for the MonthlyBillingDueEvent position. The developer had planned to set billingInitiated = true on the Saga instance after the PostgreSQL write succeeded — since the write threw, the Saga’s field remains false. The Saga is saved to SagaStore (Axon commits Saga state when the unit of work succeeds, which it does not when the handler throws) — so the in-memory Saga field changes are discarded. On the next pod start, the TEP replays from the uncommitted token position, loads the Saga from SagaStore (with billingInitiated = false), invokes the handler again, and the handler calls UUID.randomUUID(), generating UUID_B — Stripe creates ch_B while ch_A is already in the ledger:

// UNSAFE: billingInitiated set after Stripe call — after the PostgreSQL write.
// If the PostgreSQL write throws, the handler throws, TEP does not commit the token,
// Saga state changes are discarded (Axon rolls back the unit of work),
// and the event is replayed on next TEP start with billingInitiated still false.

import org.axonframework.modelling.saga.SagaEventHandler;
import org.axonframework.modelling.saga.StartSaga;

import java.util.UUID;

public class CustomerBillingSaga {

    // Persisted to SagaStore — survives pod restarts via Axon's JPA SagaStore.
    private boolean billingInitiated = false;
    private String customerId;

    @StartSaga
    @SagaEventHandler(associationProperty = "customerId")
    public void on(SubscriptionActivatedEvent event) {
        this.customerId = event.getCustomerId();
    }

    @SagaEventHandler(associationProperty = "customerId")
    public void on(MonthlyBillingDueEvent event) throws Exception {
        if (billingInitiated) {
            return; // guard — but this flag is not set on the problematic path
        }

        // UNSAFE: UUID generated here — evaluates fresh on each handler invocation.
        // If this handler is replayed (because TEP did not commit the token),
        // UUID.randomUUID() produces a different value — UUID_B instead of UUID_A.
        String idempotencyKey = UUID.randomUUID().toString();

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(customerId)
            .build();

        Charge charge = Charge.create(params, options);
        // ch_A committed to Stripe at this point.

        // UNSAFE: billingInitiated not set before Stripe call —
        // the Saga field is updated here, after Stripe returns.
        // If the next line throws (PostgreSQL timeout, constraint violation),
        // the handler fails, Axon rolls back the unit of work (Saga field
        // changes are not persisted), TEP does not commit the token.
        saveBillingRecord(customerId, event.getBillingPeriod(), charge.getId()); // may throw

        // This line never reached if saveBillingRecord() throws.
        billingInitiated = true; // too late — Saga state not saved if above line threw
    }
}

// Event replay timeline for customer "cust_123", billingPeriod="2026-11":
// 10:00:00.000  TEP processes MonthlyBillingDueEvent (position 4412)
// 10:00:00.001  Saga loaded from SagaStore — billingInitiated=false
// 10:00:00.002  UUID_A generated; POST /v1/charges — ch_A committed
// 10:00:00.800  Stripe responds 200: charge.id=ch_A
// 10:00:01.800  saveBillingRecord() — PostgreSQL write timeout after 1 second
// 10:00:01.801  Exception thrown from handler
// 10:00:01.801  Axon unit-of-work rolls back — Saga changes discarded — token NOT committed
// 10:00:01.802  Pod restarts (rolling restart triggered by OOM on sibling container)
// 10:00:02.500  TEP starts — last committed token = position 4411
// 10:00:02.501  TEP replays MonthlyBillingDueEvent (position 4412)
// 10:00:02.502  Saga loaded from SagaStore — billingInitiated=false (was never committed)
// 10:00:02.503  UUID_B generated (different from UUID_A)
// 10:00:02.504  POST /v1/charges (idempotencyKey=UUID_B) — Stripe creates ch_B
// Result: customer "cust_123" billed twice for November 2026.

The fix requires two changes applied together. First, the idempotency key must be a deterministic content-hash stored as a Saga field, so that even if the handler is replayed, it generates the same key and Stripe’s idempotency cache returns ch_A. Second, the billingInitiated flag (or, better, the persisted Saga field billingKey) must be set before the Stripe call, so that if the Stripe call succeeds and a subsequent step throws, a replay finds the flag already set and returns without calling Stripe again:

// SAFE: content-hash key stored as a Saga field — set before the Stripe call.
// billingKey is non-null after the first successful entry into the billing path.
// On replay, billingKey is already set — Stripe call uses the same key — ch_A cached.
// billingGuardSet flag set before Stripe call — replay returns immediately if already set.

import org.apache.commons.codec.digest.DigestUtils;
import org.axonframework.modelling.saga.SagaEventHandler;

public class CustomerBillingSaga {

    private String customerId;

    // SAFE: both fields are set before the Stripe call and persisted to SagaStore
    // when the Axon unit of work commits after the handler returns successfully.
    // On replay after a failed commit, the Saga is loaded from SagaStore — if the
    // handler previously threw before the Stripe call, both fields are null/false and
    // the handler re-runs normally. If the handler previously completed the Stripe call
    // but threw on a later step, the unit of work did not commit, so the fields are
    // still null/false in SagaStore — but now the Stripe key is deterministic, so
    // the Stripe idempotency cache returns ch_A regardless.
    private String billingKey = null;
    private boolean billingGuardSet = false;

    @SagaEventHandler(associationProperty = "customerId")
    public void on(MonthlyBillingDueEvent event) throws Exception {
        // Guard: if billingGuardSet is true, this Saga instance has already
        // processed this event on a prior attempt that committed successfully.
        // Return immediately — no Stripe call, no duplicate charge.
        if (billingGuardSet) {
            return;
        }

        // SAFE: content-hash key computed from stable billing fields.
        // Same result regardless of which pod computes it or how many times
        // this handler has been replayed.
        if (billingKey == null) {
            billingKey = DigestUtils.sha256Hex(
                customerId + ":" + event.getBillingPeriod() + ":axon-saga-billing"
            ).substring(0, 32);
        }

        // SAFE: billingGuardSet is set BEFORE the Stripe call.
        // If this line is reached and the Saga state is later committed to SagaStore,
        // a replay will find billingGuardSet=true and return immediately (above guard).
        // Axon commits Saga state when the unit of work succeeds. If the handler throws
        // after this assignment but before the unit of work commits, the assignment is
        // discarded — billingGuardSet remains false in SagaStore. But billingKey is
        // also discarded in that case — so the next replay recomputes the same billingKey
        // (deterministic) and calls Stripe with the same key. Stripe's idempotency cache
        // returns ch_A if ch_A was already committed on a prior attempt.
        billingGuardSet = true;

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(billingKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(customerId)
            .build();

        Charge charge = Charge.create(params, options);
        // ch_A committed. If saveBillingRecord() throws, the unit of work rolls back.
        // Next replay: billingGuardSet=false, billingKey=null in SagaStore.
        // Handler recomputes billingKey (same deterministic value) → Stripe cache → ch_A.

        saveBillingRecord(customerId, event.getBillingPeriod(), charge.getId());
        // If this succeeds, unit of work commits: billingGuardSet=true, billingKey set.
        // Next replay (if any): billingGuardSet=true → return immediately → no Stripe call.
    }
}

Understanding Axon’s unit-of-work boundary and when Saga state is committed

The critical detail is when Axon commits Saga state to SagaStore. Axon Framework processes each event within a UnitOfWork. The unit of work starts before the handler is invoked and is committed (or rolled back) after the handler returns. Saga state changes — field assignments, association additions — are tracked within the unit of work and flushed to SagaStore when the unit of work commits. If the handler throws an exception (checked or unchecked), the unit of work is rolled back, and Saga state changes are discarded.

This means there is no safe window to set billingInitiated = true “before the Stripe call but after the unit of work might fail.” The unit of work either commits all changes together at the end, or discards all changes on throw. The correct design is to ensure that the Stripe call itself is idempotent across replays — which is achieved by the content-hash key that produces the same value regardless of replay count. The billingGuardSet flag provides an optimization for the committed-but-replayed case (a Saga with billingGuardSet=true in SagaStore will skip the Stripe call entirely on replay), but the deterministic key is the primary correctness guarantee.

The pre-flight PostgreSQL guard provides an additional safety net for scenarios where Stripe’s 24-hour idempotency cache has expired between the initial charge and a very late replay:

-- Pre-flight guard: insert a billing record before calling Stripe.
-- ON CONFLICT DO NOTHING prevents duplicate Stripe calls for already-billed periods
-- even if Stripe's 24-hour idempotency cache has expired.
-- The Saga calls this before Charge.create() and checks the result.

INSERT INTO billing_records (customer_id, billing_period, idempotency_key, status)
VALUES (?, ?, ?, 'initiated')
ON CONFLICT (customer_id, billing_period) DO NOTHING;

-- If INSERT affected 0 rows: another handler already initiated billing for this period.
-- Return without calling Stripe.
-- If INSERT affected 1 row: this handler is the first — proceed with Stripe call.
-- After Stripe returns, UPDATE billing_records SET charge_id=?, status='completed'
-- WHERE customer_id=? AND billing_period=?.

Failure mode 3: Spring @Scheduled billing trigger fires on all Kubernetes replicas simultaneously — each pod dispatches ChargeBillingCommand with a distinct UUID per customer — Axon CommandBus routes by aggregate identifier, not by command content — ch_A, ch_B, ch_C per customer per billing period

In a Spring Boot + Axon application deployed to Kubernetes with replicas:3, billing is often triggered by a scheduled task. A common implementation uses Spring’s @Scheduled(cron="0 0 1 * * *") on a service bean to iterate over active customers and dispatch a ChargeBillingCommand for each. Spring’s @Scheduled is JVM-local — there is no built-in mechanism to elect one pod as the billing scheduler and suppress execution on the others. All three pods execute the annotated method at the cron interval simultaneously.

If the ChargeBillingCommand still carries the UUID anti-pattern from failure mode 1 (key generated at command construction time inside the billing service, not as a deterministic content-hash), each pod constructs a ChargeBillingCommand with a different UUID for each customer. The CommandBus routes each command to the correct aggregate by @TargetAggregateIdentifier — Axon ensures that commands to the same aggregate are serialized by the aggregate lock. But three commands with three different UUIDs are three distinct commands; each goes through the aggregate’s @CommandHandler in sequence and calls Stripe with a different idempotency key. The aggregate’s billedPeriods guard (from failure mode 1’s safe version) does protect against this — but only if it was already applied. Without the guard, all three commands flow to Stripe and produce ch_A, ch_B, and ch_C:

// UNSAFE: @Scheduled billing trigger with UUID.randomUUID() at command construction time.
// Spring @Scheduled fires on ALL pods in a Kubernetes Deployment with replicas:3.
// Each pod constructs ChargeBillingCommand with UUID.randomUUID() for each customer.
// Three pods → three different UUID values per customer → three commands → three Stripe calls.

import org.axonframework.commandhandling.gateway.CommandGateway;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.UUID;

@Service
public class BillingSchedulerService {

    private final CommandGateway commandGateway;
    private final CustomerRepository customerRepository;

    // @Scheduled fires on every pod — no cluster-wide coordination.
    // replicas:3 → three pods execute this method at cron time.
    @Scheduled(cron = "0 0 1 * * *") // 01:00 UTC first day of month
    public void runMonthlyBilling() {
        String billingPeriod = currentBillingPeriod(); // e.g., "2026-11"

        List<Customer> activeCustomers = customerRepository.findAllActive();

        for (Customer customer : activeCustomers) {
            // UNSAFE: UUID.randomUUID() here — evaluated per pod per customer.
            // Pod A: "7a3f1b2c-..." for cust_001
            // Pod B: "c9d8e7f6-..." for cust_001  ← different UUID
            // Pod C: "b2a1c3d4-..." for cust_001  ← yet another UUID
            // Three commands dispatched for cust_001 — three @CommandHandler invocations
            // (serialized by aggregate lock) — three Stripe calls if no aggregate guard.
            String idempotencyKey = UUID.randomUUID().toString(); // BUG: per-pod UUID

            commandGateway.send(new ChargeBillingCommand(
                customer.getId(),
                customer.getMonthlyAmountCents(),
                billingPeriod,
                idempotencyKey // different per pod
            ));
        }
    }
}

// Even with the aggregate guard (billedPeriods.contains()), the race is subtle:
// The three commands for cust_001 are serialized by the aggregate lock.
// Command A (Pod A, key=UUID_A): billedPeriods does NOT contain "2026-11" → Stripe call → ch_A
//   CustomerBilledEvent applied → billedPeriods.add("2026-11") persisted
// Command B (Pod B, key=UUID_B): billedPeriods DOES contain "2026-11" → returns (guard fires)
// Command C (Pod C, key=UUID_C): billedPeriods DOES contain "2026-11" → returns (guard fires)
// Aggregate guard saves you if it's in place. Without it: ch_A, ch_B, ch_C.
//
// But with content-hash keys: all three commands carry the same key → same Stripe call
// even without the aggregate guard. Defence in depth: apply both.

There are three complementary fixes, each addressing a different layer of the stack:

// Fix 1: Use content-hash key at command construction time (fixes the UUID problem).
// All three pods generate the same key for the same customer in the same period.
// Even without a distributed lock, all three Stripe calls carry the same idempotency key.
// Stripe idempotency cache returns ch_A for all three calls — no duplicate charge.

@Scheduled(cron = "0 0 1 * * *")
public void runMonthlyBilling() {
    String billingPeriod = currentBillingPeriod();
    List<Customer> activeCustomers = customerRepository.findAllActive();

    for (Customer customer : activeCustomers) {
        // SAFE: content-hash key — same value on all three pods for the same customer.
        // All three Stripe calls carry the same idempotency key → ch_A cached.
        commandGateway.send(new ChargeBillingCommand(
            customer.getId(),
            customer.getMonthlyAmountCents(),
            billingPeriod
            // No idempotencyKey parameter — computed inside ChargeBillingCommand constructor
            // as sha256(customerId:billingPeriod:axon-billing)[:32]
        ));
    }
}

// Fix 2: ShedLock distributed lock — only one pod executes the billing loop.
// ShedLock uses a database lock table to elect one executor per scheduled task.
// The other two pods acquire the lock, see it's held, and return immediately.
// No concurrent command dispatch for the same billing period.

import net.javacrumbs.shedlock.spring.annotation.SchedulerLock;

@Scheduled(cron = "0 0 1 * * *")
@SchedulerLock(name = "monthly-billing-run",
               lockAtMostFor = "PT30M",
               lockAtLeastFor = "PT5M")
public void runMonthlyBillingWithLock() {
    String billingPeriod = currentBillingPeriod();
    List<Customer> activeCustomers = customerRepository.findAllActive();
    for (Customer customer : activeCustomers) {
        commandGateway.send(new ChargeBillingCommand(
            customer.getId(),
            customer.getMonthlyAmountCents(),
            billingPeriod
        ));
    }
}

// ShedLock configuration (Spring Boot auto-configuration via @EnableSchedulerLock):
// shedlock.default-lock-at-most-for=PT30M — lock held at most 30 min (prevents deadlock)
// shedlock.default-lock-at-least-for=PT5M — lock held at least 5 min (prevents double-run
//   if the billing job finishes fast and a second pod starts immediately after)

// Fix 3: Replace @Scheduled with Axon's EventScheduler — fires the billing event once.
// EventScheduler publishes a domain event to the event bus at the scheduled time.
// The Tracking Event Processor handles the event exactly once (by design — one processor
// per event position, not one per pod). No need for a distributed lock.

import org.axonframework.eventhandling.scheduling.EventScheduler;
import org.axonframework.eventhandling.scheduling.ScheduleToken;

// Schedule billing event at application start (idempotent — use token storage to avoid
// re-scheduling if the application restarts before the scheduled time).
ScheduleToken token = eventScheduler.schedule(
    Duration.ofDays(30), // 30 days from now
    new MonthlyBillingDueEvent(customerId, billingPeriod, amountCents)
);
// Store token to cancel/reschedule if needed.
// The TEP handles MonthlyBillingDueEvent exactly once — no @Scheduled on all pods.

Why the aggregate guard alone is not sufficient when commands carry different UUID keys

The aggregate’s billedPeriods.contains(billingPeriod) guard protects against the multi-pod scenario because the aggregate lock serializes concurrent commands to the same aggregate. Pod A’s command executes the Stripe call and persists the CustomerBilledEvent (adding the period to billedPeriods); Pod B’s command then runs the guard and returns without calling Stripe. This is correct behavior — but only because the aggregate state is rebuilt from the event store on each command handling, and the guard check runs before the Stripe call.

The risk is latency: all three commands are dispatched to the CommandBus nearly simultaneously. The aggregate lock ensures they are processed sequentially, but it does not prevent the three commands from being dispatched in the first place. If the application shuts down or the event store write fails between Pod A’s Stripe call (ch_A committed) and the CustomerBilledEvent being applied (which updates billedPeriods), the aggregate guard is not yet set, and Pod B’s command — if processed after an aggregate reload — calls Stripe again. The content-hash key is what prevents ch_B in this scenario: both Pod A’s and Pod B’s commands carry the same key, so Stripe’s idempotency cache returns ch_A without creating a new charge. The aggregate guard and the idempotency key are independent correctness mechanisms; both should be in place.

Configuring a vault key spend cap as a hard financial backstop

All three failure modes described above can in principle be fixed in application code. But application-level fixes depend on developers correctly applying the content-hash key pattern in every command object, every Saga field initialization, and every billing scheduler. A missing .substring(0, 32) on a hash, a UUID.randomUUID() left in a newly added command handler, or a new @Scheduled billing trigger added without @SchedulerLock can silently re-introduce the vulnerability. A vault key spend cap at the Stripe API layer provides a hard financial ceiling that enforces a maximum on total charges routed through a given key, regardless of the application logic that calls it.

The pattern for Axon Framework applications is to issue a separate vault key per billing period. The key is configured with a spend cap equal to the expected total for that period multiplied by 1.10 (ten percent headroom for legitimate over-billing edge cases). All ChargeBillingCommand handlers call the proxy endpoint rather than api.stripe.com directly. If a duplicate-charge bug routes more than 1.10× the expected total through the proxy in a single billing period, the proxy blocks the excess calls before they reach Stripe’s API. The billing run fails loudly rather than silently creating unbounded duplicate charges:

// Vault key per billing period — configured via Keybrake proxy before billing run starts.
// The vault key wraps the real Stripe secret key and enforces the spend cap.

// Step 1: Before the @Scheduled billing run, issue a vault key for this period.
// This is done once (or idempotently) in the billing orchestration service.
VaultKey vaultKey = keybrakeClient.issueVaultKey(VaultKeyRequest.builder()
    .vendor("stripe")
    .dailyUsdCap(expectedMonthlyTotal * 1.10) // 10% headroom
    .allowedEndpoints(List.of("/v1/charges", "/v1/customers"))
    .expiresAt(Instant.now().plus(Duration.ofDays(3))) // billing window
    .label("monthly-billing-" + billingPeriod)
    .build());

// Step 2: Configure the Stripe client to call the proxy with the vault key.
// The vault key is passed as the Authorization header — the proxy substitutes
// the real Stripe key after enforcing policy.
Stripe.overrideApiBase("https://proxy.keybrake.com/stripe");
Stripe.apiKey = vaultKey.getKey(); // "vault_key_xxx"

// Step 3: Billing run proceeds normally — ChargeBillingCommand handlers call
// Charge.create() against the proxy endpoint.
// If total charges exceed expectedMonthlyTotal * 1.10, the proxy returns 429.
// The application receives StripeException — billing run stops rather than
// creating unbounded charges due to a duplicate-key bug.

// Step 4: After the billing run, rotate the vault key (or let it expire).
// The audit log on the proxy dashboard shows every Stripe call made during
// the billing period: customer ID, charge ID, amount, timestamp, idempotency key.
// Duplicate calls are visible in the log as back-to-back entries with different
// idempotency keys for the same customer.

Summary: three failure modes, three root causes, one common fix layer

Across all three failure modes, the structural fix is the same: the Stripe idempotency key must be a deterministic function of the stable billing fields — computed before any retry, event replay, or concurrent dispatch boundary is crossed — and carried as a value in the command object or Saga state rather than generated at the point of the Stripe API call. The content-hash construction is sha256(customerId + ":" + billingPeriod + ":axon-billing")[:32]. The key must exclude UUID.randomUUID(), System.currentTimeMillis(), any handler-invocation counter, any pod-specific value, and any per-replay Axon framework metadata. Pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period) in PostgreSQL provides a durable billing mutex as a last-resort backstop when Stripe’s 24-hour idempotency cache window has expired.

For the Ratpack-specific failure where UUID.randomUUID() is inside a Blocking.get() supplier that gets re-called on each Promise.retry() re-subscription, see the Ratpack and Stripe Integration post. For Spring Retry’s @Retryable AOP proxy re-invocation, see Spring Retry and Stripe Integration. For Resilience4j’s @Retry and Retry.executeCallable(), see Resilience4j and Stripe Integration. The Axon failure modes described here are distinct from all of those: the re-invocation trigger is the CommandBus dispatch mechanism and the Tracking Event Processor’s token-commit semantics, not a retry annotation interceptor or a reactive operator re-subscription.