Quarkus SmallRye Fault Tolerance and Stripe Integration: How @Retry Re-executes the CDI Bean Method Body After billingRepository.persist() Fails Generating UUID_B on Each Attempt, @Fallback Calls a Second Stripe Charge with a Fresh UUID After @Retry Exhaustion Masks a Timeout That Committed ch_A, and @Timeout Fires After Stripe Responds but Before the DB Write Commits Causing @Retry to Double-Charge with UUID_B

The prior Quarkus post in this series covered three failure modes in the reactive messaging layer: @Incoming consumer nack-and-redeliver regenerating UUID on re-invocation, Mutiny chain deferred UUID evaluation regenerating UUID on Uni re-subscription, and @Outgoing Kafka emitter committing messages outside the JTA transaction boundary. All three are reactive messaging concerns. This post moves to synchronous CDI bean fault tolerance and focuses on three failure modes that arise from how SmallRye Fault Tolerance interceptors wrap CDI method invocations: the scope of what @Retry re-executes, the assumption that retry exhaustion implies no Stripe charge was committed, and the relationship between @Timeout firing and the Stripe call’s completion. Each mode produces the same observable: Stripe records two or three PaymentIntent or Charge objects for the same logical billing event, each carrying a distinct Idempotency-Key value, and the customer’s payment method is debited multiple times.

The common thread is a misunderstanding of what the SmallRye Fault Tolerance interceptor stack actually encompasses. SmallRye FT annotations are CDI interceptors. They operate at the CDI proxy boundary, not at the level of individual statements within the annotated method. When @Retry retries, it retries the entire annotated method invocation from line 1, including any UUID.randomUUID() calls, any precondition checks that the developer assumed would only run once, and any Stripe HTTP calls that had already completed. When @Timeout fires, it fires because the CDI interceptor’s timer expired — not because the Stripe HTTP call was cancelled or the Stripe server reversed the charge. When @Fallback runs, it runs because the retry budget was exhausted, not because no work was done during those retry attempts.

All three failure modes in this post are therefore instances of the same root cause applied to three different annotation combinations: the developer writes code that treats a SmallRye FT interceptor as applying to a subset of the method body, and the actual behavior is that it applies to the entire CDI proxy invocation.

Background: SmallRye Fault Tolerance interceptor chain and CDI proxy mechanics

SmallRye Fault Tolerance is the MicroProfile Fault Tolerance 4.x implementation used by Quarkus. It provides @Retry, @Fallback, @CircuitBreaker, @Timeout, @Bulkhead, and @RateLimiter annotations. Each annotation is implemented as a CDI interceptor. When a CDI bean method carries multiple fault tolerance annotations, SmallRye registers multiple interceptors at different CDI interceptor priority levels and composes them into a chain.

The interceptor execution order, from outermost to innermost, is: @Fallback → @Retry → @CircuitBreaker → @RateLimiter → @Timeout → @Bulkhead → (method body). When @Retry and @Fallback are combined, @Fallback wraps @Retry: @Fallback activates only after @Retry has exhausted its budget and re-thrown the exception. When @Retry and @Timeout are combined, @Retry wraps @Timeout: @Retry catches TimeoutException thrown by the @Timeout interceptor and re-attempts the entire invocation, including a fresh @Timeout window on each attempt.

The key architectural fact: when @Retry retries a method invocation, it calls the CDI proxy’s InvocationContext.proceed() again. This is equivalent to calling the annotated method again from the caller’s perspective. Every statement in the method body re-executes: local variable declarations, conditional checks, UUID.randomUUID() calls, external HTTP calls, and database writes. The @Retry interceptor has no knowledge of which statements inside the method body previously completed successfully. It retries the unit of work as a whole.

This is consistent with the MicroProfile Fault Tolerance specification, which defines @Retry’s scope as the annotated method. The specification does not define any sub-method retry semantics. SmallRye FT faithfully implements this: a @Retry-annotated method that fails partway through is retried in its entirety.

For the tests in this post, the stack is: Quarkus 3.x LTS, quarkus-smallrye-fault-tolerance, quarkus-hibernate-orm-panache, PostgreSQL 15 via Testcontainers, WireMock for Stripe stubs, and JUnit 5 with @QuarkusTest and @QuarkusIntegrationTest.

Mode 1: @Retry re-executes the entire CDI bean method body — UUID.randomUUID() at method entry generates UUID_B on each attempt — ch_B alongside committed ch_A

The developer writes a billing service that charges a customer on Stripe and records the result in the database. The service is annotated with @Retry(maxRetries = 3, delay = 1000) to handle transient database connectivity issues and @Transactional to wrap the Panache persist call. The idempotency key is generated at the top of the method with UUID.randomUUID().

// BillingService.java — unsafe: UUID generated at method entry + @Retry wrapping @Transactional

@ApplicationScoped
public class BillingService {

    @Inject
    StripeClient stripeClient;

    @Inject
    BillingAttemptRepository billingAttemptRepository;

    @Retry(maxRetries = 3, delay = 1000, retryOn = PersistenceException.class)
    @Transactional
    public ChargeResult chargeCustomer(String customerId, String billingPeriod, int amountCents) {
        // UUID generated at method entry — new value on EVERY @Retry attempt
        String idempotencyKey = UUID.randomUUID().toString();

        // Step 1: Stripe HTTP call — commits ch_A on Stripe's side on attempt 1
        ChargeResponse stripeResponse = stripeClient.createCharge(
            customerId,
            amountCents,
            idempotencyKey
        );

        // Step 2: Panache persist — can fail with PersistenceException
        BillingAttempt attempt = new BillingAttempt();
        attempt.idempotencyKey = idempotencyKey;
        attempt.customerId = customerId;
        attempt.billingPeriod = billingPeriod;
        attempt.amountCents = amountCents;
        attempt.stripeChargeId = stripeResponse.getChargeId();
        attempt.status = "SUCCEEDED";
        billingAttemptRepository.persist(attempt); // <-- throws PersistenceException on attempt 1

        return new ChargeResult(stripeResponse.getChargeId());
    }
}

The developer’s mental model: the @Transactional boundary ensures that if billingAttemptRepository.persist() fails, the entire database transaction rolls back and the system is in a clean state. The @Retry then re-tries the “failed operation,” which the developer understands to mean re-trying the database write. Because the prior attempt was rolled back, the developer believes the retry starts from a clean slate.

This reasoning is correct for the database. It is wrong for Stripe. The sequence of events when billingAttemptRepository.persist() throws on attempt 1 illustrates the failure:

1. Attempt 1 begins. @Retry interceptor invokes the CDI proxy chain. @Transactional interceptor opens a JTA transaction (sets up a JDBC Connection in non-autocommit mode).
2. Method body executes from line 1. UUID.randomUUID() generates key_A. stripeClient.createCharge(customerId, amountCents, key_A) is called. Stripe’s server receives the request with Idempotency-Key: key_A, processes the charge, stores key_A in its idempotency table, and returns HTTP 200 with charge ID ch_A.
3. billingAttemptRepository.persist(attempt) is called. The Panache persist generates an INSERT INTO billing_attempts .... A PersistenceException is thrown — for example, a unique constraint violation on subscription_id + billing_period from a concurrent write, or a foreign key violation because customerId refers to a deleted customer record.
4. The PersistenceException propagates out of the method body. The @Transactional interceptor catches it, marks the JTA transaction for rollback, and calls transaction.rollback(). The JDBC INSERT is undone. The database contains no record of this billing attempt.
5. The PersistenceException propagates to the @Retry interceptor. PersistenceException is in the retryOn list. @Retry waits 1000 ms.
6. Attempt 2 begins. @Retry re-invokes the CDI proxy chain. @Transactional opens a new JTA transaction. Method body executes from line 1. UUID.randomUUID() generates key_B (statistically guaranteed to be different from key_A). stripeClient.createCharge(customerId, amountCents, key_B) is called. Stripe’s server checks its idempotency table: key_B is not there (key_A is). Stripe treats this as a new, distinct charge request. Stripe creates ch_B and charges the customer’s payment method a second time.

The customer now has two charges on their account: ch_A from attempt 1 and ch_B from attempt 2. Both charges are on Stripe’s books as succeeded. The database still contains no record if attempt 2’s persist also fails, or contains only ch_B’s record if attempt 2’s persist succeeds. The missing ch_A record means the audit log is incomplete even if the customer is successfully double-charged.

Why the developer misconception is plausible

The developer’s intuition — “the transaction rolled back, so the prior attempt is fully undone” — is accurate for every part of the operation that is enrolled in the JTA transaction. Panache’s persist() uses the JPA EntityManager, whose writes are flushed to a JDBC connection managed by the JTA transaction manager. A rollback reverses those writes.

Stripe is not enrolled in the JTA transaction. Stripe is an external HTTP service. The stripeClient.createCharge() call goes over the network to Stripe’s servers. There is no XA resource adapter for Stripe. There is no two-phase commit protocol between the JTA transaction manager and Stripe. When the JTA transaction rolls back, it contacts the JDBC DataSource and reverses JDBC operations. It does not contact Stripe and reverse charge operations.

The second contributing factor is annotation proximity. @Retry and @Transactional are on the same method, which implies to the developer that they operate at the same level of abstraction. In reality, @Transactional operates inside @Retry — @Retry is outer and re-executes the entire transaction scope, including the Stripe call that precedes the JTA-managed writes.

Detection signal

In Stripe’s dashboard: two charges for the same customer and amount within seconds of each other, carrying distinct charge IDs (ch_A and ch_B) and distinct idempotency key values. In the database: only the ch_B record (if attempt 2 succeeded) — the absence of ch_A in the database while ch_A exists in Stripe is the exact signal. A Stripe webhook reconciliation job that cross-references Stripe’s charge log against the database’s billing_attempts table will surface this gap.

The fix

Two changes are required. First, replace UUID.randomUUID() with a deterministic content-hash key derived from the parameters that identify the billing event. Second, add a pre-Stripe idempotency check: query the database for an existing record with the same key before calling Stripe. If a record exists, return it. If not, call Stripe and immediately persist the result.

// BillingService.java — safe: content-hash key + pre-check + per-item persist

@ApplicationScoped
public class BillingService {

    @Inject
    StripeClient stripeClient;

    @Inject
    BillingAttemptRepository billingAttemptRepository;

    @Transactional
    public ChargeResult chargeCustomer(String customerId, String billingPeriod, int amountCents) {
        String idempotencyKey = contentHashKey(customerId, billingPeriod, amountCents);

        // Pre-Stripe idempotency check
        Optional<BillingAttempt> existing = billingAttemptRepository.findByIdempotencyKey(idempotencyKey);
        if (existing.isPresent()) {
            return new ChargeResult(existing.get().stripeChargeId); // already processed
        }

        // Insert PENDING row BEFORE Stripe — catches concurrent duplicate requests
        BillingAttempt attempt = new BillingAttempt();
        attempt.idempotencyKey = idempotencyKey;
        attempt.customerId = customerId;
        attempt.billingPeriod = billingPeriod;
        attempt.amountCents = amountCents;
        attempt.status = "PENDING";
        billingAttemptRepository.persist(attempt); // UNIQUE constraint on idempotency_key
        billingAttemptRepository.flush();           // surface DataIntegrityViolationException NOW

        // Call Stripe only if PENDING row was inserted without conflict
        ChargeResponse stripeResponse = stripeClient.createCharge(customerId, amountCents, idempotencyKey);

        attempt.stripeChargeId = stripeResponse.getChargeId();
        attempt.status = "SUCCEEDED";
        billingAttemptRepository.persist(attempt);

        return new ChargeResult(stripeResponse.getChargeId());
    }

    private String contentHashKey(String customerId, String billingPeriod, int amountCents) {
        String raw = customerId + ":" + billingPeriod + ":" + amountCents;
        return DigestUtils.sha256Hex(raw);
    }
}

With this fix, @Retry can be removed or left in place for transient connectivity errors unrelated to the idempotency key logic. If a retry occurs, the findByIdempotencyKey check finds the PENDING or SUCCEEDED row and returns early without calling Stripe again. If two concurrent requests arrive with the same key, only one will successfully insert the PENDING row — the other will receive DataIntegrityViolationException on flush() and can be handled as a duplicate request, not retried as a new charge.

Mode 2: @Fallback uses UUID.randomUUID() after @Retry exhaustion masks a StripeTimeoutException that committed ch_A on Stripe

The developer writes a billing service with @Retry(maxRetries = 2, retryOn = Exception.class) and @Fallback(fallbackMethod = "chargeCustomerFallback"). The primary method calls Stripe. The fallback method is intended as a “last resort” that uses a different payment strategy or logs a failure. Because the developer believes that all retry attempts threw exceptions before Stripe responded, the fallback also calls Stripe with a freshly generated UUID.randomUUID() — reasoning that the prior attempts “didn’t go through.”

// BillingService.java — unsafe: @Fallback with new UUID assumes prior @Retry attempts had no effect

@ApplicationScoped
public class BillingService {

    @Inject
    StripeClient stripeClient;

    @Inject
    BillingAttemptRepository billingAttemptRepository;

    @Retry(maxRetries = 2, delay = 500, retryOn = Exception.class)
    @Fallback(fallbackMethod = "chargeCustomerFallback")
    public ChargeResult chargeCustomer(String customerId, int amountCents, String requestId) {
        String idempotencyKey = "charge_" + requestId + "_attempt_" + System.nanoTime();
        // Developer uses nanoTime() suffix thinking it disambiguates — but System.nanoTime()
        // generates a new value on each invocation, so every attempt has a distinct key
        return stripe_charge_and_record(customerId, amountCents, idempotencyKey);
    }

    // Fallback: developer assumes all @Retry attempts threw without reaching Stripe
    public ChargeResult chargeCustomerFallback(String customerId, int amountCents, String requestId) {
        String freshKey = UUID.randomUUID().toString(); // "fresh start" assumption is wrong
        log.warn("All retries exhausted for customer {} — attempting fallback charge", customerId);
        return stripe_charge_and_record(customerId, amountCents, freshKey);
    }

    @Transactional
    ChargeResult stripe_charge_and_record(String customerId, int amountCents, String key) {
        ChargeResponse response = stripeClient.createCharge(customerId, amountCents, key);
        BillingAttempt attempt = new BillingAttempt(key, customerId, amountCents, response.getChargeId());
        billingAttemptRepository.persist(attempt);
        return new ChargeResult(response.getChargeId());
    }
}

The developer’s mental model: @Retry re-tries the chargeCustomer method up to 2 times. If all three attempts (original + 2 retries) throw an exception, the developer concludes that all three attempts failed before reaching Stripe — otherwise, at least one would have returned a successful ChargeResult instead of throwing. The @Fallback activates, and the developer generates a fresh UUID because “no prior charges exist under any key.”

This reasoning contains a hidden assumption: that an exception thrown by stripeClient.createCharge() means Stripe did not process the charge. That assumption is false for a specific class of exceptions.

How a Stripe charge can succeed on the server side and throw on the client side

The Stripe Java client sends an HTTP POST to https://api.stripe.com/v1/charges (or /v1/payment_intents). The client establishes a TCP connection, sends the HTTP request headers and body, and waits to read the HTTP response. The Stripe server receives the request, authorizes the card, creates the charge object, writes it to its database, sends back an HTTP 200 response with the charge JSON body, and closes the connection.

A StripeException wrapping a SocketTimeoutException or ConnectException can occur at any point in this sequence. Critically, a read timeout fires while the client is reading the response body — after the server has already written the response. If the read timeout fires after Stripe has sent the HTTP 200 with the charge JSON but before the client has fully read the response body, the Java HTTP client throws SocketTimeoutException. The Stripe Java client wraps this as a StripeException or ApiConnectionException. From the application’s perspective, stripeClient.createCharge() threw an exception. From Stripe’s perspective, the charge was successfully committed to their database. The charge has a charge ID and an idempotency key entry in Stripe’s idempotency table.

The sequence in the failure scenario:

1. Attempt 1 of chargeCustomer: idempotencyKey = "charge_" + requestId + "_attempt_" + System.nanoTime() = key_A. stripeClient.createCharge(customerId, amountCents, key_A) is called. Stripe processes the charge and returns HTTP 200 with ch_A. The TCP connection drops mid-response-body transfer (network flap). The Stripe Java client throws StripeException(cause=SocketTimeoutException). ch_A is committed on Stripe’s side.
2. @Retry catches the exception. Waits 500 ms.
3. Attempt 2: System.nanoTime() generates a new value → key_B. stripeClient.createCharge(customerId, amountCents, key_B) is called. If the network is now stable, Stripe processes key_B as a new charge (key_A and key_B are distinct) and returns ch_B. The call succeeds. billingAttemptRepository.persist() saves key_B/ch_B. chargeCustomer returns ChargeResult(ch_B). The method succeeded on attempt 2 — @Retry stops here, @Fallback does not fire. But the customer has been charged both ch_A (from attempt 1’s timeout-masked Stripe call) and ch_B (from attempt 2).

In this variant, the double-charge happens during the @Retry attempts themselves, before @Fallback even fires. The @Fallback problem is a second layer on top: if attempt 2 also times out and throws, @Retry exhausts its budget and @Fallback fires with a third UUID, potentially creating ch_C for a customer who may already have ch_A and ch_B pending or confirmed on Stripe’s side.

The variant where @Fallback alone causes the extra charge: all three chargeCustomer attempts succeed on Stripe’s side and throw on the client side (all three are read-timeout masked). No charge ID is returned by any attempt, so no billingAttemptRepository.persist() runs. @Fallback fires. The fallback generates a fresh UUID. Stripe creates a fourth charge, ch_D. Stripe’s idempotency table has entries for key_A, key_B, key_C, and key_D — four charges for one customer billing event.

Why the developer misconception is plausible

The Java programming model presents exceptions as the signal that an operation did not succeed. stripeClient.createCharge() throwing means — in a local, in-process model — that the function did not complete and returned no result. In a distributed system, the semantics diverge: the server-side operation can succeed while the client-side response delivery fails. This divergence is a property of network communication, not of the Stripe client library. Any HTTP POST to any external service can produce the same “server processed, client timed out” outcome.

The @Fallback annotation amplifies the problem by providing a natural location for “if all else fails, do something different” logic. The developer writes the fallback to “start fresh,” which is correct behavior for operations where each attempt is truly independent (pure reads, idempotent state checks). For Stripe charges, “starting fresh” with a new UUID means creating a new financial transaction that Stripe has no way to correlate with the possibly-committed prior attempts.

Detection signal

In Stripe’s dashboard: multiple charges for the same customer and amount within the @Retry delay window (seconds), each with a distinct idempotency key that shares a common prefix ("charge_" + requestId) up to the _attempt_ segment. A Stripe webhook charge.succeeded handler that records the charge.id will capture multiple succeeded events for the same customer. Correlating by metadata.requestId (if the developer passes it in metadata) reveals duplicates. In the application database: the billing_attempts table has fewer rows than Stripe’s charge count for the same customer and period — the gap represents timeout-masked charges that reached Stripe but not the database.

The fix

Three changes together close the vulnerability:

First, remove System.nanoTime() from the key. Use a content-hash or a pre-generated, stable identifier passed in as a parameter. A content-hash over customerId + billingPeriod + amountCents produces the same key for the same billing event regardless of which attempt or fallback code path generates it.

Second, make the fallback method use the same key derivation as the primary method. The fallback should not generate a new UUID. A fallback that uses the same key as the primary attempts will be deduplicated by Stripe if a prior attempt reached Stripe — Stripe will return the existing charge result for the same key.

Third, add a pre-Stripe database idempotency check in both the primary method and the fallback. If Stripe committed ch_A and the timeout masked it, the billing_attempts table has no record of ch_A (the database write never completed). But the pre-Stripe check with a content-hash key will not find it either — so the check alone is insufficient. The complete defense requires Stripe-side deduplication via a stable idempotency key plus a reconciliation job that periodically queries Stripe for charges by customer and billing period and marks any database gaps.

// BillingService.java — safe: stable content-hash key used in both primary and fallback

@ApplicationScoped
public class BillingService {

    @Inject
    StripeClient stripeClient;

    @Inject
    BillingAttemptRepository billingAttemptRepository;

    @Retry(maxRetries = 2, delay = 500, retryOn = StripeException.class)
    @Fallback(fallbackMethod = "chargeCustomerFallback")
    public ChargeResult chargeCustomer(String customerId, int amountCents, String billingPeriod) {
        // Stable key: same value regardless of which attempt runs
        String idempotencyKey = contentHashKey(customerId, billingPeriod, amountCents);
        return stripe_charge_and_record(customerId, amountCents, idempotencyKey);
    }

    // Fallback uses THE SAME key — Stripe deduplicates if primary attempt committed ch_A
    public ChargeResult chargeCustomerFallback(String customerId, int amountCents, String billingPeriod) {
        String idempotencyKey = contentHashKey(customerId, billingPeriod, amountCents); // identical
        log.warn("All retries exhausted for customer {} — checking via stable key {}", customerId, idempotencyKey);

        // First: check if Stripe already committed a charge under this key
        Optional<BillingAttempt> existing = billingAttemptRepository.findByIdempotencyKey(idempotencyKey);
        if (existing.isPresent()) {
            return new ChargeResult(existing.get().stripeChargeId); // found from a prior DB-persisted attempt
        }

        // Not in DB, but may be in Stripe (timeout-masked) — call Stripe with same key
        // Stripe returns existing ch_A for key if committed, or creates new charge if not
        return stripe_charge_and_record(customerId, amountCents, idempotencyKey);
    }

    @Transactional
    ChargeResult stripe_charge_and_record(String customerId, int amountCents, String key) {
        Optional<BillingAttempt> existing = billingAttemptRepository.findByIdempotencyKey(key);
        if (existing.isPresent()) {
            return new ChargeResult(existing.get().stripeChargeId);
        }
        ChargeResponse response = stripeClient.createCharge(customerId, amountCents, key);
        BillingAttempt attempt = new BillingAttempt(key, customerId, amountCents, response.getChargeId());
        billingAttemptRepository.persist(attempt);
        return new ChargeResult(response.getChargeId());
    }

    private String contentHashKey(String customerId, String billingPeriod, int amountCents) {
        return DigestUtils.sha256Hex(customerId + ":" + billingPeriod + ":" + amountCents);
    }
}

With a stable key, a timeout-masked ch_A and a fallback call with the same key produce the same idempotency key at Stripe. Stripe returns ch_A for the fallback call (since ch_A was committed under that key) instead of creating ch_B. The customer is charged exactly once regardless of how many attempts and timeouts occurred.

Mode 3: @Timeout fires after Stripe responds but before billingRepository.persist() commits — @Retry re-executes the method body with UUID_B — ch_B created

The developer adds @Timeout to a billing service method to prevent it from hanging indefinitely on a slow Stripe connection or a slow database. The developer reasons: “if the @Timeout fires, the operation was cancelled — no charge was committed and no DB record was written — @Retry can safely re-try from scratch.” The @Retry annotation retries on TimeoutException. The idempotency key is generated with UUID.randomUUID() inside the method.

// BillingService.java — unsafe: @Timeout + @Retry + UUID.randomUUID() inside method body

@ApplicationScoped
public class BillingService {

    @Inject
    StripeClient stripeClient;

    @Inject
    BillingAttemptRepository billingAttemptRepository;

    @Retry(maxRetries = 2, delay = 300, retryOn = {TimeoutException.class, PersistenceException.class})
    @Timeout(2000) // 2-second wall-clock timeout on the entire method
    @Transactional
    public ChargeResult chargeCustomer(String customerId, int amountCents) {
        String idempotencyKey = UUID.randomUUID().toString(); // UUID_A on attempt 1

        // Stripe HTTP call — typically 1600-1900ms on heavy load
        ChargeResponse stripeResponse = stripeClient.createCharge(
            customerId,
            amountCents,
            idempotencyKey
        );
        // Stripe has now committed ch_A for idempotencyKey = UUID_A
        // Elapsed time: ~1800ms

        // Database persist — typically 50ms, but can spike to 300ms+ under write pressure
        BillingAttempt attempt = new BillingAttempt();
        attempt.idempotencyKey = idempotencyKey;
        attempt.customerId = customerId;
        attempt.amountCents = amountCents;
        attempt.stripeChargeId = stripeResponse.getChargeId();
        attempt.status = "SUCCEEDED";
        billingAttemptRepository.persist(attempt);
        // At t=2000ms: @Timeout fires HERE, throwing TimeoutException
        // The persist was in progress (secondary index update) when timeout fired

        return new ChargeResult(stripeResponse.getChargeId());
    }
}

The failure sequence:

1. Attempt 1 begins at t=0. @Retry invokes the CDI chain. @Timeout starts a 2000ms timer. @Transactional opens a JTA transaction.
2. Method body starts. UUID.randomUUID() generates key_A. stripeClient.createCharge(customerId, amountCents, key_A) is called.
3. t=1800ms: Stripe responds HTTP 200 with ch_A. ch_A is committed in Stripe’s database. The Stripe Java client returns ChargeResponse. The method body proceeds to billingAttemptRepository.persist(attempt).
4. t=1800ms + ε: billingAttemptRepository.persist(attempt) begins executing the INSERT INTO billing_attempts. The query is queued behind concurrent writes on the database server. The secondary index update on (customer_id, billing_period) acquires a row-level lock.
5. t=2000ms: The @Timeout interceptor’s timer fires. SmallRye FT throws org.eclipse.microprofile.faulttolerance.exceptions.TimeoutException at the CDI interceptor boundary.
6. The TimeoutException propagates inward through the method body (as a CDI interceptor exception, not a thread interrupt — the database statement may still be executing in the background depending on JDBC driver behavior). The @Transactional interceptor catches the exception, marks the JTA transaction for rollback, and calls transaction.rollback(). The INSERT INTO billing_attempts is rolled back. The database has no record of this billing attempt.
7. TimeoutException propagates to @Retry. TimeoutException is in the retryOn set. @Retry waits 300 ms.
8. Attempt 2 begins. @Timeout starts a fresh 2000ms timer. @Transactional opens a new JTA transaction. Method body re-executes from line 1. UUID.randomUUID() generates key_B. stripeClient.createCharge(customerId, amountCents, key_B) is called. Stripe checks its idempotency table: key_B is not there (key_A is). Stripe creates ch_B. Customer charged twice.

Mechanics of SmallRye FT @Timeout for synchronous CDI methods

SmallRye FT implements @Timeout for synchronous CDI methods by running the method in the calling thread and checking wall-clock time at the CDI proxy. The MicroProfile Fault Tolerance specification requires that @Timeout interrupt the ongoing execution when the timeout elapses. SmallRye FT’s implementation uses thread interruption for this: when the timeout fires, the calling thread is interrupted (Thread.interrupt() is called). Whether this actually stops the in-progress Stripe HTTP call or database write depends on whether those operations check the interrupted flag.

The Stripe Java client uses OkHttp (or HttpURLConnection in older versions) for HTTP. OkHttp does not immediately abort in-progress HTTP reads when the Java thread’s interrupted flag is set — it will eventually detect interruption at a blocking I/O point, but not instantly. The JDBC driver’s behavior on thread interruption is driver-specific; most PostgreSQL JDBC drivers do not abort an in-progress query when the thread is interrupted unless Statement.cancel() is explicitly called.

The practical consequence: by the time the thread is interrupted (and InterruptedException is eventually thrown and wrapped into TimeoutException by SmallRye FT), the Stripe HTTP call may have already returned with a committed charge. The @Timeout fires during a later, slower phase of the method (the database write). The interrupt signal terminates the method body at the database write phase, not at the Stripe call phase.

This is the critical point: @Timeout does not guarantee atomicity. It does not guarantee that all side effects of the annotated method are reversed when the timeout fires. It guarantees only that the CDI interceptor’s observation of the method’s completion is bounded in time. A method that calls Stripe (external, non-transactional) and then writes to the database (transactional) will have its transactional side effects rolled back on timeout, but its external side effects (the Stripe charge) will not be reversed.

Why the developer misconception is plausible

The name “@Timeout” implies that the operation was cancelled before completing. In synchronous sequential code, “not completing” means no observable outcome was produced. For a method that reads a database and returns a result, a timeout means no data was returned — the operation was effectively a no-op. The developer generalizes this to all operations, including those with non-transactional external side effects.

The second contributing factor is the combination with @Transactional. When the timeout fires and @Transactional rolls back, the database returns to its prior state. This rollback provides visible evidence that “something was undone.” The developer may conflate “the database was rolled back” with “the entire operation was undone,” not recognizing that the database rollback applies only to JDBC operations, not to the Stripe charge that preceded them.

Detection signal

In Stripe’s dashboard: charges for the same customer within seconds of each other, each with a unique idempotency key (since keys are from UUID.randomUUID()). The timing gap between charges corresponds approximately to the @Retry delay (300 ms in this example). In the database: if all attempts timeout, the billing_attempts table has no record at all while Stripe has multiple charges — a reconciliation query against the Stripe API (listing charges by customer, past N minutes) reveals the gap. If only attempt 1 times out and attempt 2 succeeds within the timeout, the database has only ch_B’s record — ch_A exists on Stripe but not in the database.

The fix

Two changes. First, replace UUID.randomUUID() with a content-hash key. Second, add a pre-Stripe idempotency check. The fix is structurally identical to Mode 1’s fix: a stable key ensures that @Retry’s re-execution of the method body presents the same key to Stripe on every attempt. Stripe’s own idempotency enforcement then deduplicates at the network layer even before the application’s pre-check finds a database record.

// BillingService.java — safe: content-hash key + pre-check before Stripe

@ApplicationScoped
public class BillingService {

    @Inject
    StripeClient stripeClient;

    @Inject
    BillingAttemptRepository billingAttemptRepository;

    @Retry(maxRetries = 2, delay = 300, retryOn = {TimeoutException.class, PersistenceException.class})
    @Timeout(3000) // raised to accommodate slower DB writes
    @Transactional
    public ChargeResult chargeCustomer(String customerId, String billingPeriod, int amountCents) {
        // Content-hash: same value on attempt 1, 2, 3 — same key reaches Stripe each time
        String idempotencyKey = contentHashKey(customerId, billingPeriod, amountCents);

        // Pre-Stripe check: if a prior attempt committed to DB, return early
        Optional<BillingAttempt> existing = billingAttemptRepository.findByIdempotencyKey(idempotencyKey);
        if (existing.isPresent()) {
            return new ChargeResult(existing.get().stripeChargeId);
        }

        // Insert PENDING before Stripe call
        BillingAttempt attempt = new BillingAttempt();
        attempt.idempotencyKey = idempotencyKey;
        attempt.customerId = customerId;
        attempt.billingPeriod = billingPeriod;
        attempt.amountCents = amountCents;
        attempt.status = "PENDING";
        billingAttemptRepository.persist(attempt);
        billingAttemptRepository.flush(); // surface UNIQUE violation immediately

        ChargeResponse stripeResponse = stripeClient.createCharge(customerId, amountCents, idempotencyKey);

        attempt.stripeChargeId = stripeResponse.getChargeId();
        attempt.status = "SUCCEEDED";

        return new ChargeResult(stripeResponse.getChargeId());
    }

    private String contentHashKey(String customerId, String billingPeriod, int amountCents) {
        return DigestUtils.sha256Hex(customerId + ":" + billingPeriod + ":" + amountCents);
    }
}

With the content-hash key, if @Timeout fires on attempt 1 after Stripe has committed ch_A, attempt 2 calls Stripe with the same key. Stripe’s idempotency enforcement finds key in its table (from attempt 1’s request) and returns ch_A without creating a new charge. The customer is charged once. Note: Stripe’s idempotency cache for a given key expires after 24 hours. Within that window, any number of retries with the same key are safe.

For the @Timeout value itself: raising it from 2000ms to 3000ms (or removing it and relying on Stripe’s HTTP client timeout settings) eliminates the specific timing condition where Stripe responds within 2 seconds but the DB write takes the timeout over. The @Timeout value should be set higher than the sum of the maximum expected Stripe HTTP response time and the maximum expected database write time, plus a safety margin. If both can independently approach 2 seconds under load, a 2-second @Timeout is structurally unsafe for this operation.

Integration tests with @QuarkusTest, WireMock, and Testcontainers

Testing SmallRye Fault Tolerance behavior requires that the Quarkus application runs with the fault tolerance interceptors active. @QuarkusTest starts the full Quarkus application context including CDI interceptors. @QuarkusIntegrationTest starts the packaged application. Fault tolerance annotations on CDI beans are processed by SmallRye FT’s interceptor during CDI bean resolution — they are not visible to tests that instantiate service classes directly.

WireMock is used to stub Stripe’s API. The Stripe Java client is configured with a base URL pointing to WireMock’s local server. WireMock’s request journal records all HTTP calls made to the stub server, including the Idempotency-Key header value on each call. Asserting that the same idempotency key was used across retry attempts is the core test assertion for all three modes.

// BillingServiceRetryIdempotencyTest.java — Mode 1: @Retry regenerates UUID on method re-entry

@QuarkusTest
class BillingServiceRetryIdempotencyTest {

    @WireMockInjectServer
    WireMockServer wireMock;

    @Inject
    BillingService billingService;

    @Inject
    BillingAttemptRepository billingAttemptRepository;

    @BeforeEach
    void setup() {
        // Stripe call always succeeds — we control the DB failure separately
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_test\",\"object\":\"charge\",\"status\":\"succeeded\"}")));
    }

    @Test
    @Transactional
    void retry_generates_new_uuid_on_each_attempt_when_persist_fails() {
        // Configure billingAttemptRepository to fail persist on first call, succeed on second
        // In practice: use a Quarkus mock or a controlled test data state that causes
        // a constraint violation on the first persist call

        // Act: call the service (which will fail persist, trigger @Retry, succeed on attempt 2)
        // Note: with UUID.randomUUID(), attempt 1 and attempt 2 use different keys
        // This test verifies the key difference — the FIX test would verify the same key
        assertThrows(PersistenceException.class, () -> billingService.chargeCustomer(
            "cus_test", "2026-10", 9900
        ));

        // Assert: WireMock request journal shows TWO distinct Idempotency-Key values
        List<LoggedRequest> stripeRequests = wireMock.findAll(postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests).hasSize(2); // one per @Retry attempt that reached Stripe

        String keyAttempt1 = stripeRequests.get(0).getHeader("Idempotency-Key");
        String keyAttempt2 = stripeRequests.get(1).getHeader("Idempotency-Key");

        // UNSAFE pattern detected: different keys = potential double-charge
        assertThat(keyAttempt1).isNotEqualTo(keyAttempt2);
        // The FIXED version of this test would assert:
        // assertThat(keyAttempt1).isEqualTo(keyAttempt2);
    }
}

// BillingServiceTimeoutIdempotencyTest.java — Mode 3: @Timeout fires after Stripe responds

@QuarkusTest
class BillingServiceTimeoutIdempotencyTest {

    @WireMockInjectServer
    WireMockServer wireMock;

    @Inject
    BillingService billingService;

    @Test
    void timeout_after_stripe_response_causes_retry_with_new_uuid() throws Exception {
        // Stub Stripe to respond after 1800ms (within @Timeout(2000) but close to the wire)
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withFixedDelay(1800) // 1800ms delay — Stripe responds at t=1800ms
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_test\",\"object\":\"charge\",\"status\":\"succeeded\"}")));

        // The @Timeout fires at t=2000ms during billingRepository.persist() (the 200ms gap)
        // @Retry retries with a new UUID — WireMock records both keys

        // In a controlled test environment, the @Timeout threshold is lowered to
        // reproduce the condition reliably. With @Timeout(2000) and WireMock fixed delay 1800ms,
        // whether the timeout fires during persist() depends on JVM scheduling and DB speed.
        // A test-specific @Timeout override (via quarkus.fault-tolerance.timeout=1850ms) or
        // a WireMock delay of 1950ms (leaving only 50ms for persist()) makes the condition reliable.

        // Assert: WireMock received two calls with distinct Idempotency-Key values
        List<LoggedRequest> stripeRequests = wireMock.findAll(postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(stripeRequests.size()).isGreaterThanOrEqualTo(2);

        // Key assertion: the keys differ, proving @Retry re-generated UUID on method re-entry
        String key1 = stripeRequests.get(0).getHeader("Idempotency-Key");
        String key2 = stripeRequests.get(1).getHeader("Idempotency-Key");
        assertThat(key1).isNotEqualTo(key2); // UNSAFE: two distinct keys = two Stripe charges
    }
}

// BillingServiceFallbackIdempotencyTest.java — Mode 2: @Fallback with fresh UUID

@QuarkusTest
class BillingServiceFallbackIdempotencyTest {

    @WireMockInjectServer
    WireMockServer wireMock;

    @Inject
    BillingService billingService;

    @Test
    void fallback_generates_new_uuid_after_all_retries_throw() {
        // Stub: Stripe always throws StripeException (simulates read timeout)
        // In WireMock, simulate this with a fault response
        wireMock.stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withFault(Fault.CONNECTION_RESET_BY_PEER))); // simulates connection failure

        // Act: all attempts throw StripeException — @Fallback fires
        // The fallback calls Stripe with UUID.randomUUID() — WireMock records the key

        // To test the FALLBACK key specifically, stub Stripe to succeed only on the
        // fallback call (distinguished by a header or path the fallback sets)
        // For simplicity here, we verify the key count and distinctness

        assertThrows(Exception.class, () -> billingService.chargeCustomer(
            "cus_test", 9900, "2026-10"
        ));

        // @Retry made 3 attempts (1 original + 2 retries), @Fallback made 1 attempt = 4 total
        // But because Stripe is always failing (CONNECTION_RESET), only fallback might get through
        // if the stub is changed for the fallback scenario
        // The key test: verify the fallback key differs from any primary attempt key

        // In a real scenario, log the keys from WireMock journal and assert distinctness
        // between primary attempts and fallback attempt
        List<LoggedRequest> stripeRequests = wireMock.findAll(postRequestedFor(urlEqualTo("/v1/charges")));
        // All keys should be distinct (each attempt and fallback uses UUID.randomUUID())
        List<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toList());
        long distinctKeyCount = keys.stream().distinct().count();
        assertThat(distinctKeyCount).isEqualTo(keys.size()); // all distinct = never reusing a key
        // The FIXED version asserts: distinctKeyCount == 1 (same key across all attempts and fallback)
    }
}

Comparison table: three SmallRye Fault Tolerance Stripe failure modes

What all three modes share: the CDI proxy boundary is not a transaction boundary

The three failure modes described in this post have a single underlying cause: the developer treats the SmallRye Fault Tolerance CDI proxy boundary as a transaction boundary that encompasses all side effects of the annotated method, including external HTTP calls to Stripe.

A transaction boundary has a specific meaning: changes enrolled in the transaction are atomic, consistent, isolated, and durable. They can be rolled back atomically. They are isolated from concurrent operations until committed. Stripe is not enrolled in the JTA transaction. The CDI proxy boundary, where @Retry, @Timeout, and @Fallback operate, is a call interception boundary — it observes method entry and exit, and can re-invoke the method or redirect to a fallback. It has no mechanism to undo external HTTP calls.

This distinction matters because it defines the safe scope of retry logic. Retrying a database write is safe when the database enforces a uniqueness constraint that prevents duplicate records (the retry hits the constraint and is identified as a duplicate, not silently inserted). Retrying a Stripe charge call is safe only when the retry uses the same idempotency key — Stripe’s own idempotency enforcement then provides the equivalent of a uniqueness constraint at the external API level.

The pattern that makes all three modes safe is the same:

1. Derive the idempotency key from the billing event’s content, not from UUID.randomUUID(). A content-hash key is deterministic: the same event always produces the same key, regardless of which attempt, retry, fallback, or service instance generates it.
2. Insert a PENDING record in the database before calling Stripe. Use a UNIQUE constraint on idempotency_key. Catch DataIntegrityViolationException on this insert as the concurrent-duplicate guard. Do not rely on a SELECT-then-INSERT pattern — the gap between SELECT and INSERT allows concurrent duplicates through.
3. After Stripe responds, update the PENDING record to SUCCEEDED with the charge ID. If the update fails (DB connectivity, timeout), the PENDING record acts as a tombstone: the next retry’s pre-check finds the PENDING row, confirms the charge ID via Stripe’s retrieve-by-idempotency-key endpoint, and updates the record directly without creating a new charge.

This three-step pattern is safe against all three failure modes: @Retry method re-entry (the pre-check finds the row and returns early), @Fallback with fresh UUID (the fallback uses the same content-hash key, which Stripe deduplicates), and @Timeout during the DB write (the next attempt finds the PENDING row if the pre-Stripe INSERT committed, or finds nothing and retries both the INSERT and the Stripe call with the same key).

Positioning @Retry scope relative to the Stripe call

An alternative architectural approach is to narrow the scope of @Retry to operations where retry is unambiguously safe: database reads, database writes with idempotency guards, and idempotent external reads (Stripe retrieve-charge by ID). The Stripe charge creation call itself — stripeClient.createCharge() — is placed in a method that is not annotated with @Retry. Only the outer orchestration method, which includes the pre-check and post-persist logic, is retried, and only on the specific exceptions where retrying is safe (database constraint violations that indicate a pre-existing record, not all exceptions).

// BillingOrchestrator.java — narrow retry scope: @Retry only wraps idempotency-safe operations

@ApplicationScoped
public class BillingOrchestrator {

    @Inject
    StripeGateway stripeGateway;     // no @Retry on Stripe calls

    @Inject
    BillingAttemptRepository repo;

    @Retry(maxRetries = 2, delay = 500, retryOn = DataIntegrityViolationException.class)
    @Transactional
    public ChargeResult chargeCustomer(String customerId, String billingPeriod, int amountCents) {
        String key = contentHashKey(customerId, billingPeriod, amountCents);

        Optional<BillingAttempt> existing = repo.findByIdempotencyKey(key);
        if (existing.isPresent() && "SUCCEEDED".equals(existing.get().status)) {
            return new ChargeResult(existing.get().stripeChargeId);
        }

        // Insert PENDING: DataIntegrityViolationException if concurrent duplicate
        BillingAttempt attempt = new BillingAttempt(key, customerId, billingPeriod, amountCents, "PENDING");
        repo.persist(attempt);
        repo.flush(); // <-- @Retry retries on DataIntegrityViolationException from here

        // Beyond this point: Stripe call is NOT inside the @Retry scope's safe window
        // But because the PENDING row is committed, a retry will hit the existing.isPresent()
        // check above and return early — @Retry never reaches this line on a genuine retry
        ChargeResponse response = stripeGateway.createCharge(customerId, amountCents, key);

        attempt.stripeChargeId = response.getChargeId();
        attempt.status = "SUCCEEDED";

        return new ChargeResult(response.getChargeId());
    }
}

In this design, @Retry(retryOn = DataIntegrityViolationException.class) retries only if the PENDING row insert throws a constraint violation — which means a concurrent request already inserted the row. The retry immediately finds the row in the existing.isPresent() check and returns early. The Stripe charge creation call is never retried by @Retry because no exception from the Stripe call is in the retryOn list. If the Stripe call throws, the exception propagates past @Retry (not retried) to the caller, which can decide how to handle it (for example, by marking the PENDING record as FAILED and scheduling a reconciliation).

This approach is more conservative but more precisely targeted: the fault tolerance annotations apply only to operations where their retry semantics are well-defined and safe.

Keybrake and SmallRye Fault Tolerance Stripe integrations

The three failure modes in this post — @Retry method body re-entry with new UUID, @Fallback creating a third charge under the assumption that retry exhaustion implies no prior Stripe commit, and @Timeout firing after Stripe processes the charge — are all invisible to the Stripe API caller at the moment they occur. The Stripe API returns 200 OK for ch_A, ch_B, and ch_C individually. There is no error signal from Stripe indicating that a duplicate charge was created. The detection is post-hoc: reconciling Stripe’s charge log against the application’s billing_attempts table, or monitoring Stripe’s webhook events for multiple charge.succeeded events for the same customer within a short window.

Keybrake operates as a proxy between your Quarkus application and the Stripe API. At the proxy layer, every request carries your vault key, and every Stripe response is parsed for the charge ID and amount. When Keybrake’s audit log shows two charge objects for the same customer within the same billing window — each with a distinct idempotency key — that is the earliest possible detection point: before any reconciliation job runs, before the billing_attempts table is queried, immediately after the second charge is committed on Stripe’s side. The proxy’s per-vendor daily cap is a second line of defense: a runaway retry loop or fault-tolerance misconfiguration that generates dozens of charges will trip the cap and halt further charges before the damage compounds beyond a threshold you control.