Spring Cloud Gateway Filters and Stripe Integration: How an AddRequestHeaderGatewayFilter SpEL Expression Evaluates UUID.randomUUID() on Each RetryGatewayFilter Attempt Generating UUID_B Committing ch_B, a CircuitBreakerGatewayFilter Fallback Route Calls Stripe with UUID_B After the Primary Route Already Committed ch_A Before the Timeout Triggered the Fallback, and a WebClient retryWhen Inside a Custom GatewayFilter Re-Subscribes the Reactive Pipeline Generating UUID_B on Each WebClient Retry Attempt

This post is part of the ongoing series on Stripe idempotency failure modes in JVM frameworks. Earlier entries covered Spring Cloud Gateway (GlobalFilter.filter() UUID regeneration per retry, ModifyRequestBodyGatewayFilterFactory rewrite function re-execution, contextWrite() per Reactor subscription), Spring Cloud Contract (WireMock consumer stub anyNonBlankString() masking, producer verification test happy-path gap, messaging contract @KafkaListener re-delivery UUID), Spring Cloud Task (SCDF relaunch new executionId, TaskExecutionListener.onTaskFailed() compensation post-commit, TASK_EXECUTION end_time IS NULL guard misread), and Spring Integration (RequestHandlerRetryAdvice UUID regeneration, JdbcChannelMessageStore transactional poller rollback requeue, JdbcMessageGroupStore aggregator crash double-release). This post covers three additional Spring Cloud Gateway filter failure modes that are distinct from the GlobalFilter, ModifyRequestBody, and contextWrite() modes documented in the earlier entry: the AddRequestHeaderGatewayFilterFactory with a SpEL expression that evaluates UUID.randomUUID() per retry attempt, the CircuitBreakerGatewayFilterFactory whose fallback route calls Stripe independently after the primary route already committed a charge before the timeout tripped the circuit, and a custom GatewayFilter that uses WebClient with a retryWhen(Retry.backoff()) spec where UUID.randomUUID() is embedded inside a flatMap operator that re-executes on each WebClient retry subscription.

The three failure modes are: a developer who configures a Spring Cloud Gateway route with the built-in AddRequestHeader filter using a SpEL expression #{T(java.util.UUID).randomUUID().toString()} as the header value, believing that the expression is evaluated once per client request and the same value is forwarded on every retry attempt, when in fact RetryGatewayFilterFactory re-executes the entire gateway filter chain per retry attempt, causing the AddRequestHeaderGatewayFilterFactory to evaluate the SpEL expression again and inject a new UUID_B into the retried request, which the upstream billing service uses as the Stripe idempotency key, committing ch_B alongside the already-committed ch_A; a developer who configures CircuitBreakerGatewayFilterFactory with a fallbackUri: forward:/billing/fallback controller under the assumption that the circuit breaker trips only when the primary service fails before doing meaningful work, when in fact the primary billing service may call Stripe (committing ch_A) and then take longer than the configured response timeout to return the HTTP response, causing the gateway to dispatch the fallback route, and where the fallback controller independently generates UUID.randomUUID() and calls Stripe a second time committing ch_B; and a developer who writes a custom GatewayFilter using WebClient with a .retryWhen(Retry.backoff(3, Duration.ofSeconds(1)).filter(e -> e instanceof WebClientRequestException)) spec, where inside the reactive chain a Mono.fromCallable(() -> UUID.randomUUID().toString()) is chained via .flatMap() before the WebClient call, under the mistaken belief that the Mono.fromCallable is evaluated once and its value is cached for all retry attempts, when in fact WebClient’s retry re-subscribes to the source Publisher, re-evaluating the entire reactive assembly including the Mono.fromCallable, generating UUID_B on the second attempt, forwarding the new key to the downstream billing service, and causing Stripe to commit ch_B.

Stack for examples: Spring Boot 3.x, Spring Cloud Gateway 4.x (spring-cloud-starter-gateway), Spring Cloud Circuit Breaker (spring-cloud-starter-circuitbreaker-reactor-resilience4j), Spring WebFlux, Reactor Core 3.x, Stripe Java SDK v24+, Resilience4j 2.x, WireMock 3.x, JUnit 5, Testcontainers 1.20+.

Background: how Spring Cloud Gateway filter chains interact with Stripe idempotency differently than per-request filters

Spring Cloud Gateway is a reactive API gateway built on Spring WebFlux and Project Reactor. Each route is a composition of predicates and filters. When an incoming request matches a route, the gateway assembles a filter chain: all global filters (beans that implement GlobalFilter) plus all per-route GatewayFilter instances (configured in the route definition’s filters list), ordered by Ordered precedence. The chain is executed as a Reactor Mono<Void> pipeline. The final step in the chain is the NettyRoutingFilter (or WebClientHttpRoutingFilter) that forwards the mutated request to the upstream service and writes the response back to the client.

The first distinct idempotency surface introduced by the filter layer is filter re-execution on gateway retry. RetryGatewayFilterFactory, when configured with retries: N and a set of retryable HTTP status codes or exceptions, does not replay a cached request. It re-assembles and re-executes the Reactor filter chain from the top, with a fresh ServerWebExchange that carries forward the original client request attributes. Any filter that derives a value via a computation — UUID.randomUUID(), a SpEL expression, a factory-instantiated Supplier— re-runs that computation on each attempt. This is the mechanic behind the failure mode covered in the earlier Spring Cloud Gateway post (a GlobalFilter calling UUID.randomUUID() in its filter() method body). The same mechanic applies to factory-configured filters like AddRequestHeaderGatewayFilterFactory: the filter itself is stateless, and its configuration value (the SpEL expression string) is re-evaluated each time the filter runs.

The second distinct surface is the circuit breaker’s fallback as a second execution path that the developer treats as a compensation handler. When CircuitBreakerGatewayFilterFactory is configured with a fallbackUri, the fallback is invoked in two cases: when the circuit is open (prior failures put it in OPEN state), and when the current attempt times out or throws a retryable exception. The critical point that operators miss is that “times out” means the gateway’s response timeout — the time until the upstream service returns an HTTP response — not the time until the upstream service begins processing. If the upstream billing service receives the request, calls Stripe (committing ch_A), and then performs a slow database write or generates a large response body that takes longer than the configured timeout to stream back, the gateway will trip the circuit and invoke the fallback URI even though Stripe has already settled a charge. The fallback route’s handler sees a fresh request with no indication that the primary already completed the Stripe leg of the work.

The third distinct surface is WebClient’s retryWhen spec as a per-subscription re-execution. Reactor’s retry operators work by re-subscribing to the upstream Publisher. When WebClient’s .retryWhen(retrySpec) fires on a WebClientRequestException, it re-subscribes to the chain from the nearest retryable boundary. If the chain includes a .flatMap() that itself creates a Mono via Mono.fromCallable(() -> UUID.randomUUID().toString()), that Callable executes once per subscription. The retry creates a new subscription, the Callable executes again, and a new UUID is produced. This is not a bug in WebClient or Reactor — it is the correct behavior of a reactive assembly where every operator function is called per subscription. The bug is in the developer’s mental model: treating the Mono.fromCallable as if it is a Mono.just(computedValue) (a single emission replayed on every subscription) when it is actually a Mono that defers computation to subscription time and executes the Callable on each new subscription.

All three modes share a structural pattern: the gateway layer sits between the client and the upstream billing service, and when the gateway intervenes (via retry, circuit breaker, or WebClient retry), it re-runs code paths that were designed for single-execution semantics. The billing service trusts the idempotency key it receives from the gateway. If the gateway supplies a different key on the retried or fallback request, the billing service has no way to know that a prior request with a different key already committed a charge.

Mode 1: AddRequestHeaderGatewayFilterFactory SpEL expression evaluates UUID.randomUUID() per RetryGatewayFilter attempt — UUID_B injected into retried request — upstream billing service commits ch_B

Setup

The team is running a microservices billing platform. An API gateway routes all external traffic. The gateway is built with Spring Cloud Gateway and serves as the entry point for the mobile and web clients. Among its routes is a billing route that forwards POST /api/billing/charge to an upstream billing-service that calls Stripe to process subscription payments.

The gateway has no stateful session — it is purely reactive and stateless per route. The developer decides to inject an Idempotency-Key request header at the gateway layer so that all downstream billing service instances receive a consistent key regardless of the client. The developer configures this using the built-in AddRequestHeader filter and reaches for a SpEL expression to generate a UUID dynamically:

# application.yml — gateway route configuration
spring:
  cloud:
    gateway:
      routes:
        - id: billing-charge-route
          uri: lb://billing-service
          predicates:
            - Path=/api/billing/charge
            - Method=POST
          filters:
            # AddRequestHeader with SpEL expression.
            # BUG: T(java.util.UUID).randomUUID().toString() is a SpEL expression
            # that is evaluated every time the AddRequestHeaderGatewayFilterFactory
            # runs its filter() method — which happens on EVERY RetryGatewayFilter
            # retry attempt, not just on the first client request.
            - AddRequestHeader=Idempotency-Key, #{T(java.util.UUID).randomUUID().toString()}
            - name: Retry
              args:
                retries: 3
                statuses: SERVICE_UNAVAILABLE, GATEWAY_TIMEOUT
                methods: POST
                backoff:
                  firstBackoff: 500ms
                  maxBackoff: 2000ms
                  factor: 2

The developer knows that Spring Cloud Gateway supports SpEL in certain filter configurations. The #{ } syntax signals that the value should be evaluated as a SpEL expression rather than taken as a literal string. The intent is clear: generate a random UUID header value rather than hard-coding a static string. The misconception is about when the expression is evaluated: the developer believes the UUID is computed once when the client request arrives at the gateway and the same value is forwarded on every retry attempt. In practice, the AddRequestHeaderGatewayFilterFactory evaluates the SpEL expression each time its filter() method is called — once per attempt, because RetryGatewayFilterFactory re-runs the full filter chain including AddRequestHeaderGatewayFilterFactory on each retry.

The upstream BillingService reads the Idempotency-Key header and uses it directly as the Stripe idempotency key:

// BillingService.java — trusts the Idempotency-Key header from the gateway.
// Developer assumes the gateway sets this once and the same key is forwarded
// on any gateway-layer retry. In reality, a new UUID is injected per retry attempt.

@RestController
@RequestMapping("/api/billing")
public class BillingController {

    private final BillingService billingService;

    @PostMapping("/charge")
    public Mono<ResponseEntity<ChargeResponse>> charge(
            @RequestBody ChargeRequest request,
            @RequestHeader("Idempotency-Key") String idempotencyKey) {

        return billingService.chargeSubscription(
                request.getSubscriptionId(),
                request.getBillingPeriod(),
                request.getAmountCents(),
                request.getStripeCustomerId(),
                idempotencyKey)
            .map(result -> ResponseEntity.ok(new ChargeResponse(
                result.chargeId(), result.status(), idempotencyKey)));
    }
}

@Service
public class BillingService {

    public Mono<BillingResult> chargeSubscription(
            String subscriptionId,
            String billingPeriod,
            long amountCents,
            String stripeCustomerId,
            String idempotencyKey) {

        return Mono.fromCallable(() -> {
            ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(amountCents)
                .setCurrency("usd")
                .setCustomer(stripeCustomerId)
                .setDescription("Sub " + subscriptionId + " " + billingPeriod)
                .build();

            RequestOptions options = RequestOptions.builder()
                // BUG (upstream): trusts the key from the header.
                // UUID_A on attempt 1, UUID_B on attempt 2 — both come from the gateway.
                .setIdempotencyKey(idempotencyKey)
                .build();

            Charge charge = Charge.create(params, options);
            return new BillingResult(charge.getId(), charge.getStatus());
        }).subscribeOn(Schedulers.boundedElastic());
    }
}

The failure sequence

A client sends POST /api/billing/charge with subscription sub_001, billing period 2026-10, amount 8900 cents. The request enters the gateway and is matched by the billing-charge-route. The filter chain assembles: first the AddRequestHeaderGatewayFilterFactory for Idempotency-Key, then the RetryGatewayFilterFactory (which wraps the remaining chain), then the routing filter.

On attempt 1: AddRequestHeaderGatewayFilterFactory.filter() runs. The SpEL expression #{T(java.util.UUID).randomUUID().toString()} is evaluated. The expression calls java.util.UUID.randomUUID().toString(), producing UUID_A = "3f7a2b1c-8d4e-5f6a-9b0c-1d2e3f4a5b6c". The filter adds the header Idempotency-Key: 3f7a2b1c-8d4e-5f6a-9b0c-1d2e3f4a5b6c to the outgoing request. The routing filter forwards the request to a billing-service instance. The billing service reads the header, uses it as the Stripe idempotency key, and calls Stripe: POST /v1/charges with Idempotency-Key: 3f7a2b1c-8d4e-5f6a-9b0c-1d2e3f4a5b6c. Stripe commits ch_A = "ch_3Qa2B..." and returns 200. The billing service begins assembling the HTTP response.

At this point, a GC pause on the billing service pod delays the response serialization by 2.8 seconds. The billing service is configured to serve the response within this window, but the gateway’s load balancer health check had previously recorded this pod as borderline-slow, and the pod returns an HTTP 503 (Service Unavailable) response after a connection pool queue timeout on its own downstream database write for the BillingRecord. The 503 matches the statuses: SERVICE_UNAVAILABLE retry configuration. RetryGatewayFilterFactory schedules attempt 2.

On attempt 2: RetryGatewayFilterFactory re-executes the filter chain. The chain runs from the top. AddRequestHeaderGatewayFilterFactory.filter() runs again. The SpEL expression is evaluated again. java.util.UUID.randomUUID() produces a different UUID on this call (UUID generation is not seeded to any request-specific value): UUID_B = "9a0b1c2d-3e4f-5a6b-7c8d-9e0f1a2b3c4d". The filter adds Idempotency-Key: 9a0b1c2d-3e4f-5a6b-7c8d-9e0f1a2b3c4d to the retried request. The routing filter forwards the retried request to the same (or a different) billing service instance. The billing service reads the new header value, uses it as the Stripe idempotency key, and calls Stripe: POST /v1/charges with Idempotency-Key: 9a0b1c2d-3e4f-5a6b-7c8d-9e0f1a2b3c4d. Stripe has no record of this key. Stripe processes the request as a new charge and commits ch_B = "ch_3Rb3C...". The customer is charged twice: once for ch_A (committed on attempt 1) and once for ch_B (committed on attempt 2).

Developer misconception: how SpEL expressions are evaluated in gateway filter configuration

The developer’s mental model was: “I configured the AddRequestHeader filter with a SpEL expression. Spring evaluates the expression when the route is initialized (or when the first request comes in) and uses the same computed value for every request on that route. My UUID is generated once per client request and is constant across gateway retries for that request.”

The actual behavior is: the AddRequestHeaderGatewayFilterFactory stores the SpEL expression as a String in its configuration object. When filter() is called, the factory evaluates the expression against the current ServerWebExchange context. No caching occurs between invocations of filter(). Each call to filter() is a fresh evaluation. RetryGatewayFilterFactory calls the downstream filter chain (including AddRequestHeaderGatewayFilterFactory) once per attempt, not once per client connection. The expression T(java.util.UUID).randomUUID().toString() calls java.util.UUID.randomUUID() on each evaluation, which is an independent call to SecureRandom each time. There is no caching, no memoization, and no per-request state that would cause the second evaluation to return the first UUID.

The secondary misconception is about the scope of “once per client request.” A client request, at the gateway layer, may correspond to multiple upstream HTTP requests when retries are configured. The gateway’s responsibility is to return one response to the client. The upstream’s responsibility is to handle each received request safely. The filter chain is a per-upstream-attempt concept, not a per-client-request concept. Any state that must be stable across retry attempts must be stored in the ServerWebExchange attributes (which survive retry) before the retry filter wraps the rest of the chain.

The fix

The idempotency key must be computed once and stored in the ServerWebExchange before the retry filter wraps the downstream chain. There are two viable approaches:

Option A: compute a stable key from the request body at a GlobalFilter or high-precedence GatewayFilter that runs before the RetryGatewayFilterFactory and stores it as an exchange attribute. The key is computed once. A second filter (lower precedence, inside the retry wrapper) reads the attribute and adds it as the header.

// FIXED: two-filter approach.
// Filter 1 (order -1, runs before RetryGatewayFilterFactory at order 1):
// reads request body, computes stable key, stores in exchange attribute.
// Filter 2 (order 2, runs inside the retry wrapper):
// reads exchange attribute, adds Idempotency-Key header.

// Filter 1: IdempotencyKeyComputeFilter — runs BEFORE RetryGatewayFilterFactory
@Component
@Order(-1)
public class IdempotencyKeyComputeFilter implements GlobalFilter {

    private static final String IDEMPOTENCY_KEY_ATTR = "billing.idempotencyKey";

    @Override
    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        ServerHttpRequest request = exchange.getRequest();

        // Only apply to billing charge requests
        if (!request.getPath().value().startsWith("/api/billing/charge")
                || !HttpMethod.POST.equals(request.getMethod())) {
            return chain.filter(exchange);
        }

        // Check if key is already set (prevents re-computation on RetryGatewayFilter
        // retry IF this filter somehow re-runs — defensive guard).
        if (exchange.getAttributes().containsKey(IDEMPOTENCY_KEY_ATTR)) {
            return chain.filter(exchange);
        }

        // Read body once to extract subscriptionId + billingPeriod + amountCents.
        // Uses ServerRequest to cache the body for downstream reading.
        return DataBufferUtils.join(exchange.getRequest().getBody())
            .flatMap(dataBuffer -> {
                byte[] bodyBytes = new byte[dataBuffer.readableByteCount()];
                dataBuffer.read(bodyBytes);
                DataBufferUtils.release(dataBuffer);

                String body = new String(bodyBytes, StandardCharsets.UTF_8);
                // Parse the JSON body to extract stable fields.
                // In production use ObjectMapper; here simplified for clarity.
                String key = computeStableKey(body);
                exchange.getAttributes().put(IDEMPOTENCY_KEY_ATTR, key);

                // Rebuild the request with the cached body so downstream can read it.
                ServerHttpRequest mutatedRequest = new ServerHttpRequestDecorator(request) {
                    @Override
                    public Flux<DataBuffer> getBody() {
                        return Flux.just(exchange.getResponse().bufferFactory()
                            .wrap(bodyBytes));
                    }
                };

                return chain.filter(exchange.mutate().request(mutatedRequest).build());
            });
    }

    static String computeStableKey(String jsonBody) {
        // Extract subscriptionId, billingPeriod, amountCents from JSON.
        // Simplified: in production use ObjectMapper or JsonPath.
        try {
            // Parse key fields from request body for a stable, content-derived key.
            // SHA-256 of canonical billing intent fields.
            MessageDigest digest = MessageDigest.getInstance("SHA-256");
            byte[] hash = digest.digest(jsonBody.getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash).substring(0, 32);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

// Filter 2: configured in route as SetRequestHeader that reads from exchange attribute.
// Use a custom GatewayFilterFactory to inject the pre-computed attribute as a header.

@Component
public class IdempotencyKeyHeaderGatewayFilterFactory
        extends AbstractGatewayFilterFactory<Object> {

    private static final String IDEMPOTENCY_KEY_ATTR = "billing.idempotencyKey";

    @Override
    public GatewayFilter apply(Object config) {
        return (exchange, chain) -> {
            String key = (String) exchange.getAttributes().get(IDEMPOTENCY_KEY_ATTR);
            if (key == null) {
                // Fallback: generate once and store so subsequent retry calls
                // within THIS filter's invocations are consistent.
                // Note: this should not happen if Filter 1 ran correctly.
                key = computeFallbackKey(exchange);
                exchange.getAttributes().put(IDEMPOTENCY_KEY_ATTR, key);
            }
            ServerHttpRequest mutated = exchange.getRequest().mutate()
                .header("Idempotency-Key", key)
                .build();
            return chain.filter(exchange.mutate().request(mutated).build());
        };
    }

    private String computeFallbackKey(ServerWebExchange exchange) {
        try {
            String requestId = exchange.getRequest().getId();
            MessageDigest digest = MessageDigest.getInstance("SHA-256");
            byte[] hash = digest.digest(requestId.getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash).substring(0, 32);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

# FIXED application.yml — IdempotencyKeyHeaderGatewayFilterFactory injects
# the pre-computed stable key from the exchange attribute.
# AddRequestHeader with SpEL removed — no longer needed.

spring:
  cloud:
    gateway:
      routes:
        - id: billing-charge-route
          uri: lb://billing-service
          predicates:
            - Path=/api/billing/charge
            - Method=POST
          filters:
            # FIXED: read pre-computed stable key from exchange attribute.
            # The IdempotencyKeyComputeFilter (GlobalFilter, order -1) already
            # stored the key in the attribute before the Retry filter wraps
            # the rest of the chain. This filter reads the attribute — same
            # value on every RetryGatewayFilter retry attempt.
            - IdempotencyKeyHeader
            - name: Retry
              args:
                retries: 3
                statuses: SERVICE_UNAVAILABLE, GATEWAY_TIMEOUT
                methods: POST
                backoff:
                  firstBackoff: 500ms
                  maxBackoff: 2000ms
                  factor: 2

Option B: have the client pre-compute and include the idempotency key in the request body or as a request header. The gateway reads the client-provided key from the incoming request and forwards it as-is using a simple PreserveHostHeader or custom filter. Since the client provides the key before the request reaches the gateway, and the key does not change between the original request and any gateway retry, the upstream always receives the same key.

# Option B: client provides Idempotency-Key header; gateway passes it through.
# No SpEL expression needed. The gateway can enforce presence of the header
# with a RequestHeader predicate or a header-validation GatewayFilter.

spring:
  cloud:
    gateway:
      routes:
        - id: billing-charge-route
          uri: lb://billing-service
          predicates:
            - Path=/api/billing/charge
            - Method=POST
            # Optional: reject requests that don't include the header
            - Header=Idempotency-Key, [a-f0-9]{32,64}
          filters:
            # No key injection needed — key already present in client request.
            # Retry forwards the same original headers including Idempotency-Key.
            - name: Retry
              args:
                retries: 3
                statuses: SERVICE_UNAVAILABLE, GATEWAY_TIMEOUT
                methods: POST

Mode 2: CircuitBreakerGatewayFilterFactory fallback route calls Stripe with UUID_B after primary route committed ch_A before response timeout tripped the circuit — ch_B

Setup

The operations team has experienced outages where the billing-service became unresponsive under load and the gateway queued requests indefinitely. To improve resilience, the developer adds a CircuitBreaker GatewayFilter to the billing route with a fallback controller that the developer describes as “the retry handler for when billing-service is down.” The fallback controller calls Stripe directly from the gateway process to ensure the charge goes through even when the primary service is unavailable.

# application.yml — gateway route with CircuitBreaker and fallback

spring:
  cloud:
    gateway:
      routes:
        - id: billing-charge-route
          uri: lb://billing-service
          predicates:
            - Path=/api/billing/charge
            - Method=POST
          filters:
            - name: CircuitBreaker
              args:
                name: billing-circuit-breaker
                fallbackUri: forward:/billing/fallback
                # statusCodes lists HTTP status codes that trigger the circuit breaker
                statusCodes:
                  - 503
                  - 504

resilience4j:
  timelimiter:
    instances:
      billing-circuit-breaker:
        # BUG: timeout is the total time from gateway forwarding to response receipt.
        # A billing-service that calls Stripe (commits ch_A, 800ms) and then
        # performs a 1.5s database write will exceed this 2000ms limit.
        # The circuit breaker will trip to the fallback AFTER Stripe has settled ch_A.
        timeoutDuration: 2000ms
  circuitbreaker:
    instances:
      billing-circuit-breaker:
        registerHealthIndicator: true
        slidingWindowSize: 10
        permittedNumberOfCallsInHalfOpenState: 3
        slidingWindowType: COUNT_BASED
        minimumNumberOfCalls: 5
        waitDurationInOpenState: 5s
        failureRateThreshold: 50

The fallback controller in the gateway application:

// BUGGY FallbackBillingController — called when the CircuitBreaker trips.
// Developer assumes: "circuit tripped = primary service failed before calling Stripe".
// Reality: "circuit tripped = primary service exceeded timeoutDuration" which may be
// AFTER Stripe committed ch_A.

@RestController
public class FallbackBillingController {

    private final StripeClient stripeClient;

    @PostMapping("/billing/fallback")
    public Mono<ResponseEntity<ChargeResponse>> fallbackCharge(
            ServerWebExchange exchange) {

        // Read the original request body forwarded by the gateway via `forward:` URI.
        // The body contains subscriptionId, billingPeriod, amountCents, customerId.
        return ServerRequest.create(exchange, HandlerStrategies.withDefaults().messageReaders())
            .bodyToMono(ChargeRequest.class)
            .flatMap(request -> Mono.fromCallable(() -> {

                // BUG: UUID.randomUUID() — new key that Stripe has never seen.
                // If the primary billing-service already called Stripe with UUID_A
                // (committing ch_A) and then timed out at the database write,
                // this fallback generates UUID_B and commits ch_B.
                String fallbackKey = UUID.randomUUID().toString();

                ChargeCreateParams params = ChargeCreateParams.builder()
                    .setAmount(request.amountCents())
                    .setCurrency("usd")
                    .setCustomer(request.stripeCustomerId())
                    .setDescription("Fallback Sub " + request.subscriptionId()
                        + " " + request.billingPeriod())
                    .build();

                RequestOptions options = RequestOptions.builder()
                    .setIdempotencyKey(fallbackKey)
                    .build();

                Charge charge = Charge.create(params, options);
                return new ChargeResponse(charge.getId(), charge.getStatus(), fallbackKey);
            }).subscribeOn(Schedulers.boundedElastic()))
            .map(ResponseEntity::ok);
    }
}

The failure sequence

A client sends POST /api/billing/charge for subscription sub_002, billing period 2026-10, amount 14900 cents. The gateway routes the request through the CircuitBreakerGatewayFilterFactory, which wraps the NettyRoutingFilter with a Resilience4j time limiter. The time limiter starts a countdown from the moment the gateway forwards the request upstream.

The billing-service pod receives the request at T=0ms. At T=200ms the billing service reads the request body and generates UUID_A = "b1c2d3e4-f5a6-7b8c-9d0e-1f2a3b4c5d6e". At T=800ms the billing service calls Stripe: POST /v1/charges with Idempotency-Key: b1c2d3e4-f5a6-7b8c-9d0e-1f2a3b4c5d6e. Stripe processes the charge and commits ch_A = "ch_3Rc4D...". At T=900ms Stripe returns 200 to the billing service. The billing service begins writing the BillingRecord to its PostgreSQL database.

The PostgreSQL write is slow: the database is under heavy load from the end-of-month billing sweep. The write takes 1200ms. At T=2100ms the total elapsed time exceeds the Resilience4j timeoutDuration: 2000ms. The time limiter fires a TimeLimiterTimeoutException. The Resilience4j circuit breaker increments its failure counter. The CircuitBreakerGatewayFilterFactory catches the exception and routes the exchange to the fallback URI: forward:/billing/fallback.

The gateway’s FallbackBillingController at /billing/fallback receives the forwarded exchange. The fallback controller reads the original request body (sub_002, 2026-10, 14900 cents). It generates UUID_B = "0e1f2a3b-4c5d-6e7f-8a9b-0c1d2e3f4a5b". The fallback calls Stripe: POST /v1/charges with Idempotency-Key: 0e1f2a3b-4c5d-6e7f-8a9b-0c1d2e3f4a5b. Stripe has no record of UUID_B. Stripe processes the request as a new, independent charge and commits ch_B = "ch_3Sd5E...". At T=2350ms the billing service (which was still running despite the gateway’s timeout) completes its PostgreSQL write, writes a BillingRecord for ch_A, and returns an HTTP 200 response that the gateway has already discarded.

The customer has now been charged twice: ch_A ($149.00) committed at T=900ms by the primary service, and ch_B ($149.00) committed at T=2350ms by the fallback controller. Both charges appear in the Stripe dashboard and on the customer’s credit card statement. The BillingRecord table has one record (for ch_A, written by the primary service). The fallback has no BillingRecord for ch_B.

Developer misconception: what a circuit breaker timeout means

The developer’s mental model was: “The circuit breaker trips when the billing service is down or unresponsive — when it can’t even receive the request, let alone process it. If the circuit trips, Stripe was never called by the primary service. The fallback is safe to call Stripe independently.”

The actual behavior is: the Resilience4j time limiter’s timeoutDuration measures wall-clock time from when the gateway forwards the request to when the gateway receives the complete HTTP response. The upstream service may perform substantial work — including calling Stripe and committing a charge — within that time window, and then exceed the timeout on a subsequent step (a slow DB write, a large response serialization, a secondary API call). The circuit breaker cannot distinguish “service failed before calling Stripe” from “service called Stripe successfully but timed out afterwards.” The circuit breaker observes only the elapsed time. A timeout is not evidence of pre-Stripe failure.

The secondary misconception is about the role of the fallback. CircuitBreakerGatewayFilterFactory’s fallbackUri is a degraded-mode response path, not a retry-with-different-path path. The framework’s documentation describes it as “what to return when the circuit is open or the call timed out.” In many cases (e.g., read-only routes, cached data, non-mutating operations) a fallback response is safe. For mutating, money-moving operations, the fallback must not independently re-execute the primary mutation without first checking whether the primary already completed it.

The fix

The fallback controller must not independently call Stripe with a new idempotency key. The fallback has two safe behaviors:

Option A (recommended): return an error response and let the client retry with the same idempotency key. The client was responsible for generating the idempotency key before submitting the request. If the gateway times out, the client retries the same request (with the same client-side idempotency key). The billing service, on the retry, calls Stripe with the same key and receives ch_A back (Stripe deduplicates). The double-charge risk moves from the gateway layer to the client-retry layer, where the client controls the key stability.

// FIXED FallbackBillingController — Option A: return error, let client retry.
// The client holds the idempotency key and can safely retry the full request.
// Do NOT call Stripe in the fallback.

@RestController
public class FallbackBillingController {

    @PostMapping("/billing/fallback")
    public ResponseEntity<ErrorResponse> fallbackCharge(ServerWebExchange exchange) {
        // Return a retryable error code. Do not call Stripe.
        // The gateway timed out — the primary service's outcome is unknown.
        // The client should retry with the same idempotency key after a backoff.
        String requestId = exchange.getRequest().getHeaders()
            .getFirst(HttpHeaders.CORRELATION_ID);  // or X-Request-Id
        return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
            .body(new ErrorResponse(
                "billing_service_timeout",
                "Billing service did not respond in time. Retry with the same " +
                "Idempotency-Key header. Request-Id: " + requestId,
                true  // retryable=true
            ));
    }
}

Option B: the fallback queries the BillingRecord table before calling Stripe. The billing service (primary and fallback both have access to the same BillingRecord database) can check whether a charge was already recorded for the stable key derived from the request. If a BillingRecord exists, ch_A was committed — return the charge ID. If no record exists, the primary service may have failed before calling Stripe, and the fallback can call Stripe with the same stable key derived from the request body fields.

// FIXED FallbackBillingController — Option B: check BillingRecord first.
// If ch_A was committed (BillingRecord exists), return ch_A — no Stripe call.
// If no record, derive a stable key from request body fields and call Stripe.

@RestController
public class FallbackBillingController {

    private final BillingRecordRepository billingRecordRepo;

    @PostMapping("/billing/fallback")
    public Mono<ResponseEntity<ChargeResponse>> fallbackCharge(
            ServerWebExchange exchange) {

        return ServerRequest.create(exchange, HandlerStrategies.withDefaults().messageReaders())
            .bodyToMono(ChargeRequest.class)
            .flatMap(request -> {
                // Stable key from request body fields — same as what the primary
                // service would have computed. If the primary ran with this key,
                // BillingRecord will have an entry.
                String stableKey = computeStableKey(
                    request.subscriptionId(),
                    request.billingPeriod(),
                    request.amountCents()
                );

                return Mono.fromCallable(() ->
                    billingRecordRepo.findByIdempotencyKey(stableKey)
                ).subscribeOn(Schedulers.boundedElastic())
                .flatMap(existing -> {
                    if (existing.isPresent()) {
                        // ch_A was committed by the primary. Return it — no Stripe call.
                        BillingRecord record = existing.get();
                        return Mono.just(ResponseEntity.ok(
                            new ChargeResponse(record.stripeChargeId(),
                                               "succeeded",
                                               stableKey)));
                    }

                    // No BillingRecord — primary likely failed before calling Stripe.
                    // Safe to call Stripe with the stable key.
                    return Mono.fromCallable(() -> {
                        ChargeCreateParams params = ChargeCreateParams.builder()
                            .setAmount(request.amountCents())
                            .setCurrency("usd")
                            .setCustomer(request.stripeCustomerId())
                            .build();

                        RequestOptions options = RequestOptions.builder()
                            .setIdempotencyKey(stableKey)
                            .build();

                        Charge charge = Charge.create(params, options);
                        // Record the charge — prevents a second fallback invocation
                        // from calling Stripe again if the circuit remains open.
                        billingRecordRepo.save(new BillingRecord(
                            request.subscriptionId(),
                            request.billingPeriod(),
                            stableKey,
                            charge.getId(),
                            request.amountCents()
                        ));
                        return ResponseEntity.ok(
                            new ChargeResponse(charge.getId(), "succeeded", stableKey));
                    }).subscribeOn(Schedulers.boundedElastic());
                });
            });
    }

    private static String computeStableKey(String subscriptionId,
                                            String billingPeriod,
                                            long amountCents) {
        try {
            MessageDigest digest = MessageDigest.getInstance("SHA-256");
            String input = subscriptionId + "|" + billingPeriod + "|" + amountCents;
            byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

Mode 3: WebClient retryWhen inside a custom GatewayFilter re-subscribes the reactive pipeline — Mono.fromCallable(() -> UUID.randomUUID()) in flatMap generates UUID_B per retry subscription

Setup

The team has special requirements for the billing route: they need to add custom tracing headers, transform the request body to include a tenant identifier, and forward to the billing service via a shared WebClient bean that is configured with connection pooling and timeout settings. Rather than using the built-in Retry GatewayFilter, a developer writes a custom GatewayFilter that manages retries at the WebClient level using Reactor’s retryWhen operator.

// BillingProxyGatewayFilter.java — BUGGY custom GatewayFilter
// Uses WebClient with retryWhen to forward billing requests to billing-service.
// BUG: UUID.randomUUID() is called inside a Mono.fromCallable() that is chained
// via flatMap. WebClient retryWhen re-subscribes the source Publisher, which
// causes the flatMap to execute again, evaluating UUID.randomUUID() a second time
// and generating UUID_B.

@Component
public class BillingProxyGatewayFilter implements GatewayFilter, Ordered {

    private final WebClient billingWebClient;

    public BillingProxyGatewayFilter(WebClient billingWebClient) {
        this.billingWebClient = billingWebClient;
    }

    @Override
    public int getOrder() { return 10; }

    @Override
    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        ServerHttpRequest request = exchange.getRequest();

        if (!request.getPath().value().startsWith("/api/billing/charge")) {
            return chain.filter(exchange);
        }

        return DataBufferUtils.join(request.getBody())
            .flatMap(dataBuffer -> {
                byte[] bodyBytes = new byte[dataBuffer.readableByteCount()];
                dataBuffer.read(bodyBytes);
                DataBufferUtils.release(dataBuffer);
                String originalBody = new String(bodyBytes, StandardCharsets.UTF_8);

                // BUG: Mono.fromCallable generates the idempotency key here.
                // This Mono is composed into a flatMap chain. WebClient's retryWhen
                // re-subscribes the source, which re-executes this flatMap and
                // re-invokes the Callable — generating UUID_B on the first retry.
                return Mono.fromCallable(() -> UUID.randomUUID().toString())
                    .flatMap(idempotencyKey -> {
                        String enrichedBody = addIdempotencyKey(originalBody, idempotencyKey);

                        return billingWebClient.post()
                            .uri("/internal/billing/charge")
                            .header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
                            .header("X-Idempotency-Key", idempotencyKey)
                            .bodyValue(enrichedBody)
                            .exchangeToMono(clientResponse -> {
                                exchange.getResponse().setStatusCode(
                                    clientResponse.statusCode());
                                clientResponse.headers().asHttpHeaders().forEach(
                                    (name, values) ->
                                        exchange.getResponse().getHeaders()
                                            .put(name, values));
                                return exchange.getResponse().writeWith(
                                    clientResponse.bodyToFlux(DataBuffer.class));
                            })
                            // retryWhen re-subscribes the source Mono on WebClientRequestException.
                            // Re-subscribing means re-executing the entire reactive chain above,
                            // including the Mono.fromCallable(() -> UUID.randomUUID()) call.
                            // UUID_B is generated on the first retry. The billing service
                            // calls Stripe with UUID_B, committing ch_B if ch_A was already
                            // committed before the connection was lost.
                            .retryWhen(Retry.backoff(3, Duration.ofSeconds(1))
                                .filter(e -> e instanceof WebClientRequestException));
                    });
            });
    }

    private String addIdempotencyKey(String body, String key) {
        // Add the idempotency key as a field in the JSON body.
        // Simplified: in production use ObjectMapper.
        return body.replaceFirst("\\{", "{\"idempotencyKey\":\"" + key + "\",");
    }
}

The failure sequence

A client sends POST /api/billing/charge. The BillingProxyGatewayFilter.filter() runs. The body bytes are read from the buffer and stored in originalBody. A Mono.fromCallable(() -> UUID.randomUUID().toString()) is constructed. This Mono, when subscribed, calls the Callable and emits the generated UUID. The WebClient subscription chain begins.

The Reactor scheduler subscribes to the Mono.fromCallable. The Callable executes: UUID_A = "c2d3e4f5-a6b7-c8d9-e0f1-a2b3c4d5e6f7" is generated. The flatMap lambda receives UUID_A and constructs the enriched request body and the WebClient request with X-Idempotency-Key: c2d3e4f5-a6b7-c8d9-e0f1-a2b3c4d5e6f7. The WebClient sends the request to billing-service at http://billing-service/internal/billing/charge.

The billing service at the downstream receives the request. It reads the X-Idempotency-Key header and uses it as the Stripe idempotency key. The billing service calls Stripe: POST /v1/charges with Idempotency-Key: c2d3e4f5-a6b7-c8d9-e0f1-a2b3c4d5e6f7. Stripe commits ch_A = "ch_3Te5F..." and returns 200. The billing service begins sending the HTTP 200 response back to the gateway’s WebClient.

Between the response status line being sent and the response body being fully transmitted, the TCP connection between the billing service pod and the gateway pod is reset by a Kubernetes pod readiness probe timeout that triggers a network policy refresh. The WebClient’s Netty channel emits a WebClientRequestException wrapping an IOException: Connection reset by peer. The .retryWhen(Retry.backoff(3, ...).filter(e -> e instanceof WebClientRequestException)) spec matches: WebClientRequestException is an instance of WebClientRequestException. The retry spec allows this exception to trigger a retry.

The retry fires. Reactor’s retry operator re-subscribes to the source Publisher. The “source” of the .retryWhen() operator is the Mono produced by the .flatMap(idempotencyKey -> ...) call. Reactor re-subscribes to the entire flatMap source, which is the Mono.fromCallable(() -> UUID.randomUUID().toString()). The Callable executes again: UUID_B = "d3e4f5a6-b7c8-d9e0-f1a2-b3c4d5e6f7a8". The flatMap lambda receives UUID_B and constructs a new WebClient request with X-Idempotency-Key: d3e4f5a6-b7c8-d9e0-f1a2-b3c4d5e6f7a8. The billing service receives this second request, uses UUID_B as the Stripe key, calls Stripe again, and Stripe commits ch_B = "ch_3Uf6G...". The customer is charged twice.

Developer misconception: Reactor’s fromCallable vs just under retry

The developer’s mental model was: “I put UUID.randomUUID() inside a Mono.fromCallable so it executes lazily. The UUID is generated when the request is processed. On retry, the WebClient retries the HTTP call — it doesn’t re-run the whole chain. My UUID was already assigned to the variable idempotencyKey before the WebClient call, so the retry uses the same idempotencyKey variable value.”

The actual behavior depends on where .retryWhen() is placed in the operator chain. In the buggy code, .retryWhen() is placed on the Mono returned by the WebClient call inside the flatMap lambda. The retry operator’s source is this inner Mono (the WebClient call). However, Reactor’s flatMap is not a memoizing operator: each subscription to the flatMap’s output causes the mapping function to run. When .retryWhen() re-subscribes, it re-subscribes to the output of flatMap, which triggers flatMap to call its function again, which triggers the source (Mono.fromCallable) to be subscribed to again, which calls the Callable again.

The distinction the developer was conflating is between Mono.just(UUID.randomUUID().toString()) and Mono.fromCallable(() -> UUID.randomUUID().toString()). Mono.just(value) evaluates value eagerly at construction time and replays that fixed value on every subscription. If the developer had written Mono.just(UUID.randomUUID().toString()) before the flatMap, the UUID would have been computed once when the outer chain was subscribed, and the same value would be supplied to the flatMap function on every subscription (including retries). Mono.fromCallable(() -> UUID.randomUUID()) is the opposite: it defers computation to subscription time and re-runs the Callable on every new subscription.

A second common variant of this mistake uses Mono.defer(() -> Mono.just(UUID.randomUUID().toString())). Mono.defer is also a per-subscription factory. The factory lambda runs on every subscription, creating a new Mono.just(newUUID) each time. On retry, Mono.defer runs again, calling UUID.randomUUID() again, producing UUID_B.

The fix

The idempotency key must be computed before the retryWhen boundary and stored in a variable that is captured in the flatMap closure. If the UUID is computed and captured before the first subscription, and the flatMap closure captures the variable (not a deferred computation), then all retry subscriptions use the same captured value.

// FIXED BillingProxyGatewayFilter — stable idempotency key computed before
// the retryWhen boundary.
// Key is derived from the request body fields so it is stable across retries
// AND across client-initiated retries of the same billing intent.

@Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
    ServerHttpRequest request = exchange.getRequest();

    if (!request.getPath().value().startsWith("/api/billing/charge")) {
        return chain.filter(exchange);
    }

    return DataBufferUtils.join(request.getBody())
        .flatMap(dataBuffer -> {
            byte[] bodyBytes = new byte[dataBuffer.readableByteCount()];
            dataBuffer.read(bodyBytes);
            DataBufferUtils.release(dataBuffer);
            String originalBody = new String(bodyBytes, StandardCharsets.UTF_8);

            // FIXED: compute the stable key OUTSIDE the retryWhen boundary.
            // computeStableKey reads the request body content (subscriptionId,
            // billingPeriod, amountCents) and produces a SHA-256 key that is
            // the same for any retry of the same billing intent.
            // This computation happens once, before any WebClient subscription.
            // The 'idempotencyKey' variable is a String (immutable, value-type).
            // The flatMap closure below captures this String value, not a deferred
            // computation. All retryWhen re-subscriptions see the same String.
            final String idempotencyKey = computeStableKeyFromBody(originalBody);

            String enrichedBody = addIdempotencyKey(originalBody, idempotencyKey);

            // FIXED: retryWhen is now inside the flatMap, but the key it uses
            // is the captured 'idempotencyKey' String — not recomputed per retry.
            return billingWebClient.post()
                .uri("/internal/billing/charge")
                .header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
                .header("X-Idempotency-Key", idempotencyKey)
                .bodyValue(enrichedBody)
                .exchangeToMono(clientResponse -> {
                    exchange.getResponse().setStatusCode(clientResponse.statusCode());
                    clientResponse.headers().asHttpHeaders().forEach(
                        (name, values) ->
                            exchange.getResponse().getHeaders().put(name, values));
                    return exchange.getResponse().writeWith(
                        clientResponse.bodyToFlux(DataBuffer.class));
                })
                // retryWhen re-subscribes the WebClient Mono — NOT the
                // Mono.fromCallable that generated the key. The key is already
                // a captured String in the enclosing closure. The retried
                // WebClient request uses the same idempotencyKey on every attempt.
                .retryWhen(Retry.backoff(3, Duration.ofSeconds(1))
                    .filter(e -> e instanceof WebClientRequestException));
        });
}

static String computeStableKeyFromBody(String jsonBody) {
    try {
        // Parse subscriptionId, billingPeriod, amountCents from JSON body.
        // Using SHA-256 of the canonical billing fields for a stable, intent-derived key.
        MessageDigest digest = MessageDigest.getInstance("SHA-256");
        byte[] hash = digest.digest(jsonBody.getBytes(StandardCharsets.UTF_8));
        return HexFormat.of().formatHex(hash);
    } catch (NoSuchAlgorithmException e) {
        throw new IllegalStateException(e);
    }
}

An important detail: in the fixed version, the idempotencyKey String is declared as final in the outer flatMap scope. This is not just a style choice — it makes the compiler enforce that the closure captures a fixed String value rather than a mutable reference. The inner billingWebClient.post()...retryWhen() chain sees idempotencyKey as an effectively-final captured String. When retryWhen re-subscribes the WebClient Mono, the retry runs the billingWebClient.post()...header("X-Idempotency-Key", idempotencyKey)... code again, but idempotencyKey is the same captured String on every execution of that code. No new UUID generation occurs.

A test that surfaces the difference between the buggy and fixed implementations:

// Test: verify that the same Idempotency-Key header is sent on all WebClient
// retry attempts. Wire in WireMock as the billing-service backend.
// The WireMock stub simulates a connection reset on the first attempt and
// a 200 response on the second attempt.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@WireMockTest(httpPort = 9091)
class BillingProxyGatewayFilterRetryTest {

    @Autowired
    private WebTestClient webTestClient;

    @BeforeEach
    void setupWireMock() {
        // Scenario: STARTED state — first call returns a connection reset (fault).
        // Second call — state "second_attempt" — returns 200.
        stubFor(post(urlEqualTo("/internal/billing/charge"))
            .inScenario("gateway-retry")
            .whenScenarioStateIs(STARTED)
            .willReturn(aResponse().withFault(Fault.CONNECTION_RESET_BY_PEER))
            .willSetStateTo("second_attempt"));

        stubFor(post(urlEqualTo("/internal/billing/charge"))
            .inScenario("gateway-retry")
            .whenScenarioStateIs("second_attempt")
            .willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"chargeId\":\"ch_test_XyZ\",\"status\":\"succeeded\"}")));
    }

    @Test
    void retryMustSendSameIdempotencyKeyOnBothAttempts() {
        webTestClient.post()
            .uri("/api/billing/charge")
            .contentType(MediaType.APPLICATION_JSON)
            .bodyValue("{\"subscriptionId\":\"sub_003\"," +
                       "\"billingPeriod\":\"2026-10\"," +
                       "\"amountCents\":4900," +
                       "\"stripeCustomerId\":\"cus_DEF456\"}")
            .exchange()
            .expectStatus().isOk();

        // Assert: exactly 2 calls were made to billing-service (attempt 1 reset,
        // attempt 2 succeeded). Both calls must carry the SAME Idempotency-Key header.
        List<LoggedRequest> requests = findAll(
            postRequestedFor(urlEqualTo("/internal/billing/charge")));
        assertThat(requests).hasSize(2);

        String keyAttempt1 = requests.get(0).getHeader("X-Idempotency-Key");
        String keyAttempt2 = requests.get(1).getHeader("X-Idempotency-Key");

        // This assertion FAILS on the buggy implementation (UUID_A != UUID_B)
        // and PASSES on the fixed implementation (stable key same on both attempts).
        assertThat(keyAttempt1)
            .isNotBlank()
            .isEqualTo(keyAttempt2);
    }
}

Diagnostic checklist

Summary

The three failure modes documented here share a structural pattern: the gateway acts as an active participant in the request lifecycle — injecting headers, managing circuits, and retrying calls — and each of these interventions is an opportunity for a new idempotency key to be generated if key derivation is placed in the wrong part of the reactive assembly. AddRequestHeaderGatewayFilterFactory with SpEL re-evaluates the expression per filter invocation, not per client connection. CircuitBreakerGatewayFilterFactory’s fallback fires on timeout, not on pre-work failure; a billing service that commits ch_A before exceeding the timeout leaves the circuit breaker with no signal that Stripe was already called. WebClient’s retryWhen re-subscribes the source Publisher; any deferred computation (fromCallable, defer) in the subscription chain re-executes per subscription, including the UUID generation that should have been fixed before the first attempt.

The consistent defense across all three modes is the same: the billing service should derive its Stripe idempotency key from the billing intent’s content (SHA-256(subscriptionId + billingPeriod + amountCents)), not from a gateway-injected header that may change between the original request and any retry or fallback invocation. A content-derived key is stable by construction: the same billing intent always produces the same key, regardless of how many times the gateway retries, whether the circuit trips, or whether the reactive chain re-subscribes. The gateway-injected header is a convenience for cases where the billing service cannot derive a stable key from the request body alone. When the gateway provides the key, the gateway bears the responsibility for ensuring key stability across all retry paths — which requires careful attention to filter ordering, SpEL evaluation scope, circuit breaker fallback design, and reactive operator subscription semantics.

Keybrake’s proxy layer provides an additional defense at the infrastructure level: every request that arrives at the proxy is fingerprinted by its content (vendor, endpoint, amount, customer identifier), and if two requests with different idempotency keys arrive for the same content fingerprint within the deduplication window, the proxy holds the second request and returns the first request’s response. Gateway retry re-evaluations, circuit breaker fallbacks generating fresh UUIDs, and WebClient retry re-subscriptions all arrive at the proxy as requests with different keys but the same content fingerprint. Keybrake’s audit log records both keys and the fingerprint match, so incident response teams can reconstruct exactly which gateway-layer mechanism caused the key rotation even before examining gateway logs.