Spring Cloud Config and Stripe Integration: How @RefreshScope Bean Destruction Between the Stripe Call and the Retry Clears the In-Flight Idempotency Key Cache After ch_A Is Committed, a @Value-Bound billing.idempotency.prefix Changed by ContextRefresher.refresh() Produces a Different Key on the Recovery Path Causing Stripe to Commit ch_B, and {cipher}-Encrypted HMAC Secret Decryption Failure After Keystore Rotation Returns the Literal Sentinel Value as the Signing Key Deriving UUID_B on Every Subsequent Attempt

This post is part of the ongoing series on Stripe idempotency failure modes in JVM frameworks. Earlier entries covered Spring Data JPA (JDBC transaction boundary and Stripe commit ordering), Spring Data Elasticsearch (@Retryable UUID reset after VersionConflictEngineException, NRT search gap, auto-generated @Id timeout), Spring Cloud Circuit Breaker (Resilience4j TimeLimiter post-commit fallback, CircuitBreakerOpenException open-state recovery), and Spring Cloud OpenFeign (RequestInterceptor on every retry, retryOnAllOperations re-sends POST to new instance, FallbackFactory ignores idempotencyKey parameter). This post moves to Spring Cloud Config — the centralized external configuration service for Spring Boot microservices. Spring Cloud Config introduces three runtime behaviors that are absent from statically configured applications and that each independently produce Stripe double-charges when developers apply standard idempotency patterns without accounting for them.

The three behaviors are: dynamic bean re-initialization when a @RefreshScope-annotated service is refreshed via ContextRefresher.refresh() or a Spring Cloud Bus broadcast event, which destroys the bean instance and clears all its instance-level state including any in-flight idempotency key cache; externalized @Value-injected configuration properties being refreshed at runtime, so a property used as an idempotency key component (such as a prefix or namespace discriminator) can change between the first Stripe call and the retry; and Spring Cloud Config Server’s {cipher} encryption feature silently returning a literal sentinel value (<n/a>) when the JCE keystore is unavailable after rotation, causing the application to derive all idempotency keys from the wrong HMAC signing secret for the duration of the keystore outage.

The stack for examples in this post: Spring Boot 3.x, spring-cloud-config-client (Spring Cloud 2023.x), spring-cloud-starter-bus-amqp or spring-cloud-starter-bus-kafka for broadcast refresh, Spring Retry (spring-retry), Stripe Java SDK v24+, WireMock 3.x, JUnit 5.

Background: how Spring Cloud Config changes the assumptions under which idempotency key derivation runs

In a statically configured Spring Boot application, configuration properties are resolved once at startup from application.properties, environment variables, and command-line arguments. The resolved values are bound into beans and remain constant for the lifetime of the JVM process. Idempotency key derivation strategies that rely on configuration-backed values — a prefix string, a namespace discriminator, or an HMAC signing secret — are therefore stable: the same business inputs always produce the same key as long as the process is running.

Spring Cloud Config Client breaks this assumption. It introduces two mutation points that can alter configuration values at runtime without restarting the JVM:

1. @RefreshScope beans. Any Spring bean annotated with @RefreshScope is managed by a special scope that wraps the bean in a proxy. When ContextRefresher.refresh() is called — either directly via POST /actuator/refresh, or as a result of a Spring Cloud Bus RefreshRemoteApplicationEvent — all @RefreshScope beans are marked stale. The next time a method is called on the proxy, the underlying bean instance is destroyed and a new instance is created with the current configuration values injected. Instance-level state that the bean accumulated between creation and destruction — including cached in-flight idempotency keys — is permanently lost.
2. @Value-injected properties refresh in @RefreshScope beans. A field annotated with @Value("${billing.idempotency.prefix}") in a @RefreshScope bean is re-injected with the current value from the Config Server every time the bean is re-created. If the Config Server’s backing Git repository was updated between the first Stripe call and the retry, the re-injected value is different. Idempotency keys that include this property as a component will differ across the refresh boundary.

A third issue is specific to Spring Cloud Config Server’s {cipher} property encryption feature, described in Mode 3. Together, these three behaviors produce the failure modes in this post.

Mode 1: @RefreshScope bean destruction between the Stripe call and the retry clears the in-flight idempotency key cache — retry finds no cached entry — UUID.randomUUID() generates UUID_B — ch_B committed alongside already-committed ch_A

The developer builds a billing service that handles network-level retries (Stripe call timeout, socket reset) by maintaining an in-flight key cache as an instance field on the service bean. The cache is a ConcurrentHashMap<String, String> keyed by subscriptionId + ":" + billingPeriod and storing the generated idempotency key. Before calling Stripe, the service checks the cache: if an entry exists (a prior attempt that timed out), it reuses the key; otherwise it generates a new UUID and stores it. After a successful charge, the entry is removed from the cache.

To pick up refreshed configuration properties without requiring a rolling restart — specifically, an updated Stripe rate limit (billing.stripe.max-requests-per-second) and a refreshed list of allowed billing periods — the developer annotates the billing service with @RefreshScope. The developer reasons: “I annotate the service with @RefreshScope so property changes take effect at runtime. The in-flight key cache is instance state on the same bean. During normal operation, a config refresh happens infrequently — perhaps once per deploy. The window during which a refresh could coincide with an in-flight Stripe call is tiny.”

The flaw is that the window is not as small as it appears in development and that Spring Cloud Bus broadcasts the refresh event to all running instances simultaneously. In production, a post-deploy config push triggers a bus broadcast that reaches all instances at once, and under load, the probability of a refresh coinciding with a Stripe call in-flight (including TCP-level retries) is non-negligible. And when it does coincide, the consequence is a double charge.

// BillingService.java — @RefreshScope with in-flight key cache as instance state
@Service
@RefreshScope   // re-created on ContextRefresher.refresh() or Bus RefreshRemoteApplicationEvent
public class BillingService {

    // @Value re-injected from Config Server on every bean re-creation.
    @Value("${billing.stripe.max-requests-per-second:10}")
    private int maxRequestsPerSecond;

    @Value("${billing.allowed-billing-periods:}")
    private List<String> allowedBillingPeriods;

    @Autowired
    private StripeClient stripeClient;

    @Autowired
    private SubscriptionRepository subscriptionRepository;

    // In-flight key cache: populated before the Stripe call, checked on retry.
    // This is INSTANCE STATE — it is DESTROYED when @RefreshScope re-creates the bean.
    private final ConcurrentHashMap<String, String> inflightKeys =
        new ConcurrentHashMap<>();

    public String chargeSubscription(String subscriptionId, String billingPeriod) {

        String cacheKey = subscriptionId + ":" + billingPeriod;

        // Check cache: if a prior timed-out attempt put a key here, reuse it.
        String idempotencyKey = inflightKeys.computeIfAbsent(
            cacheKey,
            k -> UUID.randomUUID().toString()
        );

        try {
            ChargeResponse charge = stripeClient.createCharge(
                subscriptionRepository.findCustomerId(subscriptionId),
                subscriptionRepository.findAmountCents(subscriptionId),
                billingPeriod,
                idempotencyKey
            );

            // Success: remove the key from the in-flight cache.
            inflightKeys.remove(cacheKey);
            return charge.getId();

        } catch (StripeConnectTimeoutException | SocketTimeoutException e) {
            // Timeout — key stays in inflightKeys so a retry reuses it.
            // If @RefreshScope destroys the bean before the retry, inflightKeys
            // is gone. The next call to chargeSubscription() finds an empty map.
            // computeIfAbsent() runs UUID.randomUUID() → UUID_B.
            throw new RetryableChargeException("Stripe call timed out", e);
        }
    }
}

The failure sequence:

1. chargeSubscription("sub_001", "2026-10") is called. Cache miss → UUID_A generated and stored in inflightKeys["sub_001:2026-10"].
2. stripeClient.createCharge(..., UUID_A) is called. Stripe commits ch_A. The TCP connection experiences a post-commit timeout (response is in Stripe’s send buffer, but the client’s socket read times out before the response is delivered).
3. StripeConnectTimeoutException is thrown. inflightKeys still holds UUID_A. The caller’s retry scheduler will call chargeSubscription again in a few seconds.
4. Before the retry, a Spring Cloud Bus RefreshRemoteApplicationEvent arrives at this instance (triggered by a config push to update billing.stripe.max-requests-per-second). ContextRefresher.refresh() processes the event. All @RefreshScope-annotated beans are marked stale.
5. The retry fires. The proxy for BillingService is called. Because the bean was marked stale, the proxy destroys the current bean instance and creates a new one. The new instance has a fresh inflightKeys = new ConcurrentHashMap<>() — empty.
6. Inside the new bean instance, inflightKeys.computeIfAbsent("sub_001:2026-10", k -> UUID.randomUUID().toString()) finds no entry. UUID_B is generated and stored.
7. stripeClient.createCharge(..., UUID_B) is called. Stripe has no record of UUID_B. It processes this as a new charge. ch_B is committed alongside already-committed ch_A.
8. Customer is billed twice for the same billing period.

The developer misconception is: “The in-flight key cache is an instance field on the bean. A config refresh updates the injected @Value fields but does not touch the instance state I manage myself. @RefreshScope does a property refresh, not a full reconstruction.”

This is incorrect. @RefreshScope is not a property-patch mechanism. It destroys the entire bean instance and constructs a new one, running all constructors and post-construct initializers. Any mutable instance state — collections, caches, counters, connection pools allocated in the constructor — is recreated from scratch. There is no merge of old instance state into the new instance. The new ConcurrentHashMap field initializer in the class body creates an empty map regardless of what the old map contained.

Why the cache cannot safely live on a @RefreshScope bean

The design intent of @RefreshScope is that the bean’s public behavior after re-creation is equivalent to the behavior before, modulo the updated configuration values. This contract holds for stateless services or for services whose state is fully derived from configuration. It does not hold for services that accumulate request-level transactional state — the in-flight key cache is precisely this kind of state.

The correct approach is to never store in-flight idempotency keys in a @RefreshScope bean. Instead, derive keys from stable business inputs so that no cache is needed. A key derived from SHA-256(subscriptionId + billingPeriod + amountCents) is the same on every call to chargeSubscription for a given billing event, regardless of which bean instance handles the call and regardless of how many times the bean has been refreshed. Stripe’s 24-hour idempotency cache matches the key and returns ch_A without creating ch_B.

// Fixed BillingService.java — no in-flight key cache; key derived from stable inputs
@Service
@RefreshScope
public class BillingService {

    @Value("${billing.stripe.max-requests-per-second:10}")
    private int maxRequestsPerSecond;

    @Autowired
    private StripeClient stripeClient;

    @Autowired
    private SubscriptionRepository subscriptionRepository;

    // No inflightKeys map. No instance state that is unsafe to lose on refresh.

    public String chargeSubscription(String subscriptionId, String billingPeriod) {

        int amountCents = subscriptionRepository.findAmountCents(subscriptionId);

        // Key derived from stable business inputs — same value on every call
        // for this billing event, across all @RefreshScope re-creations.
        // SHA-256 of subscriptionId + billingPeriod + amountCents.
        String idempotencyKey = toStableKey(subscriptionId, billingPeriod, amountCents);

        ChargeResponse charge = stripeClient.createCharge(
            subscriptionRepository.findCustomerId(subscriptionId),
            amountCents,
            billingPeriod,
            idempotencyKey
        );

        return charge.getId();
    }

    static String toStableKey(String subscriptionId, String billingPeriod, int amountCents) {
        try {
            MessageDigest sha = MessageDigest.getInstance("SHA-256");
            byte[] hash = sha.digest(
                (subscriptionId + ":" + billingPeriod + ":" + amountCents)
                    .getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash).substring(0, 36);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

If you genuinely need the @RefreshScope annotation on the billing service for other reasons but also require a durable key cache, move the cache to a separate non-@RefreshScope singleton bean. The cache bean is not marked for refresh and therefore survives config updates. The billing service (which is @RefreshScope) depends on it via injection. Because injection points on a @RefreshScope bean are resolved against the application context (not the bean instance), the injected cache bean reference remains valid across re-creations.

// IdempotencyKeyCache.java — plain singleton, NOT @RefreshScope
@Component
public class IdempotencyKeyCache {

    // Survives @RefreshScope re-creation of BillingService.
    private final ConcurrentHashMap<String, String> cache = new ConcurrentHashMap<>();

    public String computeIfAbsent(String cacheKey, Supplier<String> keyFactory) {
        return cache.computeIfAbsent(cacheKey, k -> keyFactory.get());
    }

    public void remove(String cacheKey) {
        cache.remove(cacheKey);
    }
}

// BillingService.java — @RefreshScope bean that delegates key caching to singleton
@Service
@RefreshScope
public class BillingService {

    @Autowired
    private IdempotencyKeyCache keyCache;   // injected singleton; survives refresh

    @Autowired
    private StripeClient stripeClient;

    @Autowired
    private SubscriptionRepository subscriptionRepository;

    public String chargeSubscription(String subscriptionId, String billingPeriod) {
        String cacheKey = subscriptionId + ":" + billingPeriod;

        // keyCache is a singleton. Even if BillingService is re-created by @RefreshScope,
        // keyCache continues to hold the entry written before the refresh.
        String idempotencyKey = keyCache.computeIfAbsent(
            cacheKey,
            () -> UUID.randomUUID().toString()
        );

        try {
            int amountCents = subscriptionRepository.findAmountCents(subscriptionId);
            ChargeResponse charge = stripeClient.createCharge(
                subscriptionRepository.findCustomerId(subscriptionId),
                amountCents,
                billingPeriod,
                idempotencyKey
            );
            keyCache.remove(cacheKey);
            return charge.getId();
        } catch (StripeConnectTimeoutException | SocketTimeoutException e) {
            throw new RetryableChargeException("Stripe call timed out", e);
        }
    }
}

Test: assert key stability across @RefreshScope bean re-creation

@SpringBootTest
@ActiveProfiles("test")
class BillingServiceRefreshTest {

    @Autowired
    private BillingService billingService;

    @Autowired
    private ContextRefresher contextRefresher;

    @RegisterExtension
    static WireMockExtension wm = WireMockExtension.newInstance()
        .options(wireMockConfig().dynamicPort())
        .build();

    @Test
    void sameIdempotencyKeyAcrossRefreshScopeDestruction() {
        // Stub Stripe to capture idempotency keys from all calls.
        wm.stubFor(post(urlPathEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("""
                    {"id":"ch_test","object":"charge","amount":2999,"status":"succeeded"}
                    """)));

        // First call — simulates an attempt that is intercepted before completion.
        // In this test we check that refreshing the scope does not change the key.
        billingService.chargeSubscription("sub_test_1", "2026-10");

        // Trigger @RefreshScope bean destruction (simulates config server push).
        contextRefresher.refresh();

        // Second call — @RefreshScope re-creates BillingService.
        // Key must be identical to the first call if derived from stable inputs,
        // or the same cached entry if using the singleton cache pattern.
        billingService.chargeSubscription("sub_test_1", "2026-10");

        List<LoggedRequest> stripeRequests = wm.findAll(
            postRequestedFor(urlPathEqualTo("/v1/charges")));

        Set<String> keys = stripeRequests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());

        // Both calls must carry the same idempotency key.
        assertThat(keys).hasSize(1)
            .withFailMessage("@RefreshScope re-creation must not change the idempotency key");
    }
}

Mode 2: @Value-bound billing.idempotency.prefix updated by ContextRefresher.refresh() between the first Stripe call and the retry — prefix changes from billing to billing-v2 — retry key billing-v2:sub_001:2026-10 does not match cached billing:sub_001:2026-10 — Stripe commits ch_B

The developer adds a configurable prefix to all Stripe idempotency keys. The motivation is operational: after a bulk data migration or a billing model change, the team wants to invalidate all existing Stripe idempotency cache entries so that post-migration billing runs produce fresh charges rather than returning cached results from the pre-migration key space. The prefix is externalized to Spring Cloud Config (billing.idempotency.prefix=billing) so it can be updated without redeploying the service. To make the prefix change take effect at runtime, the service is annotated with @RefreshScope.

The developer reasons: “The prefix is only updated intentionally, as a deliberate operational action, to reset the idempotency key namespace after a migration. We would never update the prefix mid-billing-cycle. During a normal billing run, the prefix is stable.”

The flaw is in the implicit assumption that “intentional” and “mid-billing-cycle” are mutually exclusive. In practice, the team’s definition of “mid-billing-cycle” is “during the hour when the monthly billing cron is actively processing.” But Stripe calls are also made in response to user actions (plan upgrades, reactivations) and webhooks (invoice.payment_failed retries), which happen around the clock. A config push that updates the prefix from billing to billing-v2 at 2 AM — outside the monthly billing window — coincides with a Stripe webhook retry for a customer whose initial charge timed out at 1:58 AM.

// BillingService.java — @Value-bound prefix used as idempotency key component
@Service
@RefreshScope
public class BillingService {

    // Prefix is read from Config Server. Updated when billing.idempotency.prefix changes.
    // After @RefreshScope re-creates the bean, this field holds the NEW prefix value.
    @Value("${billing.idempotency.prefix:billing}")
    private String idempotencyPrefix;

    @Autowired
    private StripeClient stripeClient;

    public String chargeSubscription(String subscriptionId,
                                     String billingPeriod,
                                     int amountCents) {
        // Key includes the prefix injected from Config Server.
        // "billing:sub_001:2026-10" on first attempt (prefix=billing).
        // "billing-v2:sub_001:2026-10" on retry (prefix=billing-v2 after refresh).
        String idempotencyKey =
            idempotencyPrefix + ":" + subscriptionId + ":" + billingPeriod;

        return stripeClient.createCharge(
            /* customerId */ lookupCustomer(subscriptionId),
            amountCents,
            billingPeriod,
            idempotencyKey
        ).getId();
    }
}

The failure sequence:

1. chargeSubscription("sub_001", "2026-10", 2999) is called at 1:58 AM. At this moment, idempotencyPrefix = "billing".
2. idempotencyKey = "billing:sub_001:2026-10". Stripe is called with this key. Stripe commits ch_A.
3. A SocketTimeoutException fires on the HTTP response read path. The Stripe HTTP client throws a connect-timeout exception. The application treats this as a retriable failure and schedules a retry.
4. At 2:00 AM, a scheduled config push updates billing.idempotency.prefix from billing to billing-v2 in the Config Server’s Git repository. A Spring Cloud Bus RefreshRemoteApplicationEvent is broadcast to all application instances. ContextRefresher.refresh() processes the event. The BillingService bean is marked stale.
5. At 2:01 AM, the retry fires. The BillingService proxy detects the stale mark and re-creates the bean. The new instance has idempotencyPrefix = "billing-v2".
6. idempotencyKey = "billing-v2:sub_001:2026-10". This is a completely different string from "billing:sub_001:2026-10". Stripe’s 24-hour idempotency cache is keyed by the full string. "billing-v2:sub_001:2026-10" has no entry in the cache.
7. Stripe processes this as a new charge and commits ch_B. Customer sub_001 is charged twice for the October 2026 billing period.

The developer misconception is: “Stripe’s idempotency cache deduplicates based on the combination of key, amount, currency, and customer — not just the key string. Two calls for the same customer, same amount, and same billing period should be recognized as duplicates even if the key strings differ.”

This is incorrect. Stripe’s idempotency mechanism is keyed exclusively by the Idempotency-Key header value. Two requests with the same amount, currency, and customer but different Idempotency-Key values are treated as separate, independent charges. The header value is opaque to Stripe; Stripe has no knowledge of the encoding scheme that produced it. "billing:sub_001:2026-10" and "billing-v2:sub_001:2026-10" are different strings and therefore different keys.

Why any config-derived component in an idempotency key is dangerous

The problem is not limited to explicit prefix properties. Any @Value-injected field whose value contributes to key derivation is a latent failure: the field can change independently of the business inputs that identify the billing event. Common examples include:

• A @Value("${spring.application.name}") field used to namespace keys across services (e.g., during a service split when the application is renamed).
• A @Value("${billing.version}") or @Value("${billing.schema-version}") used to version the key space after schema migrations.
• A @Value("${billing.stripe.account-id}") used to disambiguate keys across multiple Stripe accounts (if the account ID changes because the team switches to a different Stripe account for a market).
• A @Value("${app.region}") field used to shard keys by deployment region (if the region property is corrected after a misconfiguration).

The fix is the same in all cases: exclude configuration-backed values from idempotency key derivation. The key should be derived exclusively from business inputs that are intrinsic to the billing event: the subscription ID, the billing period, the charge amount, and (if needed) the currency. These inputs are stable across all retries, all bean re-creations, all config refreshes, and all application restarts. A SHA-256 hash of these inputs produces a key that is globally unique per billing event without depending on any runtime-mutable configuration state.

// Fixed BillingService.java — no config-derived key component
@Service
@RefreshScope
public class BillingService {

    // Prefix field removed entirely. Not needed if key is derived from business inputs.

    // These fields are refreshable (config Server can update them) without
    // affecting idempotency key derivation, because they are not used in toStableKey().
    @Value("${billing.stripe.max-requests-per-second:10}")
    private int maxRequestsPerSecond;

    @Autowired
    private StripeClient stripeClient;

    public String chargeSubscription(String subscriptionId,
                                     String billingPeriod,
                                     int amountCents) {
        // Key derived ONLY from business inputs. No config-backed components.
        // Same value before and after any config refresh.
        String idempotencyKey = toStableKey(subscriptionId, billingPeriod, amountCents);

        return stripeClient.createCharge(
            lookupCustomer(subscriptionId),
            amountCents,
            billingPeriod,
            idempotencyKey
        ).getId();
    }

    static String toStableKey(String subscriptionId,
                               String billingPeriod,
                               int amountCents) {
        try {
            MessageDigest sha = MessageDigest.getInstance("SHA-256");
            byte[] digest = sha.digest(
                (subscriptionId + ":" + billingPeriod + ":" + amountCents)
                    .getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(digest).substring(0, 36);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

If you need to maintain separate idempotency key namespaces for operational reasons (e.g., post-migration re-billing), encode the namespace in the business inputs rather than in a config-backed component. For example, introduce a billingRunId column in the subscription table that changes only when you explicitly initiate a new billing run, and include it in the hash. The billingRunId is a business-level datum stored in the database, not a config-level property, so it cannot be accidentally changed by a config refresh.

Test: assert key stability across @Value prefix change via config refresh

@SpringBootTest
@ActiveProfiles("test")
class BillingServicePrefixRefreshTest {

    @Autowired
    private BillingService billingService;

    @Autowired
    private ContextRefresher contextRefresher;

    @Autowired
    private ConfigurableEnvironment environment;

    @RegisterExtension
    static WireMockExtension wm = WireMockExtension.newInstance()
        .options(wireMockConfig().dynamicPort())
        .build();

    @Test
    void idempotencyKeyUnchangedAfterPrefixPropertyRefresh() {
        wm.stubFor(post(urlPathEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_test\",\"object\":\"charge\","
                        + "\"amount\":2999,\"status\":\"succeeded\"}")));

        // First call with prefix=billing.
        billingService.chargeSubscription("sub_test_1", "2026-10", 2999);

        // Simulate config server updating billing.idempotency.prefix to billing-v2.
        MapPropertySource updated = new MapPropertySource(
            "test-override",
            Map.of("billing.idempotency.prefix", "billing-v2"));
        environment.getPropertySources().addFirst(updated);
        contextRefresher.refresh();   // triggers @RefreshScope bean re-creation

        // Second call — should produce the same idempotency key as the first call.
        billingService.chargeSubscription("sub_test_1", "2026-10", 2999);

        List<LoggedRequest> requests = wm.findAll(
            postRequestedFor(urlPathEqualTo("/v1/charges")));

        Set<String> keys = requests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());

        assertThat(keys).hasSize(1)
            .withFailMessage(
                "Config prefix change must not affect idempotency key; "
                + "got multiple keys: " + keys);
    }
}

Mode 3: Spring Cloud Config {cipher}-encrypted HMAC secret decryption failure after keystore rotation returns the literal sentinel value <n/a> as the signing key — all keys computed during the keystore outage are derived from the wrong secret — recovery attempt uses wrong HMAC — UUID_B generated — ch_B committed

Spring Cloud Config Server supports encrypting sensitive configuration properties using JCE (Java Cryptography Extension) keystores. Properties are stored in Git as {cipher}AQB... ciphertext. When a client fetches configuration, the Config Server decrypts each {cipher} property and returns the plaintext value. The developer stores the HMAC signing secret used to derive Stripe idempotency keys as an encrypted property: billing.idempotency.hmac-secret={cipher}AQBk...longciphertext....

The developer’s idempotency key derivation uses this secret to compute HMAC-SHA256(hmacSecret, subscriptionId + ":" + billingPeriod + ":" + amountCents). The motivation for using an HMAC rather than a plain SHA-256 hash is to prevent external parties from predicting idempotency keys from known billing inputs, reducing the risk of key exhaustion attacks.

The Config Server’s JCE keystore is rotated as part of a routine security key rotation. During the rotation window, the old keystore is decommissioned and the new keystore is being deployed. For a brief interval — perhaps 30 to 120 seconds — the Config Server’s decryption endpoint cannot process {cipher} properties. Spring Cloud Config Server’s default behavior when decryption fails depends on the spring.cloud.config.server.encrypt.fail-on-error property.

When fail-on-error=false (the default), the Config Server does not throw a 5xx error when it cannot decrypt a property. Instead, it returns the literal string <n/a> as the property value, with an additional header X-Config-State: INVALID. The client receives billing.idempotency.hmac-secret=<n/a>. No exception is thrown at the client. The application starts (or the @RefreshScope refresh completes) with hmacSecret = "<n/a>".

# application.yml on Config Server (git repository)
billing:
  idempotency:
    hmac-secret: "{cipher}AQBkP9mXrVs2jNwFqYzHLe8oIuT4cKdG..."

# bootstrap.yml on client (spring-cloud-config client)
spring:
  cloud:
    config:
      uri: https://config.internal
      fail-fast: true   # startup failure if server unreachable
      # Note: fail-fast=true does NOT control {cipher} decryption failure behavior.
      # Decryption failure behavior is controlled by:
      #   spring.cloud.config.server.encrypt.fail-on-error (server-side, default false)
      # When fail-on-error=false, decryption failure returns <n/a> — not a client error.

// BillingService.java — HMAC-based key derivation with {cipher}-backed secret
@Service
@RefreshScope
public class BillingService {

    // Injected from Config Server. Under normal conditions: "correct-hmac-secret-value".
    // During keystore rotation outage: "<n/a>" (sentinel returned by Config Server
    // when fail-on-error=false and decryption fails).
    @Value("${billing.idempotency.hmac-secret}")
    private String hmacSecret;

    @Autowired
    private StripeClient stripeClient;

    public String chargeSubscription(String subscriptionId,
                                     String billingPeriod,
                                     int amountCents) {
        // Key is HMAC-SHA256(hmacSecret, subscriptionId:billingPeriod:amountCents).
        // When hmacSecret = "correct-secret": key = valid_hmac_hex_prefix.
        // When hmacSecret = "<n/a>":          key = wrong_hmac_hex_prefix.
        // The two keys are different strings. Stripe treats them as different keys.
        String idempotencyKey = toHmacKey(hmacSecret, subscriptionId, billingPeriod, amountCents);

        return stripeClient.createCharge(
            lookupCustomer(subscriptionId),
            amountCents,
            billingPeriod,
            idempotencyKey
        ).getId();
    }

    static String toHmacKey(String secret, String subscriptionId,
                             String billingPeriod, int amountCents) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(
                secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
            byte[] hmac = mac.doFinal(
                (subscriptionId + ":" + billingPeriod + ":" + amountCents)
                    .getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hmac).substring(0, 36);
        } catch (NoSuchAlgorithmException | InvalidKeyException e) {
            throw new IllegalStateException(e);
        }
    }
}

The failure sequence:

1. At t=0s, the Config Server begins the keystore rotation. The old keystore is decommissioned. The new keystore deployment is in progress.
2. At t=5s, a background billing job calls chargeSubscription("sub_001", "2026-10", 2999). The bean was last refreshed at t=-10s (30 seconds before the rotation started), when hmacSecret = "correct-secret". The key is HMAC("correct-secret", "sub_001:2026-10:2999") → key_A. Stripe is called with key_A. Stripe commits ch_A.
3. At t=8s, the Stripe response is delayed by a brief network spike. The client’s read timeout fires (configured at 3 seconds). StripeConnectTimeoutException is thrown. The retry is scheduled for t=13s.
4. At t=10s, a periodic Spring Cloud Bus heartbeat triggers a RefreshRemoteApplicationEvent. ContextRefresher.refresh() fires. The Config Server is queried. Because the keystore rotation is mid-flight, decryption of billing.idempotency.hmac-secret fails. The Config Server returns hmac-secret=<n/a> with header X-Config-State: INVALID. No exception is thrown by the client. The BillingService bean is re-created with hmacSecret = "<n/a>".
5. At t=13s, the retry fires. The proxy re-creates BillingService (it was marked stale). hmacSecret = "<n/a>".
6. Key derivation: HMAC("<n/a>", "sub_001:2026-10:2999") → key_B. key_B ≠ key_A.
7. Stripe is called with key_B. Stripe has no cache entry for key_B. It processes this as a new charge. ch_B is committed alongside ch_A.
8. The keystore rotation completes at t=90s. The next config refresh returns the correct hmacSecret. But ch_B is already committed and has already settled.

The developer misconception is: “The client sets spring.cloud.config.fail-fast=true, so any config fetch failure will cause the application to fail loudly rather than silently accepting a bad configuration value. Decryption failure is a config fetch failure, so it will be caught by fail-fast=true.”

This is incorrect. spring.cloud.config.fail-fast=true controls the behavior when the Config Server is unreachable — when the HTTP connection to the Config Server fails entirely. It has no effect on the behavior when the Config Server is reachable but returns a property value that is a decryption sentinel. The fail-fast property guards against Config Server downtime, not against Config Server decryption failures. The decryption failure behavior is controlled by the server-side spring.cloud.config.server.encrypt.fail-on-error property, which defaults to false (silent sentinel) rather than true (5xx error).

Why the silent sentinel behavior exists and why it makes billing dangerous

The fail-on-error=false default exists to support gradual keystore rotation: if the Config Server threw a 5xx for every encrypted property during the key rotation window, all clients would fail their refresh cycles simultaneously, potentially triggering cascading restarts. The sentinel allows clients to continue running with stale configuration for non-critical properties (e.g., feature flags, tuning parameters) while the rotation completes. The assumption baked into this default is that applications will validate or log X-Config-State: INVALID headers for critical properties rather than silently accepting the sentinel.

For an HMAC signing secret used in idempotency key derivation, this assumption fails silently in the worst possible way: the sentinel <n/a> is a valid string that passes @NotBlank validation, constructs a valid SecretKeySpec, and produces a valid-looking 64-character hex string as the idempotency key. There is no exception. There is no log line that says “the idempotency key for this request was derived from the decryption failure sentinel.” The Stripe double-charge is the first observable symptom, and it appears hours later in Stripe’s dashboard or in a customer complaint.

Fix 1: enable server-side fail-on-error=true

Set spring.cloud.config.server.encrypt.fail-on-error=true in the Config Server. With this setting, any decryption failure causes the Config Server to return a 500 Internal Server Error for the entire property set, not a partial response with sentinel values. The client’s @RefreshScope refresh fails, the bean is not re-created, and the bean retains its last-known-good configuration. The Stripe call uses the pre-rotation HMAC secret, which is still correct for keys derived before the rotation. The cost is that no configuration properties are refreshed during the keystore outage, but this is preferable to silently accepting a sentinel that corrupts idempotency key derivation.

# Config Server application.yml
spring:
  cloud:
    config:
      server:
        encrypt:
          fail-on-error: true   # return 500 on decryption failure instead of <n/a> sentinel

Fix 2: validate the injected secret and refuse to derive keys from the sentinel

Add a @PostConstruct guard in the BillingService that validates hmacSecret before the bean enters active use. If the value is the sentinel <n/a> (or any other known invalid value), throw an exception from @PostConstruct. Spring marks a bean-creation failure when @PostConstruct throws, and the proxy remains backed by the previous (valid) bean instance rather than switching to the failed new instance.

@Service
@RefreshScope
public class BillingService {

    @Value("${billing.idempotency.hmac-secret}")
    private String hmacSecret;

    @PostConstruct
    void validateHmacSecret() {
        // Sentinel returned by Config Server when {cipher} decryption fails.
        // If this guard throws, @RefreshScope re-creation fails. The proxy keeps
        // the previous (valid) bean instance. No silent sentinel propagation.
        if ("<n/a>".equals(hmacSecret) || hmacSecret.isBlank()) {
            throw new IllegalStateException(
                "billing.idempotency.hmac-secret is a decryption sentinel or blank. "
                + "Config Server keystore may be unavailable. "
                + "Refusing to initialize BillingService with invalid HMAC secret.");
        }
        // Optionally: assert minimum key length (NIST SP 800-107 recommends ≥256 bits).
        if (hmacSecret.length() < 32) {
            throw new IllegalStateException(
                "billing.idempotency.hmac-secret is shorter than 32 characters. "
                + "Minimum entropy requirement not met.");
        }
    }

    // ... rest of BillingService
}

Fix 3: eliminate HMAC-based key derivation; use plain SHA-256 of stable inputs

The simplest fix is to eliminate the HMAC secret entirely by switching to a plain SHA-256 hash of business inputs. The HMAC’s purpose is to prevent key prediction by external parties. But Stripe idempotency keys are submitted in the Idempotency-Key header, which is not visible to parties external to your own HTTP client code. The only parties who can observe the idempotency keys you submit to Stripe are: your own code, Stripe (who is trusted), and any proxy sitting between your code and Stripe (which should be your own infrastructure). Key prediction attacks against Stripe idempotency keys from external parties are not a realistic threat model for most billing systems.

If the HMAC is removed and a plain SHA-256 is used, the billing service no longer depends on any configuration-backed secret for key derivation. {cipher} decryption failures become irrelevant to idempotency correctness. The service can safely use @RefreshScope without risk of sentinel propagation.

// Fixed BillingService.java — no HMAC secret; SHA-256 of stable business inputs only
@Service
@RefreshScope
public class BillingService {

    // hmacSecret field removed. No config-backed component in key derivation.
    // Other @Value fields (rate limits, allowed periods) are safe to refresh.

    @Value("${billing.stripe.max-requests-per-second:10}")
    private int maxRequestsPerSecond;

    @Autowired
    private StripeClient stripeClient;

    public String chargeSubscription(String subscriptionId,
                                     String billingPeriod,
                                     int amountCents) {
        String idempotencyKey = toStableKey(subscriptionId, billingPeriod, amountCents);

        return stripeClient.createCharge(
            lookupCustomer(subscriptionId),
            amountCents,
            billingPeriod,
            idempotencyKey
        ).getId();
    }

    static String toStableKey(String subscriptionId,
                               String billingPeriod,
                               int amountCents) {
        try {
            MessageDigest sha = MessageDigest.getInstance("SHA-256");
            byte[] digest = sha.digest(
                (subscriptionId + ":" + billingPeriod + ":" + amountCents)
                    .getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(digest).substring(0, 36);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

Test: assert key stability when Config Server returns <n/a> sentinel

@SpringBootTest
@ActiveProfiles("test")
class BillingServiceCipherSentinelTest {

    @Autowired
    private BillingService billingService;

    @Autowired
    private ContextRefresher contextRefresher;

    @Autowired
    private ConfigurableEnvironment environment;

    @RegisterExtension
    static WireMockExtension wm = WireMockExtension.newInstance()
        .options(wireMockConfig().dynamicPort())
        .build();

    @Test
    void keyUnchangedWhenHmacSecretBecomesDecryptionSentinel() {
        wm.stubFor(post(urlPathEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody("{\"id\":\"ch_test\",\"object\":\"charge\","
                        + "\"amount\":2999,\"status\":\"succeeded\"}")));

        // First call — correct hmac secret.
        billingService.chargeSubscription("sub_test_1", "2026-10", 2999);

        // Simulate Config Server returning <n/a> for the HMAC secret
        // (as happens when keystore decryption fails with fail-on-error=false).
        MapPropertySource sentinel = new MapPropertySource(
            "sentinel-override",
            Map.of("billing.idempotency.hmac-secret", "<n/a>"));
        environment.getPropertySources().addFirst(sentinel);

        // @PostConstruct guard should prevent bean re-creation with sentinel value.
        // ContextRefresher.refresh() should leave the current bean in place.
        assertThatNoException().isThrownBy(() -> contextRefresher.refresh());

        // Second call — bean must not have been re-created with sentinel.
        billingService.chargeSubscription("sub_test_1", "2026-10", 2999);

        List<LoggedRequest> requests = wm.findAll(
            postRequestedFor(urlPathEqualTo("/v1/charges")));

        Set<String> keys = requests.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());

        assertThat(keys).hasSize(1)
            .withFailMessage(
                "Decryption sentinel must not change idempotency key; "
                + "got multiple keys: " + keys);
    }
}

Summary

Quick diagnostic checklist for Spring Cloud Config + Stripe deployments

• Does any billing-related bean carry @RefreshScope? If yes: does the bean hold any in-flight idempotency key state (a cache, a map, any instance field that stores a key between a Stripe call and its confirmation)? If yes: either derive keys from stable inputs (no cache) or move the cache to a non-@RefreshScope singleton.
• Does the idempotency key derivation code read any @Value-injected field? If yes: is that field refreshable (i.e., does it live on a @RefreshScope bean)? If yes: move the key derivation to use only parameters passed in from the call site, not injected fields.
• Is the HMAC signing secret (or any secret used in key derivation) stored as a {cipher} property in the Config Server? If yes: is spring.cloud.config.server.encrypt.fail-on-error set to true? If no: add a @PostConstruct guard that rejects the sentinel value; or replace HMAC with plain SHA-256 to eliminate the secret dependency.
• Does the Spring Cloud Bus broadcast refresh events to all instances? If yes: test what happens to in-flight Stripe calls when a refresh fires. In staging, trigger a POST /actuator/busrefresh while a Stripe call (proxied through WireMock with a 5-second delay) is in flight, and verify that the idempotency key arriving at WireMock is identical on the retry.
• Does the Config Server return X-Config-State: INVALID for any property after a keystore rotation? Instrument the @RefreshEndpoint or add a EnvironmentChangeEvent listener that logs a warning whenever a property changes to a known sentinel value.

Relationship to other Stripe idempotency failure modes

The three modes in this post share a common root cause with the modes documented in earlier posts in this series: the idempotency key derivation formula is coupled to runtime-mutable state. In the Spring Data Elasticsearch post, the mutable state is a UUID.randomUUID() call inside a @Retryable method body. In the Spring Cloud OpenFeign post, it is a RequestInterceptor called on every Feign retry attempt. In this post, the mutable state is either a @RefreshScope-cleared cache, a @Value-refreshed prefix, or a {cipher}-decrypted secret that can silently become a sentinel.

The unifying fix across all these modes is the same: derive the idempotency key from inputs that are stable across all retry attempts and all bean re-creations, using a deterministic function. For Stripe billing, the stable inputs are the subscription ID, the billing period, and the charge amount. A SHA-256 hash of these three inputs produces a key that is unique per billing event, reproducible on any retry, and immune to all the Spring Cloud Config runtime dynamics described above.

If you run autonomous agents that call Stripe — or proxy Stripe calls through a billing microservice that uses Spring Cloud Config — consider a key proxy layer that enforces idempotency key policies at the HTTP level, between your agent and Stripe. A proxy can reject requests that arrive with a key that does not match the hash of the request body, flag requests that arrive with a key that has already been used within the idempotency window for a different amount or currency, and maintain an audit log of every key that was submitted and what Stripe returned. This makes double-charge incidents detectable in near-real time rather than discoverable only after a customer complaint or a Stripe dispute.