OkHttp and Retrofit Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

OkHttp’s application interceptors run on the initial request and on every transparent connection-pool retry — UUID.randomUUID() inside Interceptor.intercept() to generate the Idempotency-Key header produces a different value on the initial request and on OkHttp’s transparent retry: the initial POST /v1/charges creates ch_A before a premature connection close from an AWS ALB idle timeout, and OkHttp’s internal retry fires the interceptor chain again, evaluates a fresh UUID, and Stripe creates ch_B. Three OkHttp and Retrofit-specific Stripe billing failure modes: an application Interceptor computes UUID.randomUUID() per intercept() invocation — subtler variant: moving UUID generation to a network interceptor via addNetworkInterceptor() makes the failure worse, not better; a Retrofit async retry handler re-invokes the service interface method to get a fresh Call object — UUID.randomUUID() at the call site creates ch_B on retry 1, with a Kotlin coroutine variant that re-evaluates UUID on every coroutine retry iteration; and a per-JVM ScheduledExecutorService fires the billing job independently on every Kubernetes replica with no cross-pod coordination — three pods pass the same concurrent database check, generate distinct UUID.randomUUID() values per customer, and create ch_A, ch_B, and ch_C per billing period.

This post covers all three failure modes with Kotlin and Java code, content-hash idempotency keys stable across OkHttp interceptor re-invocations, Retrofit service method re-calls, and multi-pod concurrent billing loops, pg_try_advisory_lock() for cross-pod scheduler serialization, pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For the AOP proxy retry pattern in Spring Boot (@Retryable), see the Spring Boot and Stripe Integration post. For the decorator chain retry pattern in Armeria (RetryingClient), see the Armeria and Stripe Integration post. For the reactive retry pattern in Spring WebFlux (Mono.retryWhen()), see the Spring WebFlux and Stripe Integration post.

Failure mode 1: OkHttp application Interceptor computes UUID.randomUUID() inside intercept() — retryOnConnectionFailure=true triggers a transparent connection-pool retry that re-invokes the interceptor chain — initial request created ch_A before a pooled-connection SocketException — retry’s interceptor creates ch_B

OkHttp supports two categories of interceptors: application interceptors (registered via addInterceptor()) and network interceptors (registered via addNetworkInterceptor()). Application interceptors are intended for logical request concerns — adding authentication headers, logging, caching — and are described in OkHttp’s documentation as running “once per logical request.” This description is accurate for redirects and authentication challenges, where OkHttp routes additional network requests through the interceptor stack. But it is incomplete for the case that matters most for Stripe billing: transparent connection-pool retries triggered by retryOnConnectionFailure=true, which is the default setting for every OkHttpClient built without explicitly calling .retryOnConnectionFailure(false).

When OkHttp attempts to reuse a pooled HTTPS connection to api.stripe.com and the server closes the connection before the response arrives — a common occurrence when an AWS ALB or Nginx upstream proxy has an idle timeout shorter than OkHttp’s connection pool keepAliveDuration — OkHttp detects the premature close (SocketException: Connection reset or StreamResetException) and transparently retries the request on a fresh connection. This retry is invisible to the calling code: no exception is thrown, no callback is invoked. The application interceptor chain runs again in full for the retry, including any interceptor that computes headers. If that interceptor calls UUID.randomUUID() inside intercept() to generate the Idempotency-Key header, the retry carries a different UUID than the original request:

// StripeIdempotencyInterceptor.kt
// UNSAFE: UUID.randomUUID() computed inside intercept() — called on the initial
// request AND on OkHttp's transparent connection-pool retry.
// If the initial request created ch_A before the pooled connection was reset,
// the retry's fresh UUID causes Stripe to create ch_B.

import okhttp3.Interceptor
import okhttp3.Response
import java.util.UUID

class StripeIdempotencyInterceptor : Interceptor {
    override fun intercept(chain: Interceptor.Chain): Response {
        val original = chain.request()

        // UNSAFE: UUID.randomUUID() called per intercept() invocation.
        // Initial request:        UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Connection-pool retry:  UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        val idempotencyKey = UUID.randomUUID().toString()

        val modified = original.newBuilder()
            .header("Idempotency-Key", idempotencyKey)
            .build()

        return chain.proceed(modified)
    }
}

// Registration — retryOnConnectionFailure is true by default:
val okHttpClient = OkHttpClient.Builder()
    .addInterceptor(StripeIdempotencyInterceptor())  // UNSAFE interceptor registered here
    .connectTimeout(10, TimeUnit.SECONDS)
    .readTimeout(30, TimeUnit.SECONDS)
    // .retryOnConnectionFailure(false)  // NOT set — default true enables transparent retries
    .build()

The failure scenario: the billing service calls stripeClient.charge("cust_123", "2026-09", 9900L). OkHttp selects a pooled HTTPS connection to api.stripe.com from its connection pool. The connection has been idle for 55 seconds. The AWS ALB between OkHttp and Stripe’s origin servers has an idle timeout of 60 seconds, but Stripe’s load balancer has a 55-second timeout. The connection is technically in the pool but the server-side socket is already closed. OkHttp dispatches the POST /v1/charges request through the interceptor chain. StripeIdempotencyInterceptor.intercept() fires. UUID.randomUUID() returns "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e". The request is sent on the stale connection. Stripe’s HTTP stack receives the complete request body before the TCP RST arrives on OkHttp’s side. Stripe processes the charge. ch_A is committed. The RST arrives at OkHttp. OkHttp receives a SocketException: Connection reset. Because retryOnConnectionFailure=true and the request method is POST — OkHttp checks Request.isIdempotent(), and since OkHttp 4.x treats POST as retryable on connection failures for idempotency-key-annotated paths — OkHttp opens a fresh connection and re-runs the request through the interceptor chain. StripeIdempotencyInterceptor.intercept() fires again. UUID.randomUUID() returns a new, completely independent value. The retry reaches Stripe with a different Idempotency-Key. Stripe has ch_A cached against the original UUID. The retry’s UUID is new to Stripe. Stripe processes it as a fresh charge. ch_B is created. Customer 123 is charged $99 twice for September 2026.

This failure is common because the StripeIdempotencyInterceptor pattern looks correct: a dedicated interceptor for a dedicated concern, registered once, applied uniformly. The problem is invisible at code review because the interceptor’s single responsibility — set the Idempotency-Key header — appears to satisfy the Stripe documentation requirement (“set a unique idempotency key per request”). The documentation refers to per-billing-operation, not per-interceptor-invocation. The interceptor is invoked once per network send attempt, which is per-interceptor-invocation, not per-billing-operation, when connection-pool retries are active.

The subtler variant: addNetworkInterceptor() instead of addInterceptor() — network interceptors run per physical network request including redirects and auth retries — UUID in a network interceptor fires even more frequently than in an application interceptor

OkHttp documentation describes network interceptors as running “closer to the wire” than application interceptors, and some developers move idempotency key generation to a network interceptor under the assumption that it offers finer control or that it runs “only once per real network call.” This is incorrect in the context that creates duplicate charges. Network interceptors run per physical network request — including HTTP redirects (307/308 from Stripe’s CDN edge), HTTP authentication challenges (401 responses triggering OkHttp’s Authenticator), and connection-pool retries. A network interceptor that calls UUID.randomUUID() fires on each of these, not just on the application-level retry.

// UNSAFE: UUID.randomUUID() inside addNetworkInterceptor().
// Network interceptors fire on every physical network attempt:
//   - Initial request
//   - Connection-pool retry (same as application interceptor)
//   - HTTP 307/308 redirect from Stripe edge to origin
//   - Auth challenge if the Stripe key is expired (401 → Authenticator → retry)
// Each physical attempt evaluates UUID.randomUUID() independently.

val okHttpClient = OkHttpClient.Builder()
    .addNetworkInterceptor { chain ->
        val original = chain.request()
        // UNSAFE: fires per physical network request, including redirects and auth retries.
        val key = UUID.randomUUID().toString()
        chain.proceed(original.newBuilder().header("Idempotency-Key", key).build())
    }
    .build()

The developer who uses addNetworkInterceptor() thinking it avoids the application-interceptor retry problem has made the problem strictly worse: the UUID now also changes on redirects and auth challenges that the application interceptor would have treated as a single logical operation. The correct approach is the same regardless of interceptor type: compute the stable key outside OkHttp entirely, in the calling code, and set it on the request before handing it to the client.

The fix for failure mode 1

The idempotency key must be computed once per billing operation, before the OkHttpClient dispatches any network activity. An interceptor should read the key from the request it receives — set by the caller — not generate one. If the request already contains an Idempotency-Key header, the interceptor can pass it through unchanged. If it does not, the interceptor should throw rather than silently generating a random value that will differ on retries:

// Safe: idempotency key computed ONCE in calling code — before the OkHttp call.
// Interceptor reads from the incoming request — does not call UUID.randomUUID().
// OkHttp's connection-pool retry reuses the same Request object with the same header.

// BillingService.kt — calling code computes stable key before dispatching.
class BillingService(private val okHttpClient: OkHttpClient, private val stripeKey: String) {

    fun chargeCustomer(customerId: String, billingPeriod: String, amountCents: Long): ChargeResult {
        // Computed once per billing operation — deterministic across retries and pod restarts.
        val idempotencyKey = stableKey(customerId, billingPeriod)

        // Pre-flight: claim the billing slot in the database before any network activity.
        val claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)
        if (!claimed) {
            return billingRepository.findExistingCharge(customerId, billingPeriod)
        }

        val requestBody = FormBody.Builder()
            .add("amount", amountCents.toString())
            .add("currency", "usd")
            .add("customer", customerId)
            .build()

        // Key set here — before OkHttp dispatches the request.
        // Both the initial send and any transparent connection-pool retry use this header value.
        val request = Request.Builder()
            .url("https://api.stripe.com/v1/charges")
            .header("Authorization", "Bearer $stripeKey")
            .header("Idempotency-Key", idempotencyKey)  // stable, pre-computed
            .post(requestBody)
            .build()

        val response = okHttpClient.newCall(request).execute()
        return parseResponse(response)
    }

    companion object {
        fun stableKey(customerId: String, billingPeriod: String): String {
            val digest = java.security.MessageDigest.getInstance("SHA-256")
            val hash = digest.digest(
                "$customerId:$billingPeriod:okhttp-billing"
                    .toByteArray(Charsets.UTF_8)
            )
            return hash.take(16).joinToString("") { "%02x".format(it) }
        }
    }
}

// Safe pass-through interceptor — reads key from incoming request, never generates one.
class StripeIdempotencyInterceptor : Interceptor {
    override fun intercept(chain: Interceptor.Chain): Response {
        val original = chain.request()

        // Safe: read from the incoming request — key was set by the caller.
        // If the caller forgot to set it, throw rather than silently generating a random UUID
        // that will differ on OkHttp's transparent retry.
        val existingKey = original.header("Idempotency-Key")
            ?: throw IllegalStateException(
                "Stripe POST call missing Idempotency-Key header — " +
                "set it in calling code before dispatching via OkHttpClient"
            )

        // Pass through unchanged — the key is already correct and stable.
        return chain.proceed(original)
    }
}

With the key computed by stableKey(customerId, billingPeriod) in the calling code and set on the Request before okHttpClient.newCall(request).execute(), OkHttp’s connection-pool retry reuses the same Request object. The Request is immutable in OkHttp — it cannot be modified in flight. The retry sends the same headers, including the same stable Idempotency-Key. Stripe receives the same key on the initial attempt and on the retry and returns the cached ch_A result without creating ch_B. The pre-flight claimSlot() write ensures that if the process is killed after the Stripe call commits ch_A but before the response is processed, the next startup finds the slot claimed and resumes rather than creating a new charge with a fresh UUID.

Failure mode 2: Retrofit async Callback retry re-invokes the service interface method to obtain a fresh Call — UUID.randomUUID() at the call site re-evaluates per retry invocation — initial call created ch_A before IOException — retry’s fresh Call carries a new UUID — ch_B created

Retrofit generates Call objects from annotated interface methods. A Call object encapsulates a fully prepared OkHttpClient request: the URL, headers, and body are captured at the moment the interface method is invoked. Call.clone() creates a new, executable copy of the same prepared request — the headers, including any Idempotency-Key set at invocation time, are cloned intact. This means Call.clone() alone is safe: if the UUID was set when the original Call was constructed, the clone carries the same UUID.

The unsafe pattern is not clone() itself but re-invoking the service interface method to obtain a fresh Call on retry. Re-invoking the interface method re-executes the method call expression, including any arguments. If UUID.randomUUID() is computed as an argument to the @Header-annotated parameter, the argument is re-evaluated per invocation. Teams that implement retry by calling the service method again — rather than cloning the original Call — trigger this failure:

// StripeService.kt — Retrofit interface
import retrofit2.Call
import retrofit2.http.Body
import retrofit2.http.Header
import retrofit2.http.POST

interface StripeService {
    @POST("v1/charges")
    fun createCharge(
        @Header("Idempotency-Key") idempotencyKey: String,
        @Body body: ChargeRequest
    ): Call<Charge>
}

// BillingService.kt — UNSAFE async retry that re-invokes the interface method.
// stripeService.createCharge(UUID.randomUUID(), body) is called on the initial attempt
// AND called again inside onFailure() for the retry.
// Each call to createCharge() evaluates UUID.randomUUID() independently.

class BillingService(
    private val stripeService: StripeService,
    private val billingRepository: BillingRepository
) {
    fun chargeCustomerAsync(customerId: String, billingPeriod: String, amountCents: Long) {
        val body = ChargeRequest(amount = amountCents, currency = "usd", customer = customerId)

        // UNSAFE: UUID.randomUUID() evaluated here — argument expression is call-site code,
        // not Retrofit-internal. Re-invoking createCharge() re-evaluates this argument.
        stripeService.createCharge(UUID.randomUUID().toString(), body)
            .enqueue(object : Callback<Charge> {

                override fun onResponse(call: Call<Charge>, response: Response<Charge>) {
                    if (response.isSuccessful) {
                        billingRepository.record(customerId, billingPeriod, response.body()!!.id)
                    } else if (response.code() in 500..599) {
                        // Server error — retry by calling the service method again.
                        // UNSAFE: new createCharge() invocation evaluates UUID.randomUUID()
                        // afresh — different key from the initial attempt.
                        // If the initial request created ch_A before the 503, this creates ch_B.
                        stripeService.createCharge(UUID.randomUUID().toString(), body)
                            .enqueue(this)
                    }
                }

                override fun onFailure(call: Call<Charge>, t: Throwable) {
                    if (isTransient(t)) {
                        // UNSAFE: same re-invocation pattern — new UUID per retry call.
                        stripeService.createCharge(UUID.randomUUID().toString(), body)
                            .enqueue(this)
                    }
                }
            })
    }
}

The failure scenario: chargeCustomerAsync("cust_456", "2026-09", 9900L) is called. stripeService.createCharge(UUID.randomUUID(), body) evaluates the argument list. UUID.randomUUID() returns "4d8a2c1e-5b3f-4d7e-9c0a-1b2d3e4f5a6b". A Retrofit Call is created capturing this key. The call is dispatched via OkHttp. Stripe receives the POST /v1/charges. The card is authorized and ch_A is committed. Stripe’s response is a 503 Service Unavailable due to a brief load spike. The onResponse handler fires with response.code() == 503. The retry branch executes. stripeService.createCharge(UUID.randomUUID(), body) is called again. The UUID.randomUUID() argument expression evaluates a second time — independently of the first evaluation — and returns "9e7c3a5b-1d2f-4e8c-6a0b-3c4d5e6f7a8b". A new Call is created with this different key. The retry reaches Stripe. Stripe has ch_A cached against the original UUID. The retry’s UUID is new. Stripe creates ch_B. Customer 456 is charged $99 twice for September 2026.

The failure is structurally invisible at the call site: the argument looks like “generate an idempotency key for this request” and it does — it generates a key for each request. The problem is that the initial call and the retry are the same billing operation and must share the same key, but the argument expression has no memory of the first invocation.

The subtler variant: Kotlin coroutine retry { } loop re-calls the suspend fun — UUID.randomUUID() in the argument position re-evaluates on every coroutine iteration — ch_B on the first coroutine retry

Retrofit’s Kotlin coroutines adapter (via retrofit2:retrofit-adapters-kotlin:<version> or the built-in suspend fun support in Retrofit 2.6+) converts interface methods to suspend functions. Teams that use Kotlin coroutines for retry commonly write a retry helper that catches exceptions and re-calls the suspended action inside a loop. If UUID.randomUUID() is in the argument list of the suspend interface method call inside the retry lambda, each loop iteration re-invokes the method call with a new UUID:

// UNSAFE: UUID.randomUUID() inside the retry block's lambda.
// Each coroutine retry iteration re-evaluates the lambda body — including the argument
// expression passed to createCharge(). UUID.randomUUID() at the call site
// produces a new UUID per iteration.

interface StripeService {
    @POST("v1/charges")
    suspend fun createCharge(
        @Header("Idempotency-Key") idempotencyKey: String,
        @Body body: ChargeRequest
    ): Charge
}

// Coroutine retry helper
suspend fun <T> retryOnTransient(maxAttempts: Int = 3, block: suspend () -> T): T {
    var lastException: Throwable? = null
    repeat(maxAttempts) { attempt ->
        try {
            return block()  // re-executes the entire block() lambda on each attempt
        } catch (e: IOException) {
            lastException = e
            delay(500L * (attempt + 1))  // exponential backoff
        }
    }
    throw lastException!!
}

// UNSAFE call site — UUID.randomUUID() evaluated per block() invocation:
suspend fun chargeWithRetry(customerId: String, billingPeriod: String, amountCents: Long): Charge {
    val body = ChargeRequest(amount = amountCents, currency = "usd", customer = customerId)

    return retryOnTransient {
        // UNSAFE: UUID.randomUUID() inside the retry lambda re-evaluates on every attempt.
        // Attempt 0: UUID = "4d8a2c1e..."  → creates ch_A before IOException
        // Attempt 1: UUID = "9e7c3a5b..."  → Stripe creates ch_B
        // Attempt 2: UUID = "7f3e2a1c..."  → Stripe creates ch_C
        stripeService.createCharge(
            idempotencyKey = UUID.randomUUID().toString(),  // UNSAFE
            body = body
        )
    }
}

This pattern is especially common in Kotlin because Kotlin’s lambda syntax makes retry helpers feel like first-class control flow: retryOnTransient { stripeService.createCharge(...) } reads as “retry this if it fails” without drawing attention to the fact that the entire lambda body — including the argument expressions — re-executes on each retry. The UUID.randomUUID() call is an expression inside the lambda, not a constant. It evaluates fresh on every block() invocation.

The fix for both variants: compute the stable key once before the retry boundary and capture it as a val (Kotlin) or final variable (Java) that the retry lambda and the service method call site read without re-evaluating. For the async callback pattern, use Call.clone() on the original Call instead of re-invoking the interface method:

// Safe: stableKey computed ONCE before any retry boundary.
// Kotlin coroutine pattern — key captured as val outside the retry lambda:

suspend fun chargeWithRetry(customerId: String, billingPeriod: String, amountCents: Long): Charge {
    // Computed once — captured as effectively-final in the lambda closure.
    val idempotencyKey = BillingService.stableKey(customerId, billingPeriod)

    // Pre-flight: claim the billing slot before any network activity.
    if (!billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)) {
        return billingRepository.findExistingCharge(customerId, billingPeriod)
    }

    val body = ChargeRequest(amount = amountCents, currency = "usd", customer = customerId)

    return retryOnTransient {
        // Safe: idempotencyKey is a captured val from the outer scope.
        // Every retry iteration reads the same value — no UUID.randomUUID() here.
        stripeService.createCharge(
            idempotencyKey = idempotencyKey,  // stable, pre-computed
            body = body
        )
    }
}

// Safe: async Callback pattern — clone the original Call on retry, not a new service invocation:
fun chargeCustomerAsync(customerId: String, billingPeriod: String, amountCents: Long) {
    val idempotencyKey = BillingService.stableKey(customerId, billingPeriod)
    if (!billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)) return

    val body = ChargeRequest(amount = amountCents, currency = "usd", customer = customerId)

    // Safe: UUID computed once — initial Call captures it.
    val originalCall = stripeService.createCharge(idempotencyKey, body)

    originalCall.enqueue(object : Callback<Charge> {
        override fun onResponse(call: Call<Charge>, response: Response<Charge>) {
            if (response.isSuccessful) {
                billingRepository.record(customerId, billingPeriod, response.body()!!.id)
            } else if (response.code() in 500..599) {
                // Safe: clone() reuses the same Request with the same Idempotency-Key.
                // Does NOT re-evaluate UUID.randomUUID() — the key was captured when
                // originalCall was created.
                originalCall.clone().enqueue(this)
            }
        }

        override fun onFailure(call: Call<Charge>, t: Throwable) {
            if (isTransient(t)) {
                originalCall.clone().enqueue(this)  // same key, same request
            }
        }
    })
}

The val idempotencyKey = stableKey(customerId, billingPeriod) line is the critical boundary. Everything inside the retry lambda and inside the Callback reads the same captured value. Call.clone() constructs a new executable instance from the same Request object, preserving all headers including the stable key. Stripe receives the same Idempotency-Key on the initial attempt and all retries and returns the cached ch_A result.

Failure mode 3: per-JVM ScheduledExecutorService billing job runs on all Kubernetes replicas independently — TOCTOU race on hasCompletedForPeriod() — three pods generate distinct UUID.randomUUID() per customer — ch_A, ch_B, and ch_C per customer per billing period

OkHttp and Retrofit are often used in server-side JVM applications deployed as Kubernetes Deployments with multiple replicas. Billing jobs in such applications are commonly scheduled via a plain ScheduledExecutorService — Executors.newSingleThreadScheduledExecutor() or Kotlin’s fixedRateTimer() — created at application startup. Each JVM creates its own independent executor. No cross-pod coordination exists. When a billing job is scheduled on this executor, every replica runs the job independently at the same scheduled time.

If the billing job includes a guard check like hasCompletedForPeriod() that queries a shared database but is not made atomic with the billing claim, all three pods can pass the guard concurrently before any pod commits the billing-started record. All three proceed to stream the customer list and call UUID.randomUUID() per customer independently — producing ch_A from pod 1, ch_B from pod 2, and ch_C from pod 3 for each customer per billing period:

// MonthlyBillingJob.kt
// UNSAFE: scheduled at startup on every Kubernetes pod independently.
// No cross-pod coordination — all three replicas fire at the same scheduled second.
// hasCompletedForPeriod() is a non-atomic SELECT — all three pods pass before any commits.

import java.util.UUID
import java.util.concurrent.Executors
import java.util.concurrent.TimeUnit

class MonthlyBillingJob(
    private val billingRepository: BillingRepository,
    private val stripeService: StripeService
) {
    private val scheduler = Executors.newSingleThreadScheduledExecutor()

    fun start() {
        // UNSAFE: this start() is called at application startup on EVERY Kubernetes pod.
        // All three pods schedule independent billing runs with no shared lock.
        scheduler.scheduleAtFixedRate(
            { runBilling(currentBillingPeriod()) },
            computeInitialDelay(),
            30L * 24 * 60 * 60,  // 30 days in seconds
            TimeUnit.SECONDS
        )
    }

    private fun runBilling(billingPeriod: String) {
        // UNSAFE: hasCompletedForPeriod() is a plain SELECT.
        // All three pods call it at approximately the same time.
        // September billing hasn't started yet — all three return false.
        // All three pods pass this guard simultaneously (TOCTOU race).
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) return

        // All three pods reach this point simultaneously.
        val customers = billingRepository.listActive()
        for (customer in customers) {
            // UNSAFE: UUID.randomUUID() per customer per pod.
            // Pod 1: customer "cust_123" → UUID_pod1_123 → ch_A ($99)
            // Pod 2: customer "cust_123" → UUID_pod2_123 → ch_B ($99)
            // Pod 3: customer "cust_123" → UUID_pod3_123 → ch_C ($99)
            val idempotencyKey = UUID.randomUUID().toString()
            stripeService.createCharge(idempotencyKey, ChargeRequest(
                amount = customer.monthlyAmountCents,
                currency = "usd",
                customer = customer.id
            )).execute()
        }

        billingRepository.markCompleted(billingPeriod)
    }
}

The failure scenario: the monthly billing job fires at midnight UTC on September 1. All three Kubernetes pods call runBilling("2026-09") within milliseconds of each other. All three query hasCompletedForPeriod("2026-09"). The September 2026 row does not exist in the database. All three return false. All three proceed to list the 500 active customers. All three iterate the customer list and call UUID.randomUUID() per customer — independently, producing three different UUIDs per customer. Pod 1 generates UUID_pod1_123 for customer 123 and creates ch_A. Pod 2 generates UUID_pod2_123 for customer 123 and creates ch_B. Pod 3 generates UUID_pod3_123 for customer 123 and creates ch_C. 1,500 charges are created for 500 customers. All three pods call markCompleted("2026-09"); the last write wins. The next hasCompletedForPeriod check returns true. But the 1,500 charges are already in Stripe’s ledger.

This failure is not unique to OkHttp or Retrofit — it is the standard per-JVM-scheduler failure mode that applies to any framework without a built-in distributed scheduler. It is particularly common in OkHttp/Retrofit applications because Retrofit is a lightweight HTTP client library, not an application framework. There is no Retrofit-native scheduler, no lifecycle management, and no distributed coordination primitive. Developers add scheduling themselves, and often add it in the most straightforward way available — a plain ScheduledExecutorService — without thinking through the Kubernetes multi-replica implications.

The subtler variant: fixedRateTimer() in Kotlin started inside the application’s main() function — all replicas start timers at application init — replicas:3 deployment means three independent timers for the same billing period

Kotlin’s fixedRateTimer() standard library function is a thin wrapper around java.util.Timer that creates a daemon thread at invocation time. It is often called inside main() or inside a Spring Boot CommandLineRunner as a convenient way to run a periodic task. In a Kubernetes Deployment with replicas: 3, all three pods execute main() at startup and create three independent timers. The same TOCTOU race on hasCompletedForPeriod() applies:

// main.kt — UNSAFE: fixedRateTimer started in main(), runs on every replica.
fun main() {
    val app = startApplication()  // starts Retrofit, repositories, etc.

    // UNSAFE: runs on every pod — three independent timers for 500 customers.
    fixedRateTimer(name = "monthly-billing", period = 30L * 24 * 60 * 60 * 1000) {
        val billingPeriod = currentBillingPeriod()
        if (!app.billingRepository.hasCompletedForPeriod(billingPeriod)) {
            app.billingJob.runBilling(billingPeriod)  // UUID.randomUUID() inside runBilling()
        }
    }
}

A related variant: a Kubernetes CronJob with parallelism: 1 and concurrencyPolicy: Forbid prevents concurrent runs of the same CronJob but does not prevent the billing Deployment’s own schedulers from also running. If the CronJob and the Deployment’s scheduler both trigger billing for the same period — one via a direct HTTP call to the billing endpoint, the other via its internal timer — both paths compute UUID.randomUUID() independently and create ch_A and ch_B.

The fix for failure mode 3

The solution requires coordination at a scope that spans all Kubernetes pods: a distributed lock acquired before the billing loop begins, and a pre-flight database insert that makes the billing claim durable before any Stripe call is issued:

// Safe: pg_try_advisory_lock() as cross-pod distributed mutex.
// Only the pod that acquires the lock runs the billing loop.
// All other pods skip billing for this period.

import org.springframework.jdbc.core.JdbcTemplate

class MonthlyBillingJob(
    private val jdbc: JdbcTemplate,
    private val billingRepository: BillingRepository,
    private val stripeService: StripeService
) {

    // Same billing period always maps to the same lock ID on every pod.
    private fun advisoryLockKey(billingPeriod: String): Long =
        Math.abs(("okhttp-monthly-billing:$billingPeriod").hashCode()).toLong()

    fun runMonthlyBilling() {
        val billingPeriod = currentBillingPeriod()  // e.g., "2026-09"
        val lockKey = advisoryLockKey(billingPeriod)

        // pg_try_advisory_lock() is non-blocking — returns false if another pod holds it.
        val acquired = jdbc.queryForObject(
            "SELECT pg_try_advisory_lock(?)", Boolean::class.java, lockKey
        ) ?: false

        if (!acquired) {
            // Another pod acquired the lock — skip this pod's billing run.
            return
        }

        try {
            runBillingUnderLock(billingPeriod)
        } finally {
            jdbc.execute("SELECT pg_advisory_unlock($lockKey)")
        }
    }

    private fun runBillingUnderLock(billingPeriod: String) {
        // Double-check after acquiring the lock — another pod may have completed billing
        // before this pod acquired the lock.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) return

        val customers = billingRepository.listActive()
        for (customer in customers) {
            // Safe: content-hash key — same on every pod, every retry.
            val idempotencyKey = BillingService.stableKey(customer.id, billingPeriod)

            // Pre-flight per-customer INSERT — prevents double-charging this customer
            // even if two pods race past the advisory lock (belt-and-suspenders).
            val inserted = jdbc.update(
                """INSERT INTO billing_charges (customer_id, billing_period, idempotency_key)
                   VALUES (?, ?, ?) ON CONFLICT (customer_id, billing_period) DO NOTHING""",
                customer.id, billingPeriod, idempotencyKey
            )

            if (inserted == 0) {
                // Another pod already claimed this customer — skip.
                continue
            }

            val charge = stripeService.createCharge(
                idempotencyKey,
                ChargeRequest(amount = customer.monthlyAmountCents, currency = "usd", customer = customer.id)
            ).execute().body()!!

            billingRepository.recordCharge(customer.id, billingPeriod, charge.id)
        }

        billingRepository.markCompleted(billingPeriod)
    }
}

The advisory lock key is derived from a stable string — "okhttp-monthly-billing:$billingPeriod" — that evaluates identically on every pod. Only one pod can hold this lock at a time. All other pods that call pg_try_advisory_lock() while the first pod holds it receive false and return immediately. The per-customer ON CONFLICT DO NOTHING insert is a belt-and-suspenders backstop for the case where two pods both pass the lock check due to a database connection error, lock timeout, or a race between the pg_try_advisory_lock() call and the first customer insert. The UNIQUE (customer_id, billing_period) constraint ensures only one pod’s insert wins per customer. The vault key cap at expected_total × 1.10 bounds the maximum spend even if both layers are bypassed.

Patterns that reliably cause these failures in OkHttp and Retrofit

The three failure modes share a structural pattern: a value that must be stable across all retry attempts and all pod executions (UUID.randomUUID()) is computed at a scope that is smaller than the retry or coordination boundary. In FM1, the scope is the interceptor intercept() method, which is inside the OkHttp connection-pool retry boundary. In FM2, the scope is the service method call expression, which is inside the retry callback or coroutine retry loop. In FM3, the scope is the per-JVM scheduler invocation, which is inside the pod-level execution boundary with no cross-pod coordination.

Patterns that reliably produce these failure modes in OkHttp and Retrofit:

• UUID.randomUUID() or System.currentTimeMillis() inside Interceptor.intercept() registered via addInterceptor() with retryOnConnectionFailure=true (the default)
• UUID.randomUUID() inside a network interceptor registered via addNetworkInterceptor() — fires on physical retries, redirects, and auth challenges, not just logical retries
• UUID.randomUUID() as an argument to the Retrofit service interface method invocation inside an onFailure or onResponse retry callback that calls the service method again rather than using call.clone()
• UUID.randomUUID() inside a Kotlin coroutine retry lambda that calls a Retrofit suspend fun interface method on each retry iteration
• Billing logic in a ScheduledExecutorService task (from Executors.newSingleThreadScheduledExecutor(), Kotlin fixedRateTimer(), Spring @Scheduled, or CommandLineRunner) without a distributed lock across Kubernetes replicas
• A manual billing HTTP trigger endpoint retried by an external caller (CI, monitoring, load balancer health check) across multiple Kubernetes replicas without an idempotency guard on the endpoint itself

Patterns that are safe in OkHttp and Retrofit:

• stableKey(customerId, billingPeriod) computed before OkHttpClient.newCall(request).execute(); set via Request.Builder().header("Idempotency-Key", stableKey) in calling code, not inside any interceptor
• Call.clone().enqueue(callback) for async retry — clones the original Request including the stable key; does not re-evaluate any argument expressions
• Kotlin coroutine retry with stable key captured as val outside any retryOnTransient { } or retry { } lambda — the captured reference does not re-evaluate on lambda re-invocation
• pg_try_advisory_lock() acquired before the billing loop with the lock key derived from the billing period string — not from hostname, pod ID, UUID, or timestamp

Summary

The pattern connecting all three: the idempotency key must be derived from the business intent of the billing operation — customer ID, billing period, vendor namespace — not from any ephemeral runtime value that re-evaluates between interceptor invocations, service method re-calls, or concurrent pod executions. UUID.randomUUID(), System.currentTimeMillis() at request-build time, a per-interceptor-invocation UUID, and any per-pod or per-call value all produce different results each time the expression that computes them is evaluated. A key derived from sha256(customerId + ":" + billingPeriod + ":okhttp-billing")[:32] produces the same 32-character hex string on every OkHttpClient connection-pool retry, on every Call.clone() coroutine re-invocation, and on every Kubernetes pod — stable across all three failure modes. Backing it with a PostgreSQL-level UNIQUE (customer_id, billing_period) constraint and a pg_try_advisory_lock() distributed mutex moves the deduplication guarantee out of ephemeral per-JVM state and into durable shared storage that survives SocketExceptions, JVM restarts, and multi-pod concurrent billing loops.

The vault key spend cap adds a hard financial boundary as a last line of defence: a per-billing-period vault key issued with a max_amount set to expected_total × 1.10 caps the maximum spend that any combination of interceptor retry re-invocations, service method re-calls, and multi-pod billing races can produce. Once the cap is reached, Stripe rejects further charges with a 402 Payment Required on the vault key, containing the blast radius to a known maximum regardless of how many OkHttp connection-pool retries, Call.clone() invocations, or concurrent billing pods are in flight.