Play Framework and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Play Framework’s Akka-backed concurrency model and Future-based async APIs make retry feel natural to compose — and make three distinct idempotency failures easy to introduce when wiring Stripe billing. Where the idempotency key is computed relative to the retry boundary is what determines whether a transient Stripe error produces a safe retry or a duplicate charge.

This post covers three Play Framework-specific failure modes with Scala code: akka.pattern.retry re-invoking the () => Future[T] supplier on each attempt; Future.recoverWith chaining a new billing call with a fresh UUID.randomUUID() when the initial Future fails; and Akka Scheduler’s scheduleAtFixedRate running independently on every Kubernetes replica with no cross-pod coordination. Each failure mode includes a subtler variant, the exact timing of how the duplicate charge is created, a content-hash fix, and a pre-flight spend-cap vault key as a hard financial backstop. For the reactive retry failure mode in Spring’s non-blocking layer, see the Spring WebFlux and Stripe Integration post. For the Akka Streams and Akka HTTP retry failure modes, see the Akka Streams and Akka HTTP posts.

Failure mode 1: akka.pattern.retry re-invokes the () => Future[T] supplier on every retry attempt — UUID.randomUUID() inside the supplier evaluates fresh per invocation — initial attempt creates ch_A before a connection timeout — first retry creates ch_B

akka.pattern.retry accepts a supplier function () => Future[T] and re-invokes it on each retry attempt. The supplier is a plain Scala function literal — a closure — and every expression inside the closure executes each time the closure is called. UUID.randomUUID() inside the supplier body is an ordinary method call that evaluates a new random value on every invocation. There is no memoization, no call-once guarantee, and no signal in the API that the supplier will be called multiple times:

// BillingService.scala
// UNSAFE: akka.pattern.retry re-invokes the supplier on every retry attempt.
// UUID.randomUUID() inside the supplier evaluates a fresh value on every call.
// Attempt 0 (initial): idempotencyKey = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
// Attempt 1 (first retry): idempotencyKey = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
// Attempt 2 (second retry): idempotencyKey = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"

import akka.actor.ActorSystem
import akka.pattern.retry
import com.stripe.model.Charge
import com.stripe.net.RequestOptions
import scala.concurrent.{ExecutionContext, Future}
import scala.concurrent.duration._
import java.util.{HashMap, UUID}

class BillingService(implicit system: ActorSystem, ec: ExecutionContext) {

  def chargeCustomer(customerId: String, billingPeriod: String, amountCents: Long): Future[Charge] = {
    retry(
      attempt = () => {
        // UNSAFE: computed inside the supplier.
        // akka.pattern.retry calls this function body on every retry attempt.
        // Each invocation of attempt() calls UUID.randomUUID() independently.
        val idempotencyKey = UUID.randomUUID().toString

        val options = RequestOptions.builder()
          .setIdempotencyKey(idempotencyKey)
          .build()

        val params = new HashMap[String, Object]()
        params.put("amount", Long.box(amountCents))
        params.put("currency", "usd")
        params.put("customer", customerId)
        params.put("description", s"Subscription $billingPeriod")

        Future(Charge.create(params, options))
      },
      attempts = 3,
      minBackoff = 1.second,
      maxBackoff = 10.seconds,
      randomFactor = 0.2
    )
  }
}

The failure scenario: a billing cron or agent invocation calls billingService.chargeCustomer("cust_123", "2026-09", 9900L). akka.pattern.retry calls attempt() for the first time. The supplier body executes. UUID.randomUUID() returns "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e". The Stripe Java SDK sends POST /v1/charges with Idempotency-Key: 3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e to Stripe’s API.

Stripe receives the request. The card is authorized. The charge object ch_A is committed to Stripe’s ledger. Before Stripe’s API server finishes writing the HTTP response body, a transient connection interruption occurs — a TCP connection reset, a read timeout on the Play application side, or an upstream load balancer closing the connection. The Stripe Java SDK catches the resulting ApiConnectionException and propagates it as a failed Future. akka.pattern.retry catches the failed Future, applies the backoff delay, and calls attempt() again for the first retry.

The supplier body re-executes from the beginning. UUID.randomUUID() evaluates again and returns "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d". A new RequestOptions object is built with this new key. The SDK sends POST /v1/charges with Idempotency-Key: b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d.

Stripe has ch_A committed against "3f7a9b2c...". The new request carries a key Stripe has never seen. Stripe processes it as a new charge request and creates ch_B. Customer 123 is charged $99 twice for September 2026.

This failure is invisible at code review because akka.pattern.retry is explicitly designed to improve resilience, and adding it to a billing call looks like a straightforward reliability improvement. The UUID.randomUUID() call is at the top of the supplier body, not in a visible retry callback or interceptor, so there’s no obvious signal that it will re-evaluate. Integration tests typically stub the Stripe SDK and assert on the happy-path charge response without verifying idempotency key uniqueness across supplier invocations.

The subtler variant: Play WS client with UUID.randomUUID() in the header-setting expression inside the supplier — evaluates fresh per supplier invocation — each retry sends a different Idempotency-Key to Stripe

Teams calling Stripe via Play’s WSClient rather than the Stripe Java SDK directly face the same issue in a slightly different surface. The header-setting expression in a ws.url(url).withHttpHeaders(...).post(body) call evaluates when the withHttpHeaders call executes — which is inside the supplier body — so it re-evaluates on every retry:

// UNSAFE: withHttpHeaders evaluates UUID.randomUUID() at call time.
// The call is inside the supplier body, so it re-evaluates on every retry invocation.
// Attempt 0: Idempotency-Key: UUID_0 → Stripe creates ch_A before timeout.
// Attempt 1: withHttpHeaders call re-executes → UUID_1 → ch_B.

import play.api.libs.ws.WSClient
import scala.concurrent.{ExecutionContext, Future}
import java.util.UUID

class WsBillingService(ws: WSClient, stripeBaseUrl: String)
                      (implicit ec: ExecutionContext) {

  def chargeViaWs(customerId: String, billingPeriod: String, amountCents: Long): Future[String] = {
    retry(
      attempt = () => {
        // UNSAFE: UUID.randomUUID() is the argument to withHttpHeaders.
        // It is evaluated when withHttpHeaders executes, which is inside this supplier.
        // Every retry invocation of this supplier re-executes withHttpHeaders
        // with a fresh UUID.randomUUID() value.
        ws.url(s"$stripeBaseUrl/v1/charges")
          .withHttpHeaders(
            "Authorization"   -> s"Bearer $stripeSecretKey",
            "Idempotency-Key" -> UUID.randomUUID().toString,
            "Content-Type"    -> "application/x-www-form-urlencoded"
          )
          .post(Map(
            "amount"      -> Seq(amountCents.toString),
            "currency"    -> Seq("usd"),
            "customer"    -> Seq(customerId),
            "description" -> Seq(s"Subscription $billingPeriod")
          ))
          .map { response =>
            (response.json \ "id").as[String]
          }
      },
      attempts = 3,
      minBackoff = 1.second,
      maxBackoff = 10.seconds,
      randomFactor = 0.2
    )
  }
}

The failure is identical to the Stripe Java SDK case: the UUID expression is inside the supplier closure, so it evaluates fresh on every retry invocation. Whether the idempotency key is set via RequestOptions or via withHttpHeaders makes no difference — what matters is where in the code the UUID.randomUUID() call appears relative to the retry boundary. Any expression inside the supplier closure re-evaluates on every supplier invocation.

The fix for failure mode 1

The idempotency key must be computed before the akka.pattern.retry call, outside the supplier closure. A deterministic content-hash key derived from the billing intent’s stable inputs produces the same value on every supplier invocation:

// SAFE: stableKey is computed before retry() is called, outside the supplier closure.
// akka.pattern.retry re-invokes the supplier, but the closure captures stableKey
// as a stable val — no re-evaluation on retry, same Idempotency-Key on every attempt.

import java.security.MessageDigest
import java.util.UUID

def stableKey(customerId: String, billingPeriod: String): String = {
  val digest = MessageDigest.getInstance("SHA-256")
  val input  = s"$customerId:$billingPeriod:play-billing"
  val bytes  = digest.digest(input.getBytes("UTF-8"))
  bytes.map("%02x".format(_)).mkString.take(32)
  // Result for ("cust_123", "2026-09"): "a7f3d291e8b4c056..."
  // Stable: same inputs → same output on every attempt, every JVM, every restart.
  // Must NOT include: UUID.randomUUID(), System.currentTimeMillis() at call time,
  // attempt counter, thread ID, object hash, or any per-invocation value.
}

def chargeCustomer(customerId: String, billingPeriod: String, amountCents: Long): Future[Charge] = {
  // Compute once, before the retry supplier.
  // stableKey is captured as an effectively-final val in the closure.
  val idempotencyKey = stableKey(customerId, billingPeriod)

  retry(
    attempt = () => {
      // Safe: idempotencyKey is a captured val from the outer scope.
      // It does not re-evaluate when the supplier is re-invoked.
      // Every retry attempt sends the same Idempotency-Key.
      val options = RequestOptions.builder()
        .setIdempotencyKey(idempotencyKey)
        .build()

      val params = new HashMap[String, Object]()
      params.put("amount", Long.box(amountCents))
      params.put("currency", "usd")
      params.put("customer", customerId)
      params.put("description", s"Subscription $billingPeriod")

      Future(Charge.create(params, options))
    },
    attempts = 3,
    minBackoff = 1.second,
    maxBackoff = 10.seconds,
    randomFactor = 0.2
  )
}

With the content-hash key, if the initial attempt creates ch_A before a connection timeout, and akka.pattern.retry invokes the supplier again for the first retry, the SDK sends the same Idempotency-Key it sent on the first attempt. Stripe looks up its idempotency cache by key, finds the ch_A result, and returns it without creating a new charge. The retry is safe: the customer is charged exactly once regardless of how many times the supplier is invoked.

Failure mode 2: Future.recoverWith chains a new billing call with UUID.randomUUID() when the initial Future fails — the initial call already committed ch_A to Stripe’s ledger before the Future failed — the recovery’s fresh UUID causes Stripe to create ch_B

Future.recoverWith is Play’s idiomatic pattern for handling a failed Future by substituting a fallback Future. The partial function passed to recoverWith executes only when the primary Future completes with a failure. When a developer uses recoverWith to retry a failed Stripe billing call, they typically re-invoke the billing function or reconstruct the WS request inside the recovery closure. If the billing function computes UUID.randomUUID() at its entry point, that computation runs fresh on every invocation, including every recoverWith-triggered invocation:

// BillingService.scala
// UNSAFE: chargeBilling() calls UUID.randomUUID() at the top of its body.
// recoverWith invokes chargeBilling() again on failure.
// The initial call creates ch_A before a timeout.
// recoverWith fires, chargeBilling() is called again, UUID evaluates to UUID_1,
// Stripe creates ch_B — customer charged twice.

class BillingService(implicit ec: ExecutionContext) {

  def chargeBilling(customerId: String, billingPeriod: String, amountCents: Long): Future[Charge] = {
    // UNSAFE: computed inside the billing function body.
    // This function is called fresh on every recoverWith invocation.
    val idempotencyKey = UUID.randomUUID().toString

    val options = RequestOptions.builder()
      .setIdempotencyKey(idempotencyKey)
      .build()

    val params = new HashMap[String, Object]()
    params.put("amount", Long.box(amountCents))
    params.put("currency", "usd")
    params.put("customer", customerId)
    params.put("description", s"Subscription $billingPeriod")

    Future(Charge.create(params, options))
  }

  def chargeWithFallback(customerId: String, billingPeriod: String, amountCents: Long): Future[Charge] = {
    chargeBilling(customerId, billingPeriod, amountCents)
      .recoverWith {
        // UNSAFE: recoverWith fires when chargeBilling fails.
        // If chargeBilling already committed ch_A to Stripe before failing
        // (e.g., Stripe processed the charge but the response arrived corrupted,
        // or the WS read timeout fired after the HTTP response started but before
        // the body was fully received), chargeBilling() re-invoked here calls
        // UUID.randomUUID() again at its entry point and sends a new Idempotency-Key.
        // Stripe creates ch_B.
        case _: ApiConnectionException =>
          chargeBilling(customerId, billingPeriod, amountCents)
        case _: SocketTimeoutException =>
          chargeBilling(customerId, billingPeriod, amountCents)
      }
  }
}

The failure scenario: the agent calls chargeWithFallback("cust_456", "2026-09", 9900L). chargeBilling is invoked. UUID.randomUUID() at line 1 of chargeBilling returns "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e" (UUID_0). The SDK sends POST /v1/charges with Idempotency-Key: 3f7a9b2c....

Stripe authorizes the card and commits ch_A. At this point, ch_A exists in Stripe’s ledger. Stripe begins writing the HTTP response body. The Play application’s WS client has a read timeout of 20 seconds. Stripe’s response body starts arriving, but a transient network congestion event delays the tail of the response body. The WS client’s read timeout fires before the response body is fully received. The Future returned by chargeBilling completes with a SocketTimeoutException.

recoverWith matches the SocketTimeoutException case. The recovery closure invokes chargeBilling(customerId, billingPeriod, amountCents) again. chargeBilling’s body executes from its first line. UUID.randomUUID() returns "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d" (UUID_1). The SDK sends POST /v1/charges with Idempotency-Key: b8d2e4f6....

Stripe looks up its idempotency cache for "b8d2e4f6..." and finds nothing — this key has never been seen before. Stripe processes the request as a new charge intent and creates ch_B. Customer 456 is charged $99 twice for September 2026.

This failure is particularly common in Play applications that migrate from a simple map/recover chain to a recoverWith-based fallback for reliability. The recovery pattern reads as correct intent: “if the first attempt fails, try again.” The critical oversight is that “the first attempt failed” does not imply “the first attempt did nothing.” For network operations, a failure at the caller can mean success at the server.

The subtler variant: recoverWith inside a retry() loop — each recoverWith invocation calls the billing function with a new UUID — ch_B on attempt 2, ch_C on attempt 3 if all timeouts fire after Stripe commits each charge

Teams that combine recoverWith with an outer akka.pattern.retry wrapper compound the failure. Each retry invocation of the supplier fires a fresh chargeBilling call, which fires a new recoverWith chain, which fires another fresh chargeBilling call on failure:

// UNSAFE: retry() + recoverWith, both calling chargeBilling() with UUID inside.
// Attempt 0: chargeBilling() → UUID_0, ch_A created, timeout fires.
//   recoverWith → chargeBilling() again → UUID_1, ch_B created, timeout fires again.
// retry() sees failed Future from recoverWith, re-invokes supplier:
// Attempt 1: chargeBilling() → UUID_2, ch_C created, response arrives.
// Customer charged three times: ch_A, ch_B, ch_C.

def chargeWithCompoundRetry(customerId: String, billingPeriod: String, amountCents: Long): Future[Charge] = {
  retry(
    attempt = () => {
      chargeBilling(customerId, billingPeriod, amountCents)
        .recoverWith {
          case _: ApiConnectionException =>
            chargeBilling(customerId, billingPeriod, amountCents)
        }
    },
    attempts = 3,
    minBackoff = 1.second,
    maxBackoff = 5.seconds,
    randomFactor = 0.2
  )
}

With three retry attempts and one recoverWith per attempt, the worst case is six invocations of chargeBilling — each with a distinct UUID.randomUUID() — creating six separate charges if all invocations reach Stripe before any one of them produces a successful response. The compound pattern is most dangerous when Stripe is under load and timing out consistently: every timeout at the read level leaves a committed charge in Stripe’s ledger, and every retry fires another call with a new UUID.

The fix for failure mode 2

The idempotency key must be computed once and passed into every billing call along the retry chain, rather than computed inside the billing function on each invocation:

// SAFE: stableKey computed once, outside all retry and recovery boundaries.
// Passed as a parameter into chargeBilling on every invocation.
// recoverWith and akka.pattern.retry both re-call chargeBilling
// with the same stableKey — Stripe's idempotency cache handles dedup.

def chargeBillingSafe(
    customerId: String,
    billingPeriod: String,
    amountCents: Long,
    idempotencyKey: String  // caller-supplied stable key
)(implicit ec: ExecutionContext): Future[Charge] = {
  val options = RequestOptions.builder()
    .setIdempotencyKey(idempotencyKey)
    .build()

  val params = new HashMap[String, Object]()
  params.put("amount", Long.box(amountCents))
  params.put("currency", "usd")
  params.put("customer", customerId)
  params.put("description", s"Subscription $billingPeriod")

  Future(Charge.create(params, options))
}

def chargeWithStableKey(
    customerId: String,
    billingPeriod: String,
    amountCents: Long
)(implicit ec: ExecutionContext): Future[Charge] = {
  // Compute once, before any retry or recoverWith boundary.
  val stableKey = computeContentHash(customerId, billingPeriod, "play-billing")

  retry(
    attempt = () => {
      chargeBillingSafe(customerId, billingPeriod, amountCents, stableKey)
        .recoverWith {
          case _: ApiConnectionException =>
            // Safe: stableKey captured from outer scope, not re-computed.
            chargeBillingSafe(customerId, billingPeriod, amountCents, stableKey)
        }
    },
    attempts = 3,
    minBackoff = 1.second,
    maxBackoff = 5.seconds,
    randomFactor = 0.2
  )
}

def computeContentHash(customerId: String, billingPeriod: String, context: String): String = {
  val digest = MessageDigest.getInstance("SHA-256")
  val bytes  = digest.digest(s"$customerId:$billingPeriod:$context".getBytes("UTF-8"))
  bytes.map("%02x".format(_)).mkString.take(32)
}

A second layer is required for the recoverWith case specifically: a pre-flight billing-period claim in the database. Even with a stable key, if the initial call committed ch_A and the timeout fires, the recovery call will hit Stripe’s idempotency cache and return ch_A’s result without creating ch_B — this is correct. But a pre-flight INSERT ... ON CONFLICT DO NOTHING on (customer_id, billing_period) adds a database-layer guarantee that is independent of Stripe’s idempotency cache, and survives scenarios where the cache has expired (Stripe’s idempotency cache has a 24-hour TTL):

// Pre-flight billing claim: only one call per (customer_id, billing_period) proceeds to Stripe.
// Other concurrent calls find the existing row and return its cached charge ID.

def chargeWithPreflightClaim(
    customerId: String,
    billingPeriod: String,
    amountCents: Long
)(implicit ec: ExecutionContext): Future[Charge] = {
  val stableKey = computeContentHash(customerId, billingPeriod, "play-billing")

  Future {
    // INSERT ... ON CONFLICT DO NOTHING: only the first call for this
    // (customer_id, billing_period) proceeds; concurrent calls from
    // recoverWith chains, retry loops, or other pods find the existing row.
    db.run(sql"""
      INSERT INTO billing_records (customer_id, billing_period, idempotency_key, status)
      VALUES ($customerId, $billingPeriod, $stableKey, 'pending')
      ON CONFLICT (customer_id, billing_period) DO NOTHING
      RETURNING idempotency_key
    """.as[String].headOption)
  }.flatMap {
    case Some(_) =>
      // This call won the pre-flight claim — proceed to Stripe.
      retry(
        attempt = () => chargeBillingSafe(customerId, billingPeriod, amountCents, stableKey),
        attempts = 3,
        minBackoff = 1.second,
        maxBackoff = 5.seconds,
        randomFactor = 0.2
      )
    case None =>
      // Pre-flight found existing row — another call already claimed this period.
      // Return the existing charge ID from the database without calling Stripe.
      Future(lookupExistingCharge(customerId, billingPeriod))
  }
}

Failure mode 3: Akka.system.scheduler.scheduleAtFixedRate billing task runs on every Kubernetes replica independently — TOCTOU race on hasCompletedForPeriod() — all three pods fire simultaneously — distinct UUID.randomUUID() per pod per customer — ch_A, ch_B, ch_C per customer per billing period

Play Framework applications use Akka’s scheduler via actorSystem.scheduler.scheduleAtFixedRate or scheduleWithFixedDelay for recurring billing tasks. Akka’s scheduler is per-ActorSystem, and each JVM starts its own independent ActorSystem. There is no cross-pod scheduling coordination built into the scheduler. With replicas: 3 in a Kubernetes Deployment, all three pods start their ActorSystem and all three register the billing task with their local scheduler. All three fire the billing task at the same delay boundary:

// BillingModule.scala
// UNSAFE: scheduleAtFixedRate creates one independent timer per JVM.
// With replicas:3 on Kubernetes, all three pods register this task
// at application startup and all three fire it simultaneously.

import akka.actor.{ActorSystem, Cancellable}
import play.api.inject.{ApplicationLifecycle, SimpleModule, bind}
import javax.inject.{Inject, Singleton}
import scala.concurrent.duration._
import scala.concurrent.{ExecutionContext, Future}
import java.util.UUID

@Singleton
class BillingScheduler @Inject()(
    actorSystem: ActorSystem,
    billingRepo: BillingRepository,
    lifecycle: ApplicationLifecycle
)(implicit ec: ExecutionContext) {

  private val cancellable: Cancellable = actorSystem.scheduler.scheduleAtFixedRate(
    initialDelay = 30.seconds,
    interval     = 30.days
  ) { () =>
    runMonthlyBilling()
  }

  lifecycle.addStopHook { () =>
    cancellable.cancel()
    Future.successful(())
  }

  private def runMonthlyBilling(): Unit = {
    val billingPeriod = currentBillingPeriod()

    // TOCTOU: all three pods query this simultaneously.
    // None has written the billing-started record yet.
    // All three read false and all three proceed.
    if (!billingRepo.hasCompletedForPeriod(billingPeriod)) {
      billingRepo.findAllActive().foreach { customer =>
        // UNSAFE: UUID.randomUUID() per customer per pod.
        // Pod 1: UUID_A for cust_123 → ch_A
        // Pod 2: UUID_B for cust_123 → ch_B
        // Pod 3: UUID_C for cust_123 → ch_C
        val idempotencyKey = UUID.randomUUID().toString
        chargeCustomer(customer.id, billingPeriod, customer.amountCents, idempotencyKey)
      }
    }
  }
}

The failure scenario: all three pods start within seconds of each other during a Kubernetes rolling deploy or initial cluster startup. Their scheduleAtFixedRate timers are registered with a 30-second initialDelay. At the 30-second mark, all three pods fire runMonthlyBilling() concurrently. All three pods execute billingRepo.hasCompletedForPeriod(billingPeriod) against the shared PostgreSQL database. The query is a SELECT EXISTS on the billing_runs table. No pod has written a record yet. All three queries return false.

All three pods proceed to billingRepo.findAllActive(). All three query the same customer table and receive the same 500 active customers. All three call UUID.randomUUID() per customer independently. Pod 1 generates "3f7a9b2c..." for customer 123, pod 2 generates "b8d2e4f6...", pod 3 generates "1c9d3e5a...". Stripe receives three POST /v1/charges requests for customer 123 with three distinct idempotency keys, treats them as three distinct charge intents, and creates ch_A, ch_B, and ch_C. Repeated across all 500 customers: 1,500 charges where 500 were intended.

Even with content-hash keys, the multi-pod issue persists without a distributed coordination layer. Content-hash keys ensure that if pod 1’s stableKey("cust_123", "2026-09") produces "a7f3d291...", pod 2’s computation of the same function produces the same value. All three pods send the same idempotency key to Stripe. Stripe’s idempotency layer handles the concurrent requests: it processes the first request and returns a 409 for the concurrent duplicate. However, all three pods still send 500 Stripe API requests each (1,500 total), consuming rate-limit quota proportional to the replica count, and all three pods simultaneously stream the full customer list from the database three times.

The subtler variant: ClusterSingleton billing actor migrates during a rolling deploy — the new singleton starts a billing run while the old singleton’s run is still in progress — the old pod already committed ch_A for some customers before being killed — the new pod creates ch_B for those customers

Akka Cluster’s ClusterSingleton pattern correctly solves the three-pod TOCTOU race: exactly one JVM in the cluster runs the singleton actor at any time. But the handoff during a rolling deploy creates a time window where two billing runs overlap:

// ClusterSingleton migration failure scenario:
//
// t=0:   Old pod (pod A) starts billing run. hasCompletedForPeriod() = false.
//        Pod A begins charging customers 1-500. Customer 123 charged → ch_A.
// t=90s: Kubernetes terminates pod A (rolling deploy).
//        ClusterSingleton manager starts singleton on pod B.
// t=91s: Pod B's singleton starts. Calls hasCompletedForPeriod().
//        Pod A's billing run was interrupted mid-way — completion record never written.
//        hasCompletedForPeriod() returns false.
// t=91s: Pod B starts billing run from customer 1.
//        Customer 123: UUID.randomUUID() = UUID_B → ch_B (duplicate of ch_A).
//
// The ClusterSingleton solved the three-concurrent-pods problem
// but introduced a mid-run interruption problem during rolling deploys.

class BillingSingletonActor(billingRepo: BillingRepository)(implicit ec: ExecutionContext)
    extends Actor {

  override def preStart(): Unit = {
    // Fired when singleton starts on a new node after migration.
    // UNSAFE: queries hasCompletedForPeriod() without knowing
    // whether the previous singleton was mid-run when it was killed.
    val billingPeriod = currentBillingPeriod()
    if (!billingRepo.hasCompletedForPeriod(billingPeriod)) {
      self ! RunBilling(billingPeriod)
    }
  }

  override def receive: Receive = {
    case RunBilling(billingPeriod) =>
      billingRepo.findAllActive().foreach { customer =>
        val idempotencyKey = UUID.randomUUID().toString // UNSAFE
        chargeCustomer(customer.id, billingPeriod, customer.amountCents, idempotencyKey)
      }
  }
}

The ClusterSingleton does not know whether the previous singleton was mid-billing or had not started. hasCompletedForPeriod() only checks for a completion record — not a start record. The new singleton sees no completion record and starts a fresh billing run, re-charging every customer the previous singleton already processed before being killed. Content-hash keys partially mitigate this: Stripe’s idempotency cache returns ch_A for any customer already charged on the previous singleton’s run (within the 24-hour cache window). But the new singleton still makes one Stripe API call per already-charged customer before discovering the duplicate, consuming rate-limit quota and latency.

The fix for failure mode 3

Two layers are required: a distributed serialization mechanism that ensures only one pod runs the billing task per period, and a per-customer pre-flight claim that makes individual customer charges idempotent regardless of which pod runs them:

// Layer 1: PostgreSQL advisory lock — only one pod's JDBC connection holds the lock.
// Other pods call pg_try_advisory_lock() and get false — they skip billing entirely.
// Advisory lock is session-scoped: released automatically when the JDBC connection
// is returned to the HikariCP pool (or closed on pod termination).
//
// Layer 2: Per-customer pre-flight INSERT ON CONFLICT DO NOTHING.
// Even if two pods somehow both acquire the advisory lock (e.g., mid-migration overlap),
// only the first INSERT per (customer_id, billing_period) proceeds to Stripe.

private def runMonthlyBillingWithLock(): Unit = {
  val billingPeriod = currentBillingPeriod()
  val lockKey = Math.abs(s"play-monthly-billing:$billingPeriod".hashCode.toLong)

  db.withConnection { conn =>
    val stmt = conn.prepareStatement("SELECT pg_try_advisory_lock(?)")
    stmt.setLong(1, lockKey)
    val rs = stmt.executeQuery()
    rs.next()
    val acquired = rs.getBoolean(1)
    rs.close()
    stmt.close()

    if (acquired) {
      try {
        billingRepo.findAllActive().foreach { customer =>
          // Content-hash key: same value on every pod for same (customer, period).
          val stableKey = computeContentHash(customer.id, billingPeriod, "play-billing")

          // Pre-flight claim: only the first INSERT per (customer_id, billing_period) wins.
          val claimed = db.run(sql"""
            INSERT INTO billing_records (customer_id, billing_period, idempotency_key, status)
            VALUES (${customer.id}, $billingPeriod, $stableKey, 'pending')
            ON CONFLICT (customer_id, billing_period) DO NOTHING
          """.asUpdate) > 0

          if (claimed) {
            // Only the pod that successfully inserted calls Stripe.
            chargeCustomerSafe(customer.id, billingPeriod, customer.amountCents, stableKey)
          }
          // Other pods or recoverWith chains: existing row found, skip Stripe call.
        }
      } finally {
        // Release advisory lock: allow next billing period's coordinator to acquire.
        val releaseStmt = conn.prepareStatement("SELECT pg_advisory_unlock(?)")
        releaseStmt.setLong(1, lockKey)
        releaseStmt.execute()
        releaseStmt.close()
      }
    }
    // Lock not acquired: another pod is running the billing task. Skip.
  }
}

For the ClusterSingleton variant, the fix adds a per-customer INSERT ON CONFLICT DO NOTHING check inside the singleton’s billing loop so that each customer is charged at most once regardless of how many singleton lifetimes process the same billing period:

// ClusterSingleton billing with per-customer pre-flight claim.
// New singleton after migration re-runs the billing loop from customer 1,
// but ON CONFLICT DO NOTHING prevents re-charging customers the old singleton
// already processed before being killed.
// Stripe's idempotency cache is a backup (within 24h TTL);
// the database constraint is the authoritative durable guard.

class BillingSingletonActorSafe(billingRepo: BillingRepository, db: Database)
    extends Actor {

  override def receive: Receive = {
    case RunBilling(billingPeriod) =>
      billingRepo.findAllActive().foreach { customer =>
        val stableKey = computeContentHash(customer.id, billingPeriod, "play-billing")

        // Pre-flight: INSERT wins for customers not yet processed in this period.
        // If old singleton already charged customer.id, row exists → DO NOTHING → skip.
        val inserted = db.run(sql"""
          INSERT INTO billing_records (customer_id, billing_period, idempotency_key, status)
          VALUES (${customer.id}, $billingPeriod, $stableKey, 'pending')
          ON CONFLICT (customer_id, billing_period) DO NOTHING
        """.asUpdate)

        if (inserted > 0) {
          // New customer for this period — proceed to Stripe.
          chargeCustomerSafe(customer.id, billingPeriod, customer.amountCents, stableKey)
        }
        // Customer already processed — skip regardless of which pod or singleton lifetime.
      }
  }
}

Gap analysis: Play Framework billing failure modes not covered here

The three failure modes above are the most common Play-specific patterns, but Play’s Future composition model creates several related edge cases worth noting:

Play WS timeout after Stripe commits the charge but before the response body is fully received. This is the trigger for failure mode 2’s recoverWith chain and is worth making explicit. Play WS’s read timeout fires based on the time to receive the response body, not the time to receive the HTTP 200 header. Stripe writes the HTTP 200 status line and begins flushing the response body before the internal charge record is fully committed in some edge cases, or the body arrives in fragments over a slow connection. The result: the timeout fires after ch_A is created, the Play application sees a TimeoutException, and any retry pattern without a stable key or pre-flight claim creates ch_B.

Future.sequence billing fan-out with one recoverWith per customer. A common batch pattern is Future.sequence(customers.map(c => chargeCustomer(c).recoverWith { ... })). If each individual chargeCustomer call computes UUID.randomUUID() independently, and any of them fail and trigger recoverWith, the recovery call for that customer produces a new UUID and a new charge while the first call’s charge is already in Stripe’s ledger. The content-hash fix (pass the stable key as a parameter) applies to each customer’s chargeCustomer call individually.

Play’s Iteratee/Enumeratee streaming APIs (Play 2.x) retried from outside the stream. Legacy Play 2.x applications that use Iteratee/Enumeratee for streaming billing responses sometimes wrap the entire stream invocation in an external retry loop. If the billing call is inside the Enumeratee transform and computes UUID.randomUUID(), retrying the outer stream from the beginning re-invokes the transform for each customer with a fresh UUID per retry, producing the same three-pod / single-pod failure mode at the stream level.

Akka BackoffSupervisor wrapping a billing actor. akka.pattern.BackoffSupervisor restarts a failed child actor after an exponential backoff. If the billing actor computes UUID.randomUUID() inside its receive handler (rather than before the Charge.create call), and the actor crashes after committing ch_A but before persisting the charge ID, the BackoffSupervisor restarts the actor and its receive handler runs again with a new UUID. Content-hash key plus ON CONFLICT DO NOTHING are the correct fix; the actor should not store the UUID as ActorRef state that resets on crash.

Comparison: Play Framework billing failure modes

FAQ

Does akka.pattern.retry’s attempts parameter count the initial attempt or only retries?

The attempts parameter counts total attempts including the initial one. With attempts = 3, akka.pattern.retry makes up to three total supplier invocations: the initial invocation and two retries. If UUID.randomUUID() is inside the supplier, it evaluates three times producing three distinct keys. The content-hash fix produces the same key on all three invocations regardless of the attempts count.

Does Stripe’s own Java SDK retry internally? Does that interact with akka.pattern.retry?

Yes. The Stripe Java SDK has a built-in retry mechanism for transient network errors (ApiConnectionException, 429 rate limit responses, and certain 500 errors). The SDK’s internal retry uses the same Idempotency-Key set in RequestOptions across all its retry attempts — the key is fixed when RequestOptions is built. If you wrap the Stripe SDK call in akka.pattern.retry, there are two retry layers: the SDK’s internal retries (safe if you use a stable key) and the outer akka.pattern.retry supplier re-invocations (unsafe if UUID.randomUUID() is inside the supplier). The content-hash fix ensures both layers use the same stable key.

Does Future.recoverWith always indicate a server-side failure, or could the server have committed the charge?

A failed Future in Play does not indicate what happened on Stripe’s server. A SocketTimeoutException can fire because: (a) Stripe’s server never received the request (charge not committed); (b) Stripe’s server received and processed the request but the response was delayed past the client’s timeout (charge committed); (c) Stripe’s server processed the request and is writing the response, but a transient network event interrupted the body transmission (charge committed). In cases (b) and (c), a recoverWith-triggered retry with a new UUID creates a duplicate charge. The safe assumption is always that a timeout may indicate a committed charge; the safe response is always to retry with the same idempotency key.

Can I use Play’s ApplicationLifecycle hook to ensure billing runs only once per pod startup?

No. ApplicationLifecycle hooks fire on every pod startup, not on exactly one pod per cluster. They do not provide cross-pod coordination. A billing trigger in lifecycle.addStopHook (or symmetrically in a module’s constructor) fires independently on each of the three pods. The three-pod TOCTOU race applies equally to lifecycle-triggered billing and scheduler-triggered billing. The correct fix is pg_try_advisory_lock() or a similar cross-pod distributed coordination mechanism, regardless of the trigger source.

Does Akka’s ClusterSingleton fully solve the multi-pod billing problem?

ClusterSingleton solves the simultaneous-pods race: at most one pod runs the singleton actor at any given time. It does not solve the rolling-deploy overlap race: during a rolling deploy, the old singleton is terminated mid-run, and the new singleton starts on the new pod and begins processing from the beginning of the customer list. Customers the old singleton already charged before being killed receive a second charge from the new singleton unless a per-customer pre-flight claim (ON CONFLICT DO NOTHING) is in place. ClusterSingleton plus per-customer pre-flight is the correct combination; neither alone is sufficient.

What does Keybrake add on top of these code-level fixes?

Code-level fixes (content-hash keys, ON CONFLICT DO NOTHING, advisory locks) prevent logical duplicate charges by ensuring Stripe only creates one charge per intent. They do not cap the total financial exposure if billing logic has a bug, an incorrect amountCents value, or an off-by-one in the customer query. A Keybrake vault key issued per billing period with a daily_usd_cap set at expected_total × 1.10 is a hard enforcement backstop: the proxy returns a 429 before forwarding any request that would push the period’s spend above the cap. If the billing logic has a bug that would create 1,500 charges instead of 500, the cap fires after charge 550 (at 110% of the expected 500 × $99 = $49,500 total) and the remaining 950 charges never reach Stripe. The code-level fixes are correct by construction; the vault key cap is the fire extinguisher for accounting bugs that slip through.