Kubernetes CronJob Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Kubernetes CronJobs are a standard way to run periodic billing workloads on clusters that already host your AI agent infrastructure. The Job controller handles scheduling, retries, and failure reporting, making CronJobs an attractive choice for monthly subscription renewals, usage-based billing runs, and agent-driven per-customer charges. But three CronJob behaviors — the default concurrencyPolicy: Allow, the Job backoffLimit Pod replacement mechanism, and the catch-up scheduling triggered by an unset startingDeadlineSeconds — introduce Stripe billing failure modes that neither Kubernetes RBAC nor Stripe restricted keys prevent on their own.

This post covers all three failure modes with Kubernetes YAML and Python billing container code, and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that eliminates all three without restructuring your CronJob.

Failure mode 1: concurrencyPolicy: Allow fires two overlapping billing runs for the same billing period

The default concurrencyPolicy for a Kubernetes CronJob is Allow, which permits multiple Job instances for the same CronJob to run concurrently. This is correct behavior for idempotent workloads where each Job is fully independent — log rotation, cache warming, health report generation. The failure mode occurs when the billing CronJob's schedule interval is shorter than a typical billing run duration. A monthly billing run that processes 800 customers against the Stripe API can take 12–15 minutes on slow networks or under Stripe rate limiting. A CronJob scheduled with schedule: "0 0 1 * *" fires once per month, but if a billing Job from a prior month was delayed by a cluster upgrade and is still running when the next month's trigger fires, the scheduler creates a second Job instance. Both Job instances execute the billing script in parallel, each independently calling stripe.charges.create() for the same set of customers. Because neither instance knows about the other and neither uses idempotency keys, both charges succeed and both Stripe events show charge.succeeded. The customer's bank sees two charges.

The same scenario occurs during cluster control-plane failure recovery. If the kube-controller-manager restarts mid-Job, it may re-create a Job for a schedule it does not recognize as already running, depending on the controller's internal state at crash time.

# UNSAFE: default concurrencyPolicy fires overlapping billing Jobs
apiVersion: batch/v1
kind: CronJob
metadata:
  name: monthly-billing
spec:
  schedule: "0 0 1 * *"
  # concurrencyPolicy defaults to Allow — overlapping Jobs permitted
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: billing
            image: my-registry/billing:latest
            env:
            - name: STRIPE_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: stripe-credentials
                  key: secret-key
          restartPolicy: OnFailure

The fix is concurrencyPolicy: Forbid. When the next scheduled trigger fires and a Job for this CronJob is already active, the scheduler skips the new Job rather than creating a concurrent one. This is safe for billing because the goal is exactly-once execution per billing period — skipping a duplicate trigger is the correct behavior. Pair concurrencyPolicy: Forbid with content-hash idempotency keys so that the extremely rare case where two Jobs do run concurrently (during controller restart) produces at most one Stripe charge per customer.

# SAFE: concurrencyPolicy: Forbid + content-hash idempotency key
apiVersion: batch/v1
kind: CronJob
metadata:
  name: monthly-billing
spec:
  schedule: "0 0 1 * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 3600
      template:
        spec:
          initContainers:
          - name: issue-vault-key
            image: my-registry/keybrake-init:latest
            env:
            - name: KEYBRAKE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: keybrake-credentials
                  key: api-key
            - name: BILLING_PERIOD
              value: "$(date +%Y-%m)"
            volumeMounts:
            - name: vault-key
              mountPath: /vault
          containers:
          - name: billing
            image: my-registry/billing:latest
            env:
            - name: BILLING_PERIOD
              value: "$(date +%Y-%m)"
            volumeMounts:
            - name: vault-key
              mountPath: /vault
          volumes:
          - name: vault-key
            emptyDir: {}
          restartPolicy: OnFailure

Failure mode 2: Job backoffLimit creates a new billing Pod from line 1 after a downstream write failure that follows a successful Stripe charge

When a Pod exits with a non-zero exit code, the Kubernetes Job controller creates a new Pod up to backoffLimit times (default: 6). The new Pod starts the billing container from the beginning. This is the correct behavior for a billing script that has not yet called Stripe — restart and try again. The failure mode is that the billing script calls stripe.charges.create() early in execution, that call returns a successful charge object, and then a downstream step — writing the charge ID to PostgreSQL, posting to an audit webhook, updating a Redis usage ledger — raises an exception. The billing container exits with code 1. The Job controller creates a new Pod. The new Pod starts the billing script from line 1 and calls stripe.charges.create() again with no idempotency key. Stripe has no record of a previous call with matching parameters and creates a second charge. The customer is charged twice.

This pattern is amplified when billing runs process multiple customers in a loop. A partial failure after charging the first 200 of 800 customers causes the replacement Pod to start from customer 1 and re-charge all 200 before reaching the new failure point (if the failure was transient) or the same failure point (if the failure was persistent). With backoffLimit: 6, a persistent downstream failure can produce seven billing runs before the Job transitions to Failed, charging each already-processed customer up to seven times.

# billing_job.py — UNSAFE: no idempotency key, downstream failure triggers duplicate charge
import stripe
import os
import psycopg2

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

def run_billing(customers: list[dict], billing_period: str):
    conn = psycopg2.connect(os.environ["DATABASE_URL"])

    for customer in customers:
        customer_id = customer["id"]
        amount_cents = customer["amount_cents"]

        # UNSAFE: no idempotency key.
        # If conn.commit() raises below, the Job Pod exits non-zero.
        # The Job controller creates a new Pod, which starts from the top of run_billing().
        # stripe.charges.create() fires again for this customer → ch_B created.
        charge = stripe.Charge.create(
            amount=amount_cents,
            currency="usd",
            customer=customer_id,
            description=f"Billing {billing_period}",
        )

        # Downstream write: if this raises (deadlock, connection lost, disk full),
        # the pod exits → Job controller creates new Pod → ch_B for this customer.
        cur = conn.cursor()
        cur.execute(
            "INSERT INTO charges (customer_id, charge_id, billing_period, amount_cents) VALUES (%s, %s, %s, %s)",
            (customer_id, charge["id"], billing_period, amount_cents)
        )
        conn.commit()

The fix is a content-hash idempotency key derived from the billing inputs — customer ID, amount, and billing period — combined with a stable salt. This key is computed before stripe.charges.create() and is identical across every Pod restart for the same billing inputs. When the Job controller creates a new Pod after a downstream write failure, stripe.charges.create() fires with the same idempotency key, and Stripe returns the original charge object without creating a new one. Add a pre-flight query to skip customers whose charge record already exists in the database, so the replacement Pod avoids redundant Stripe calls entirely for already-processed customers.

# billing_job.py — SAFE: content-hash idempotency key + pre-flight dedup + vault key
import stripe
import os
import hashlib
import psycopg2

VAULT_KEY_PATH = "/vault/key"
KEYBRAKE_PROXY_URL = os.environ.get("KEYBRAKE_PROXY_URL", "https://proxy.keybrake.com")

def billing_idempotency_key(customer_id: str, amount_cents: int, billing_period: str) -> str:
    raw = f"{customer_id}:{amount_cents}:{billing_period}:k8s-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def run_billing(customers: list[dict], billing_period: str):
    # Read vault key written by the init container
    with open(VAULT_KEY_PATH) as f:
        vault_key = f.read().strip()

    # Point the Stripe SDK at the Keybrake proxy using the per-run vault key
    stripe.api_key = vault_key
    stripe.api_base = f"{KEYBRAKE_PROXY_URL}/stripe"

    conn = psycopg2.connect(os.environ["DATABASE_URL"])

    for customer in customers:
        customer_id = customer["id"]
        amount_cents = customer["amount_cents"]

        # Pre-flight: skip customers already billed this period.
        # On Pod restart, already-processed customers are skipped entirely.
        cur = conn.cursor()
        cur.execute(
            "SELECT charge_id FROM charges WHERE customer_id = %s AND billing_period = %s",
            (customer_id, billing_period)
        )
        if cur.fetchone():
            continue

        # Idempotency key is stable across every Pod restart for the same inputs
        idempotency_key = billing_idempotency_key(customer_id, amount_cents, billing_period)

        try:
            charge = stripe.Charge.create(
                amount=amount_cents,
                currency="usd",
                customer=customer_id,
                description=f"Billing {billing_period}",
                idempotency_key=idempotency_key,
            )
        except stripe.error.CardError:
            # Permanent: log and continue — do not exit non-zero for card declines
            print(f"Card declined for {customer_id}, skipping")
            continue
        except stripe.error.InvalidRequestError as e:
            print(f"Invalid request for {customer_id}: {e}")
            continue

        cur.execute(
            "INSERT INTO charges (customer_id, charge_id, billing_period, amount_cents) VALUES (%s, %s, %s, %s) ON CONFLICT DO NOTHING",
            (customer_id, charge["id"], billing_period, amount_cents)
        )
        conn.commit()

Failure mode 3: Unset startingDeadlineSeconds fires catch-up Jobs when the CronJob controller recovers from downtime

The Kubernetes CronJob controller tracks the number of schedule intervals that passed while it was unavailable. When startingDeadlineSeconds is not set, the controller counts all missed schedules since the last successful Job creation and fires one Job per missed schedule, up to 100, when it recovers. For a monthly billing CronJob, this behavior is catastrophic after a control-plane outage that spans two billing periods: the controller fires two catch-up Jobs simultaneously when it recovers, both processing the same customer list with the same billing period parameters.

A three-hour outage of the kube-controller-manager during a cluster upgrade fires three catch-up Jobs if the billing CronJob schedule is */60 * * * * (hourly). For a daily billing CronJob, a two-day outage fires two catch-up Jobs. Each catch-up Job runs the full billing script independently. With concurrencyPolicy: Allow (the default), all three Jobs run in parallel. With no idempotency keys, each Job's stripe.charges.create() calls succeed independently, and the customer is charged three times for three billing periods — or, if the billing period is determined dynamically from the current date, three times for the same billing period.

# UNSAFE: no startingDeadlineSeconds — catch-up Jobs fire on controller recovery
apiVersion: batch/v1
kind: CronJob
metadata:
  name: hourly-usage-billing
spec:
  schedule: "0 * * * *"
  # startingDeadlineSeconds not set — controller fires all missed Jobs on recovery.
  # A 3-hour outage fires 3 catch-up Jobs simultaneously for the 3 missed schedules.
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: billing
            image: my-registry/billing:latest
          restartPolicy: OnFailure

The fix is two-part. First, set startingDeadlineSeconds: 300 (five minutes). The controller only fires catch-up Jobs for schedules that were missed in the last 300 seconds. For a monthly billing CronJob, a 300-second window never contains more than one missed monthly schedule, so no catch-up Jobs fire for outages shorter than the billing interval. For an hourly CronJob, a 300-second window permits at most one catch-up Job (one missed 60-minute schedule fits in a 5-minute window). Second, use content-hash idempotency keys so that the catch-up Job for an already-completed billing period produces no new Stripe charges. The combination of startingDeadlineSeconds and idempotency keys provides defense-in-depth: startingDeadlineSeconds drops most catch-up Jobs at the scheduler level before they consume cluster resources, and idempotency keys protect against the residual cases where two Jobs do reach Stripe.

# SAFE: startingDeadlineSeconds limits catch-up window + idempotency keys protect residual cases
apiVersion: batch/v1
kind: CronJob
metadata:
  name: hourly-usage-billing
spec:
  schedule: "0 * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 1800
      template:
        spec:
          initContainers:
          - name: issue-vault-key
            image: my-registry/keybrake-init:latest
            env:
            - name: KEYBRAKE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: keybrake-credentials
                  key: api-key
            volumeMounts:
            - name: vault-key
              mountPath: /vault
          containers:
          - name: billing
            image: my-registry/billing:latest
            env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: url
            - name: KEYBRAKE_PROXY_URL
              value: "https://proxy.keybrake.com"
            volumeMounts:
            - name: vault-key
              mountPath: /vault
          volumes:
          - name: vault-key
            emptyDir: {}
          restartPolicy: OnFailure

Approach comparison

Gap analysis: four scenarios the above does not cover

1. concurrencyPolicy: Replace terminates the running Pod with SIGTERM before the replacement starts. A billing Pod that receives SIGTERM mid-loop (after calling stripe.charges.create() for 300 of 800 customers) may exit non-zero if it does not handle SIGTERM gracefully. The backoffLimit retry creates a new Pod that starts from customer 1. Customers 1–300 are protected by idempotency keys and the pre-flight DB check. But if the pre-flight query itself is slow (full table scan on an unindexed column) and the Pod is killed at exactly the point where the idempotency key is computed but before the DB check, there is a brief window where a concurrent pod could charge the same customer. Use an indexed compound key (customer_id, billing_period) in the dedup table and ensure your graceful termination handler completes the current customer's DB write before exiting.

2. activeDeadlineSeconds kills the Pod with exit code 1 after a billing run that partially completed. When a Job exceeds activeDeadlineSeconds, Kubernetes kills all running Pods with exit code 1. If the billing Pod had already charged customers 1–400 and committed those records before the deadline, the next backoffLimit retry will correctly skip them via the pre-flight check. But if the Pod was killed between the Stripe call and the DB write for a single customer, that customer's charge exists in Stripe but not in the dedup table. The replacement Pod will not find the DB record and will call stripe.charges.create() again — protected by the idempotency key, which returns the original charge. Ensure the idempotency key matches and the DB write uses ON CONFLICT DO NOTHING so the existing charge ID is preserved.

3. A Job with parallelism > 1 dispatches multiple billing Pods within the same Job. Increasing parallelism to shard the customer list across multiple Pods requires a coordination mechanism (a work queue, a database claim table, or a Kubernetes indexed Job) to ensure each Pod processes a disjoint subset of customers. Without explicit sharding, multiple Pods will read the same customer list and process the same customers concurrently. Idempotency keys and the pre-flight check prevent duplicate charges, but both Pods will attempt to INSERT the same row, and the DB write race produces one success and one ON CONFLICT DO NOTHING skip. This is safe but inefficient — use indexed Jobs (completionMode: Indexed) to partition customers by pod index and avoid the race entirely.

4. Vault key TTL expires if the billing Job runs longer than expected. The init container issues a vault key with a TTL sized for the expected billing run duration. If the billing Pod is paused by a cluster-level resource contention event (node pressure, CPU throttling, OOM killer targeting another container) and the run extends past the vault key TTL, subsequent Stripe calls using the expired key will return 401 from the proxy. Set vault key TTL to expected_duration + 30 minutes to account for typical worst-case delays, and monitor vault key expiry events via the proxy audit log.

Enforcement tests

import hashlib
import pytest
from unittest.mock import patch, MagicMock

def billing_idempotency_key(customer_id, amount_cents, billing_period):
    raw = f"{customer_id}:{amount_cents}:{billing_period}:k8s-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def test_idempotency_key_is_deterministic_across_pod_restarts():
    key1 = billing_idempotency_key("cus_abc", 4999, "2026-07")
    key2 = billing_idempotency_key("cus_abc", 4999, "2026-07")
    assert key1 == key2
    assert len(key1) == 32

def test_idempotency_key_differs_per_customer_and_period():
    key1 = billing_idempotency_key("cus_abc", 4999, "2026-07")
    key2 = billing_idempotency_key("cus_def", 4999, "2026-07")
    key3 = billing_idempotency_key("cus_abc", 4999, "2026-08")
    assert key1 != key2
    assert key1 != key3

def test_card_decline_does_not_exit_nonzero(capsys):
    import stripe
    with patch("stripe.Charge.create") as mock_charge:
        mock_charge.side_effect = stripe.error.CardError(
            "Card declined", "card_declined", "card_declined", 402
        )
        # billing loop should catch CardError, log, continue — not raise
        try:
            raise stripe.error.CardError("Card declined", "card_declined", "card_declined", 402)
        except stripe.error.CardError:
            pass  # correct: caught, not re-raised

def test_preflight_check_skips_already_billed_customer():
    existing_charges = {("cus_abc", "2026-07"): "ch_existing"}
    def preflight(customer_id, billing_period):
        return existing_charges.get((customer_id, billing_period))
    assert preflight("cus_abc", "2026-07") == "ch_existing"
    assert preflight("cus_def", "2026-07") is None

def test_vault_key_read_from_init_container_volume(tmp_path):
    vault_file = tmp_path / "key"
    vault_file.write_text("rk_vault_test_abc123\n")
    with open(vault_file) as f:
        vault_key = f.read().strip()
    assert vault_key == "rk_vault_test_abc123"
    assert not vault_key.endswith("\n")

Frequently asked questions

Can I use the Job name or Pod name as the Stripe idempotency key?

No. The Job name includes a timestamp suffix generated by the CronJob controller (e.g., monthly-billing-28542000), which changes with every Job instance. Two concurrent Jobs for the same billing period will have different names and thus different idempotency keys — meaning Stripe treats them as independent requests and creates two charges. Use a content-hash derived from the billing inputs themselves, which is stable across any number of Job or Pod restarts for the same period and customer.

What happens if the init container fails to issue the vault key?

The main billing container never starts. Kubernetes considers the Pod failed (init containers must complete successfully before main containers run), the Job controller increments the failure count, and a new Pod is created after the backoffLimit delay. The init container will retry the vault key issuance on the new Pod. This is safe — no billing has occurred at this point, so retrying is correct. Set a short activeDeadlineSeconds on the Job so that persistent init container failures do not loop indefinitely.

How does this work with parallelism > 1 (multiple billing Pods in one Job)?

Each parallel Pod needs its own vault key. Use a Kubernetes indexed Job (completionMode: Indexed) so each Pod receives a unique JOB_COMPLETION_INDEX environment variable. In the init container, use the index to partition the customer list and issue a vault key scoped to that partition's total amount. Each Pod calls the init container with its own index, retrieves a unique vault key, and processes a disjoint subset of customers. The content-hash idempotency key still uses the customer ID and billing period as inputs, ensuring correctness even if partitioning logic changes between Pod restarts.

Does concurrencyPolicy: Replace protect against duplicate billing?

Not fully. concurrencyPolicy: Replace terminates the running Job and all its Pods before starting the new Job for the next schedule. The terminated Pods exit with SIGTERM (exit code 143 by default). If the billing Pod does not handle SIGTERM by completing the current customer's DB write before exiting, the replacement Pod may start from the beginning or from an inconsistent state. Use concurrencyPolicy: Forbid instead — it skips the new Job trigger entirely while an existing Job is active, making the behavior explicit and easy to monitor via CronJob status events.

What happens to vault keys when a Pod is rescheduled mid-billing?

The vault key is written to an emptyDir volume by the init container and is local to the Pod. When the Pod is evicted or killed and the Job controller creates a new Pod, the new Pod runs the init container again, which issues a new vault key. The old vault key is gone with the old Pod. This is correct: the new Pod issues a fresh vault key scoped to the remaining billing amount (estimated from the number of unprocessed customers remaining in the DB) and continues from where the pre-flight check shows it left off.

Is this compatible with Helm chart management and GitOps (ArgoCD, Flux)?

Yes. All changes are declarative YAML modifications to the CronJob spec — concurrencyPolicy: Forbid, startingDeadlineSeconds: 300, the init container definition, and the emptyDir volume. These can be expressed as Helm values overrides or Kustomize patches and synced via ArgoCD or Flux like any other Kubernetes resource. The Keybrake API key is stored as a Kubernetes Secret and referenced by secretKeyRef, keeping credentials out of your GitOps repository.