AWS Batch Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

AWS Batch is a managed service that schedules and runs containerized jobs on EC2 or Fargate. Teams reach for it when billing pipelines outgrow cron and need queue-backed execution, managed retries, and Array Job fan-out for parallel customer cohorts. But three Batch behaviors — automatic job retries that restart the container from line 1, Array Job fan-out that injects the same unrestricted Stripe key into all child containers, and EventBridge Scheduler's at-least-once delivery that submits duplicate jobs for the same billing period — introduce Stripe billing failure modes that restricted API keys and Batch's own observability do not surface on their own.

This post covers all three failure modes with AWS Batch job definition YAML and Python billing code, and the two-layer governance pattern — content-hash idempotency keys plus per-child vault keys via a spend-cap proxy — that eliminates all three without restructuring your Batch job graph.

Failure mode 1: retryStrategy.attempts re-runs the billing container from line 1 after a successful charge and a downstream write failure

AWS Batch job retries are configured in the job definition under retryStrategy.attempts. When a job container exits with a non-zero exit code, Batch automatically schedules a new container attempt — a completely fresh container process starting from the beginning of the billing script. The retry gives no indication that a prior attempt ran partial work; the new container has the same environment variables, the same billing inputs, and the same Stripe API key as the original attempt.

The failure mode is straightforward: the billing container calls stripe.charges.create() for a customer, the call returns a charge object (ch_A), and then a downstream write fails — a DynamoDB PutItem raises ProvisionedThroughputExceededException, a PostgreSQL INSERT times out, or a Redis SET returns a connection error. The container exits non-zero. Batch creates a new container attempt. The new container starts from line 1 of the billing script, reaches stripe.charges.create() for the same customer, and — without an idempotency key — creates a second charge (ch_B). With retryStrategy.attempts: 3 and a persistent downstream failure (for example, a DynamoDB table under sustained traffic), the same customer can be charged up to three times before the job transitions to FAILED.

# billing_job.py — UNSAFE: no idempotency key
import stripe
import boto3
import os

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]  # full-access key from Secrets Manager
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(os.environ["BILLING_TABLE"])

def run_billing(customer_id: str, amount_cents: int, billing_period: str):
    # UNSAFE: no idempotency key.
    # If this container exits non-zero after this call (e.g., DynamoDB write fails),
    # Batch creates a new container attempt that starts from line 1 and calls
    # stripe.charges.create() again for the same customer → ch_B, ch_C, etc.
    charge = stripe.Charge.create(
        amount=amount_cents,
        currency="usd",
        customer=customer_id,
        description=f"Billing {billing_period}",
    )

    # UNSAFE: if this write fails after the Stripe call succeeds,
    # the container exits non-zero and Batch retries from line 1.
    table.put_item(Item={
        "customer_id": customer_id,
        "billing_period": billing_period,
        "charge_id": charge["id"],
        "amount_cents": amount_cents,
    })

The fix has two parts. First, compute a content-hash idempotency key from the billing inputs — not from the job ID, attempt number, or any value that changes between retries. The idempotency key must be stable across all attempts: sha256(customer_id:amount_cents:billing_period:aws-batch-billing)[:32]. Stripe deduplicates charges with the same idempotency key within a 24-hour window, so a Batch retry that reaches stripe.charges.create() with the same key returns the original charge object rather than creating a new one. Second, use a pre-flight check before calling Stripe: query the billing table for a record with the same customer and billing period. If the record exists, skip the Stripe call entirely. The pre-flight check is the protection for retries that run more than 24 hours after the original attempt — past Stripe's idempotency window.

# billing_job.py — SAFE: content-hash idempotency key + pre-flight check + vault key
import stripe
import boto3
import os
import hashlib
import sys

KEYBRAKE_PROXY_URL = os.environ.get("KEYBRAKE_PROXY_URL", "https://proxy.keybrake.com")
stripe.api_key = os.environ["KEYBRAKE_VAULT_KEY"]  # scoped vault key, not full Stripe key
stripe.api_base = f"{KEYBRAKE_PROXY_URL}/stripe"

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(os.environ["BILLING_TABLE"])

def billing_idempotency_key(customer_id: str, amount_cents: int, billing_period: str) -> str:
    raw = f"{customer_id}:{amount_cents}:{billing_period}:aws-batch-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def run_billing(customer_id: str, amount_cents: int, billing_period: str):
    # Pre-flight: skip customers already billed this period (protects past Stripe's 24h window)
    existing = table.get_item(Key={"customer_id": customer_id, "billing_period": billing_period})
    if "Item" in existing:
        print(f"Skipping {customer_id} — already billed for {billing_period}")
        return

    idempotency_key = billing_idempotency_key(customer_id, amount_cents, billing_period)

    try:
        charge = stripe.Charge.create(
            amount=amount_cents,
            currency="usd",
            customer=customer_id,
            description=f"Billing {billing_period}",
            idempotency_key=idempotency_key,
        )
    except stripe.error.CardError as e:
        print(f"Card declined for {customer_id}: {e.user_message}")
        return  # do not exit non-zero — permanent error, not transient
    except stripe.error.InvalidRequestError as e:
        print(f"Invalid request for {customer_id}: {e}")
        return  # do not exit non-zero

    # Write billing record. If this fails and the container exits non-zero,
    # the next Batch attempt will find the record in pre-flight and skip.
    # Note: we write BEFORE exiting so partial progress is preserved.
    table.put_item(Item={
        "customer_id": customer_id,
        "billing_period": billing_period,
        "charge_id": charge["id"],
        "amount_cents": amount_cents,
    })

if __name__ == "__main__":
    customer_id = os.environ["CUSTOMER_ID"]
    amount_cents = int(os.environ["AMOUNT_CENTS"])
    billing_period = os.environ["BILLING_PERIOD"]
    run_billing(customer_id, amount_cents, billing_period)

Failure mode 2: Array Job fan-out shares one unrestricted Secrets Manager key across all child containers with no per-child spend cap

AWS Batch Array Jobs dispatch N child jobs in parallel, each identified by an index (AWS_BATCH_JOB_ARRAY_INDEX) from 0 to N−1. Teams use Array Jobs to partition a billing cohort: index 0 bills customers 0–99, index 1 bills customers 100–199, and so on. Environment variables defined in the job definition — including secrets injected from AWS Secrets Manager via secrets — are the same for all child jobs. Every child container receives the same STRIPE_SECRET_KEY from Secrets Manager, and that key is unrestricted: it can charge any amount to any customer, call any Stripe endpoint, and has no daily or per-run spend cap.

The failure mode emerges when a data error propagates uniformly across child jobs. A calculation bug that multiplies amount_cents by 100 twice (cents converted to cents again) produces an amount 100× too large. Because all N child containers receive the same unrestricted key and all execute in parallel, the calculation error reaches stripe.charges.create() in every child simultaneously. With arraySize: 500 and 100 customers per child, 50,000 customers are charged 100× the correct amount before any alert fires. There is no circuit breaker — no per-child spend cap prevents any individual child from charging beyond the expected cohort amount, and the full-access Stripe key does not track cumulative spend across all children.

# job-definition.json — UNSAFE: all children share one unrestricted key
{
  "jobDefinitionName": "monthly-billing",
  "type": "container",
  "containerProperties": {
    "image": "my-registry/billing-job:latest",
    "resourceRequirements": [
      {"type": "VCPU", "value": "1"},
      {"type": "MEMORY", "value": "2048"}
    ],
    "secrets": [
      {
        "name": "STRIPE_SECRET_KEY",
        "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:stripe/secret-key"
      }
    ],
    "environment": [
      {"name": "BILLING_TABLE", "value": "billing-charges"},
      {"name": "BILLING_PERIOD", "value": "2026-07"}
    ]
  },
  "retryStrategy": {"attempts": 2}
}

The fix is to issue per-child vault keys before dispatching the Array Job. A dispatcher Lambda (or Step Function) computes the customer partition for each Array Job index, calls the Keybrake API to issue a vault key scoped to that index's expected spend, and passes the vault key as a container environment variable override in the SubmitJob call. Each child container receives a vault key capped at cohort_spend_cents × 1.10 + 100 — 10% headroom for rounding differences plus a one-dollar buffer. A calculation error that causes one child to attempt 100× overcharging will be stopped at the proxy after the cap is reached, limiting blast radius to the child's cohort rather than the entire billing run.

# dispatcher.py — SAFE: issue per-child vault key before submitting Array Job
import boto3
import requests
import os
import json

batch = boto3.client("batch", region_name="us-east-1")
KEYBRAKE_API_KEY = os.environ["KEYBRAKE_API_KEY"]
KEYBRAKE_API_URL = "https://keybrake.com/api/v1/keys"

def issue_vault_key(cohort_spend_usd: float, billing_period: str, index: int) -> str:
    """Issue a vault key capped at cohort spend + 10% headroom."""
    cap_usd = cohort_spend_usd * 1.10 + 1.0
    resp = requests.post(
        KEYBRAKE_API_URL,
        json={
            "vendor": "stripe",
            "daily_usd_cap": cap_usd,
            "allowed_endpoints": ["/v1/charges"],
            "expires_in_seconds": 7200,  # 2 hours
            "label": f"batch-billing-{billing_period}-idx-{index}",
        },
        headers={"Authorization": f"Bearer {KEYBRAKE_API_KEY}"},
        timeout=10,
    )
    resp.raise_for_status()
    return resp.json()["vault_key"]

def submit_billing_array_job(
    billing_period: str,
    cohort_partitions: list[dict],  # [{customers: [...], spend_usd: float}]
):
    array_size = len(cohort_partitions)

    # Issue a vault key per partition before dispatching the array job
    vault_keys = []
    for idx, partition in enumerate(cohort_partitions):
        vault_key = issue_vault_key(partition["spend_usd"], billing_period, idx)
        vault_keys.append(vault_key)

    # Store vault keys in Secrets Manager or SSM Parameter Store,
    # indexed by billing_period and array index, for the child to fetch.
    # Here we pass a vault-key mapping as a JSON env var (safe for short lists).
    vault_key_map = json.dumps({str(i): k for i, k in enumerate(vault_keys)})

    batch.submit_job(
        jobName=f"monthly-billing-{billing_period}",
        jobQueue=os.environ["BATCH_JOB_QUEUE"],
        jobDefinition=os.environ["BATCH_JOB_DEFINITION"],
        arrayProperties={"size": array_size},
        containerOverrides={
            "environment": [
                {"name": "BILLING_PERIOD", "value": billing_period},
                {"name": "VAULT_KEY_MAP", "value": vault_key_map},
            ]
        },
    )

# In the child container: read vault key for this index
import os, json
vault_key_map = json.loads(os.environ["VAULT_KEY_MAP"])
my_index = os.environ["AWS_BATCH_JOB_ARRAY_INDEX"]
stripe.api_key = vault_key_map[my_index]
stripe.api_base = f"{os.environ['KEYBRAKE_PROXY_URL']}/stripe"

Failure mode 3: EventBridge Scheduler at-least-once delivery submits two concurrent Batch jobs for the same billing period

AWS EventBridge Scheduler triggers Batch jobs on a cron or rate expression by calling the SubmitJob API. EventBridge Scheduler guarantees at-least-once delivery: in rare cases, particularly when the Batch API returns a transient error or the EventBridge scheduler experiences a retry event, the same schedule trigger fires twice. Both SubmitJob API calls succeed. AWS Batch returns two distinct Job IDs. Both jobs are queued, both enter RUNNABLE, both receive compute resources, and both containers execute the billing script in parallel. Because neither job knows the other exists — Batch does not enforce uniqueness on jobName within a queue by default — both jobs reach stripe.charges.create() independently. Both calls succeed. Stripe shows two separate charge objects. The Batch dashboard shows two SUCCEEDED jobs, both named monthly-billing-2026-07, with no indication that the customer cohort was charged twice.

This failure mode is less frequent than job retries or Array Job key sharing, but it is also the least visible. A retry shows up as attempt 2 in job history. Two jobs from EventBridge look identical in the Batch console — the same job name, the same job definition, the same environment variables, both succeeded. The only signal is a Stripe API log showing two charges per customer on the same day, which may not be checked unless billing-related support tickets arrive.

# CloudFormation / CDK — EventBridge Scheduler targeting AWS Batch
# UNSAFE: at-least-once delivery can submit two SubmitJob calls for the same period.
# Both Batch jobs execute, both call stripe.charges.create() for all customers.
{
  "ScheduleExpression": "cron(0 0 1 * ? *)",  // 1st of each month at midnight UTC
  "Target": {
    "Arn": "arn:aws:batch:us-east-1:123456789:job-queue/billing-queue",
    "BatchParameters": {
      "JobDefinition": "arn:aws:batch:...:job-definition/monthly-billing:5",
      "JobName": "monthly-billing"
      // UNSAFE: static JobName — Batch allows multiple jobs with the same name
    }
  }
}

The fix combines three layers. First, use a period-specific JobName (monthly-billing-2026-07) and wrap the SubmitJob call in a Lambda that checks DynamoDB for an in-flight or completed job before submitting. If a job for this period is already SUBMITTED, PENDING, RUNNABLE, STARTING, or RUNNING, the Lambda exits without submitting a duplicate. Second, use the content-hash idempotency key in the billing container — even if two jobs do reach stripe.charges.create() simultaneously, Stripe deduplicates the charges within a 24-hour window. Third, the pre-flight DynamoDB check in the billing container is the final backstop: the second job to reach a customer finds the billing record already written by the first and skips the Stripe call.

# dispatcher_lambda.py — SAFE: pre-submit dedup check + period-specific job name
import boto3
import os
from datetime import datetime, timezone

batch = boto3.client("batch", region_name="us-east-1")
dynamodb = boto3.resource("dynamodb")
jobs_table = dynamodb.Table(os.environ["BILLING_JOBS_TABLE"])

def handler(event, context):
    billing_period = datetime.now(timezone.utc).strftime("%Y-%m")
    job_name = f"monthly-billing-{billing_period}"

    # Check DynamoDB for an in-flight or completed job for this period.
    # EventBridge at-least-once delivery may call this Lambda twice.
    existing = jobs_table.get_item(Key={"billing_period": billing_period})
    if "Item" in existing:
        status = existing["Item"].get("status", "")
        if status in ("submitted", "running", "completed"):
            print(f"Batch job for {billing_period} already {status} — skipping duplicate submit")
            return {"skipped": True, "status": status}

    # Record intent before submitting (prevents race between two Lambda invocations)
    jobs_table.put_item(Item={
        "billing_period": billing_period,
        "status": "submitted",
        "job_name": job_name,
        "submitted_at": datetime.now(timezone.utc).isoformat(),
    }, ConditionExpression="attribute_not_exists(billing_period)")
    # ConditionExpression: fails if a concurrent invocation already inserted this record.
    # On ConditionalCheckFailedException, the second Lambda exits without submitting.

    response = batch.submit_job(
        jobName=job_name,
        jobQueue=os.environ["BATCH_JOB_QUEUE"],
        jobDefinition=os.environ["BATCH_JOB_DEFINITION"],
        containerOverrides={
            "environment": [
                {"name": "BILLING_PERIOD", "value": billing_period},
                {"name": "KEYBRAKE_VAULT_KEY", "value": issue_vault_key(billing_period)},
            ]
        },
    )
    print(f"Submitted Batch job {response['jobId']} for {billing_period}")
    return {"job_id": response["jobId"]}

Approach comparison

Gap analysis: four scenarios the above does not cover

1. Batch job with retryStrategy.evaluateOnExit retrying on SPOT_INSTANCE_RECLAIMED re-runs billing from line 1 on Spot interruption. When a Batch job runs on a Spot compute environment and the Spot instance is reclaimed, Batch marks the attempt as failed with reason Host EC2 (instance-id) terminated. If evaluateOnExit is configured to retry on this reason code, Batch creates a new attempt on a fresh instance. The new container has no awareness that the prior container may have already called stripe.charges.create() before the instance was reclaimed. The content-hash idempotency key is the correct protection here — it is stable regardless of whether the prior Spot instance reached the Stripe call — but only within Stripe's 24-hour idempotency window. For billing jobs that may be delayed by Spot market conditions for longer than 24 hours, the pre-flight DynamoDB check is essential.

2. dependsOn job graph with a SEQUENTIAL dependency re-runs the downstream billing job when the upstream job retries. AWS Batch job dependencies use dependsOn with type: SEQUENTIAL. If a billing job depends on a data-preparation job and the data-preparation job fails and retries, Batch may also re-run the billing job as a downstream dependency. Whether the billing job re-runs depends on the job graph configuration — but in complex pipelines with multi-level dependencies, a retry of an upstream job can cascade downstream. Check all Batch job dependency graphs that include billing steps for retry propagation; apply the content-hash idempotency key and pre-flight check to every billing step, not just the top-level job.

3. Vault key TTL set to job timeout duration does not account for queue wait time. AWS Batch jobs can sit in RUNNABLE state for an extended period if compute resources are not immediately available — common in Spot compute environments during high-demand periods. If the vault key is issued with a TTL equal to the expected job execution time, but the job waits 45 minutes in queue before starting, the vault key may expire partway through the billing run. Subsequent Stripe calls through the proxy return 401, leaving the remaining customer cohort unbilled without a clear error. Issue vault keys with TTL set to expected_queue_wait + expected_execution_time + 30 minutes, and monitor proxy 401 rates broken down by vault key label to detect TTL undersizing before it causes partial billing runs.

4. Batch job sharing the same jobName across billing periods produces ambiguous job history when debugging duplicate charges. If the same jobName (e.g., monthly-billing) is reused across billing periods, Batch job history lists multiple jobs with the same name differing only by job ID and submit time. When investigating a duplicate charge report, an engineer filtering by job name sees multiple entries and must correlate timestamps with Stripe charge timestamps manually. Use period-specific job names (monthly-billing-2026-07) to make each billing period's job uniquely identifiable in Batch job history, CloudWatch Logs, and the DynamoDB billing jobs table. This also enables the conditional-write dedup in the dispatcher Lambda above.

Enforcement tests

import hashlib
import pytest
from unittest.mock import patch, MagicMock
import stripe

def billing_idempotency_key(customer_id, amount_cents, billing_period):
    raw = f"{customer_id}:{amount_cents}:{billing_period}:aws-batch-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def test_idempotency_key_is_stable_across_batch_attempts():
    """Same inputs on any attempt number must produce the same key."""
    key_attempt_1 = billing_idempotency_key("cus_abc", 4999, "2026-07")
    key_attempt_2 = billing_idempotency_key("cus_abc", 4999, "2026-07")
    assert key_attempt_1 == key_attempt_2
    assert len(key_attempt_1) == 32

def test_idempotency_key_differs_per_customer_and_per_period():
    key_a = billing_idempotency_key("cus_abc", 4999, "2026-07")
    key_b = billing_idempotency_key("cus_xyz", 4999, "2026-07")
    key_c = billing_idempotency_key("cus_abc", 4999, "2026-08")
    assert key_a != key_b
    assert key_a != key_c

def test_preflight_check_skips_already_billed_customer(mocker):
    """DynamoDB pre-flight must skip customers with existing billing records."""
    mock_table = MagicMock()
    mock_table.get_item.return_value = {
        "Item": {"customer_id": "cus_abc", "billing_period": "2026-07", "charge_id": "ch_xxx"}
    }
    mocker.patch("billing_job.table", mock_table)
    with patch("billing_job.stripe.Charge.create") as mock_charge:
        from billing_job import run_billing
        run_billing("cus_abc", 4999, "2026-07")
        mock_charge.assert_not_called()

def test_card_error_does_not_exit_nonzero():
    """CardError must be caught silently — not re-raised to trigger Batch retry."""
    try:
        raise stripe.error.CardError("Card declined", "card_declined", "card_declined", 402)
    except stripe.error.CardError:
        pass  # correct: caught, container continues to next customer

def test_dispatcher_lambda_skips_duplicate_eventbridge_trigger(mocker):
    """Second Lambda invocation for same billing period must not call SubmitJob."""
    mock_table = MagicMock()
    mock_table.get_item.return_value = {
        "Item": {"billing_period": "2026-07", "status": "submitted"}
    }
    mocker.patch("dispatcher_lambda.jobs_table", mock_table)
    with patch("dispatcher_lambda.batch.submit_job") as mock_submit:
        from dispatcher_lambda import handler
        result = handler({}, {})
        mock_submit.assert_not_called()
        assert result["skipped"] is True

Frequently asked questions

Can I use AWS Batch job IDs as idempotency keys instead of content hashes?

No. The AWS Batch job ID is assigned at SubmitJob time and is different for each job submission, including for retry attempts triggered by the same EventBridge event and for duplicate jobs submitted by at-least-once delivery. Within a single job's retry attempts, each attempt has the same job ID but the container receives no mechanism to read the job ID as an environment variable without a custom entrypoint that calls the Batch DescribeJobs API. Content-hash idempotency keys derived from billing inputs are stable across all submission and retry paths. See also: idempotency key patterns for agent workflows.

Does AWS Batch's native deduplication prevent duplicate jobs from EventBridge?

No. AWS Batch does not enforce job name uniqueness within a queue. You can submit 100 jobs named monthly-billing to the same queue and Batch will accept and run all 100. The uniqueness enforcement must happen in the caller — either in the Lambda dispatcher via a conditional DynamoDB write, or by checking existing job status via batch.list_jobs() with a name filter before submitting. list_jobs() filtering by name is eventually consistent, so the conditional-write pattern is more reliable for concurrent invocations.

How should vault keys be passed to Array Job child containers?

There are two practical patterns. First, the dispatcher issues all vault keys before calling SubmitJob and stores them in a DynamoDB table keyed by (billing_period, array_index). Each child container fetches its vault key from DynamoDB at startup using its AWS_BATCH_JOB_ARRAY_INDEX. This scales to thousands of children and keeps the SubmitJob payload small. Second, for small array sizes (under ~100), the dispatcher serializes the vault key map as a JSON environment variable override passed in containerOverrides.environment. Each child deserializes the map and reads its index's key. The first pattern is preferred for production because it avoids environment variable size limits and keeps keys out of CloudWatch Logs for the parent job.

Does retryStrategy.evaluateOnExit help prevent the retry double-charge?

evaluateOnExit lets you configure which exit codes or status reasons trigger a retry. You can configure it to not retry on exit code 0 (already handled — success doesn't retry), and to only retry on specific transient reasons like SPOT_INSTANCE_RECLAIMED. This reduces the number of spurious retries but does not eliminate the underlying risk: even a single retry after a successful Stripe charge and a failed downstream write creates a duplicate charge. The content-hash idempotency key and the pre-flight check are the correct fixes regardless of how many retries evaluateOnExit allows.

What happens if the Keybrake vault key expires before the Batch job reaches all customers in a large cohort?

Stripe calls through an expired vault key return HTTP 401 from the proxy. The billing container catches the 401 as a stripe.error.AuthenticationError — if not explicitly caught, it will bubble up as an unhandled exception, causing the container to exit non-zero and triggering a Batch retry with a new container. The new container will receive an environment variable pointing to the same expired vault key, so subsequent attempts will also fail. Monitor proxy 401 rates by vault key label; if a billing job's vault key is expiring before the run completes, increase the TTL in the key-issuance call. For very large cohorts with unpredictable queue wait times, issue the vault key inside the job container at startup rather than in the dispatcher — this avoids TTL decay during queue wait.

How does Keybrake's audit log help with AWS Batch billing investigations?

The Keybrake audit log records every Stripe API call forwarded through the proxy with the vault key label, timestamp, endpoint, and response status. For a Batch billing investigation — for example, a customer reports being charged twice on July 1 — you can query the audit log for all charges on that date filtered by vault key label (batch-billing-2026-07-*). If two charges appear for the same customer with the same idempotency key and both returned HTTP 200, Stripe deduplicated them (only one real charge was created). If two charges appear with different idempotency keys, the duplicate-charge failure mode occurred and the idempotency key was not stable across attempts. The audit log surface this within minutes, before your Stripe dashboard shows the customer's charge history.