AWS Glue Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

AWS Glue is a serverless data integration service that runs ETL jobs in a managed Apache Spark environment on dynamically provisioned workers. Teams use it for billing pipelines when the input is usage data partitioned in S3 or a Glue Data Catalog table: the Glue job reads records, applies transformations, and calls stripe.charges.create() for each customer before writing a billing summary back to DynamoDB or Redshift. But three Glue behaviors — the MaxRetries setting that re-executes the entire job script from line 1 after a downstream write failure, the at-least-once delivery semantics of Scheduled Triggers that can fire two concurrent job runs for the same billing period, and Job Bookmark state that is not committed until clean job success — introduce Stripe billing failure modes that neither Glue's job history nor Stripe's Restricted Key feature surfaces on their own.

This post covers all three failure modes with AWS Glue job Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-job vault keys via a spend-cap proxy — that eliminates all three without restructuring your Glue pipeline.

Failure mode 1: MaxRetries re-runs the billing script from line 1 after Stripe charges succeeded but a downstream write failed

AWS Glue job runs are configured with a MaxRetries parameter (default 0; commonly set to 1 or 2 for resilience on transient failures). When a job run exits with a non-zero exit code — due to a Python exception, a Spark job failure, or an out-of-memory error — Glue creates a new job run automatically, up to MaxRetries times, starting the job script from line 1. Each retry is a new job run with a new job run ID in Glue's run history, a fresh Python process, and fresh Spark context.

The failure mode appears when the billing job has a structure where stripe.charges.create() is called before the downstream durability operation — the DynamoDB write, the S3 billing summary, the Redshift insert — and the job fails after the Stripe call succeeds but before the downstream write completes. A common pattern is a billing script that processes all customers in a loop, calls Stripe for each, accumulates the results in a list, and writes the entire list to DynamoDB at the end of the loop. If the DynamoDB write fails (throttling, connection timeout, provisioned throughput exceeded), the job exits non-zero. Glue starts a new job run. The new run calls stripe.charges.create() again for every customer in the original cohort — none of them have a DynamoDB record yet (the write failed), so a pre-flight DynamoDB check would find nothing and allow the charge to proceed. Without an idempotency key, Stripe creates a second charge for every customer. With MaxRetries: 2, the same customer can be charged three times before Glue marks the job as permanently failed.

# UNSAFE: billing script that charges all customers before writing to DynamoDB
# If the DynamoDB write fails, Glue's MaxRetries re-runs stripe.charges.create() for all customers

import boto3
import stripe

stripe.api_key = "sk_live_..."  # UNSAFE: unrestricted key, no per-job spend cap

def main():
    s3 = boto3.client("s3")
    dynamodb = boto3.resource("dynamodb")
    billing_table = dynamodb.Table("billing_records")

    # Read customer cohort from S3
    response = s3.get_object(Bucket="billing-data", Key="2026-07/customer_cohort.json")
    customers = json.loads(response["Body"].read())

    charged = []
    for customer in customers:
        # UNSAFE: no idempotency key — retry creates a second charge for each customer
        charge = stripe.Charge.create(
            amount=customer["amount_cents"],
            currency="usd",
            customer=customer["stripe_customer_id"],
            description=f"Billing for {customer['billing_period']}"
        )
        charged.append({
            "customer_id": customer["id"],
            "charge_id": charge.id,
            "amount_cents": customer["amount_cents"],
            "billing_period": customer["billing_period"]
        })

    # UNSAFE: all Stripe charges succeed above, but if this DynamoDB batch write fails
    # (throttling, connection timeout), the job exits non-zero and Glue retries from line 1.
    # All stripe.charges.create() calls above will fire again — each creating a second charge.
    with billing_table.batch_writer() as batch:
        for record in charged:
            batch.put_item(Item=record)  # if this raises, the job exits non-zero

main()

The fix is a content-hash idempotency key computed before each stripe.charges.create() call, derived from values that are stable across all job retry attempts — not from the Glue job run ID (which changes on each retry). The correct key is sha256(customer_id:amount_cents:billing_period:glue-billing)[:32]. The billing period must be an explicit parameter passed to the job, not computed from datetime.now() at script runtime — a job that fails on day 1 and is retried on day 2 (after Glue's retry delay) would compute a different billing period string if it used the current date, breaking idempotency. Additionally, write each billing record to DynamoDB immediately after the Stripe call, not at the end of the loop — this minimizes the window where charges exist without a DynamoDB record and makes the failure surface smaller on any partial retry.

# SAFE: content-hash idempotency key + immediate per-customer DynamoDB write
# + per-job vault key via Keybrake proxy

import sys
import json
import hashlib
import boto3
import stripe

from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ["billing_period", "keybrake_vault_key"])
billing_period = args["billing_period"]       # e.g. "2026-07" — passed at job invocation
vault_key      = args["keybrake_vault_key"]   # per-job vault key with daily USD cap

stripe.api_key = vault_key
stripe.api_base = "https://proxy.keybrake.com/stripe"  # spend-cap proxy

def main():
    s3 = boto3.client("s3")
    dynamodb = boto3.resource("dynamodb")
    billing_table = dynamodb.Table("billing_records")

    response = s3.get_object(Bucket="billing-data", Key=f"{billing_period}/customer_cohort.json")
    customers = json.loads(response["Body"].read())

    for customer in customers:
        customer_id   = customer["id"]
        amount_cents  = customer["amount_cents"]
        stripe_cus_id = customer["stripe_customer_id"]

        # Pre-flight: skip if already billed (protects beyond Stripe's 24-hour idempotency window)
        existing = billing_table.get_item(
            Key={"customer_id": customer_id, "billing_period": billing_period}
        ).get("Item")
        if existing:
            print(f"Skipping {customer_id}: already billed ({existing['charge_id']})")
            continue

        # Content-hash idempotency key — stable across all job retries
        raw_key = f"{customer_id}:{amount_cents}:{billing_period}:glue-billing"
        idempotency_key = hashlib.sha256(raw_key.encode()).hexdigest()[:32]

        try:
            charge = stripe.Charge.create(
                amount=amount_cents,
                currency="usd",
                customer=stripe_cus_id,
                description=f"Billing for {billing_period}",
                idempotency_key=idempotency_key  # Stripe deduplicates on this key
            )
        except stripe.error.CardError as e:
            print(f"CardError for {customer_id}: {e} — skipping (no retry)")
            continue  # permanent error — do NOT raise, do NOT let Glue retry this customer
        except stripe.error.InvalidRequestError as e:
            print(f"InvalidRequestError for {customer_id}: {e} — skipping")
            continue

        # Write to DynamoDB immediately after each charge — minimize the un-persisted window
        billing_table.put_item(Item={
            "customer_id": customer_id,
            "billing_period": billing_period,
            "charge_id": charge.id,
            "amount_cents": amount_cents,
            "idempotency_key": idempotency_key,
            "charged_at": charge.created
        })

main()

With CardError and InvalidRequestError caught and handled with continue rather than raise, permanent per-customer failures do not exit the job non-zero — only transient infrastructure failures (DynamoDB throttling, network errors) will trigger Glue's retry mechanism. When the job retries, the pre-flight DynamoDB check skips customers already written, and the idempotency key deduplicates Stripe calls for customers charged but not yet written before the failure.

Failure mode 2: a Glue Scheduled Trigger fires two concurrent job runs for the same billing period

AWS Glue Scheduled Triggers use Amazon EventBridge Scheduler under the hood. EventBridge Scheduler provides at-least-once delivery semantics for scheduled invocations: in rare cases involving scheduler service failures, state machine anomalies, or cross-AZ failover during schedule execution, the same schedule window can produce two trigger events in rapid succession. Both events start separate Glue job runs with separate run IDs, separate Spark contexts, and separate DynamoDB write windows. Glue's job concurrency controls are set at the job level via MaxConcurrentRuns — the default is 1, which should prevent concurrent runs, but this protection has gaps.

The first gap is the trigger delay. Glue does not start the new job run atomically with the trigger event — there is a lag of several seconds between the EventBridge event and the Glue job run reaching RUNNING status. If two trigger events arrive within this window (both arrive before the first run reaches RUNNING), Glue's MaxConcurrentRuns check may not yet see the first run as active when the second trigger event is processed, and both runs proceed to RUNNING. The second gap is manual re-triggers: an engineer who sees a job in WAITING or STARTING state in the Glue console may click "Run now" thinking the scheduled run failed, starting a second run that overlaps the scheduled run by seconds or minutes before the scheduled run's status updates in the UI.

The failure mode is identical in both cases: two Glue job runs execute the billing script concurrently with the same billing_period parameter. Both runs read the same customer cohort from S3 (S3 GET operations are independent reads — no locking). Both runs call stripe.charges.create() for each customer. Both runs write to DynamoDB. Both runs succeed. Glue's run history shows two SUCCEEDED runs for the same billing period, no error, and no duplicate signal — the only indication that something went wrong is in the Stripe dashboard or a billing reconciliation query.

# The trigger duplication sequence

# T=0:   EventBridge Scheduler fires for the 2026-07-01 schedule window
# T=0+ε: Second EventBridge event fires (scheduler at-least-once duplication or engineer re-trigger)
# T=3s:  First Glue job run starts — status: STARTING
# T=4s:  Second Glue job run starts — MaxConcurrentRuns check passes (first run not yet RUNNING)
# T=8s:  Both runs reach RUNNING status — both read same customer cohort from S3
# T=12s: Run 1 calls stripe.charges.create() for customer cus_abc → charge ch_A ($99.00)
# T=12s: Run 2 calls stripe.charges.create() for customer cus_abc → charge ch_B ($99.00)
# T=15s: Both runs write to DynamoDB (last-write-wins; DynamoDB shows charge ch_B or ch_A)
# T=20s: Both runs exit 0 — Glue marks both SUCCEEDED

# Customer cus_abc was charged $198.00 for a $99.00 billing period.
# Glue run history: two SUCCEEDED runs — no error, no duplicate flag.
# DynamoDB billing_records: one row (last write wins) — no duplicate signal.
# Stripe: two charges — visible only in payment dashboard or via stripe.charges.list().

The fix is a distributed billing lock at the DynamoDB level using a conditional write — the same pattern that works for AWS Batch at-least-once EventBridge delivery. Before the billing loop begins, each Glue job run attempts a DynamoDB PutItem with a ConditionExpression="attribute_not_exists(billing_period)" on a dedicated billing_locks table keyed by billing_period. Only one job run can succeed this write — DynamoDB's conditional write is atomic at the table level. The run that loses the condition check detects this, logs the conflict, and exits cleanly (exit 0, no retry). The run that wins the lock proceeds to the billing loop.

# Distributed billing period lock via DynamoDB conditional write

import sys
import boto3
from botocore.exceptions import ClientError
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ["billing_period", "keybrake_vault_key"])
billing_period = args["billing_period"]
vault_key      = args["keybrake_vault_key"]

dynamodb = boto3.resource("dynamodb")
lock_table = dynamodb.Table("billing_locks")

def acquire_billing_lock(billing_period: str, job_run_id: str) -> bool:
    try:
        lock_table.put_item(
            Item={
                "billing_period": billing_period,
                "job_run_id": job_run_id,
                "locked_at": int(__import__("time").time()),
                "status": "in_progress"
            },
            ConditionExpression="attribute_not_exists(billing_period)"
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False  # another run already holds the lock for this period
        raise  # unexpected error — let Glue retry

def main():
    # Glue job run ID is available as a sys.argv parameter
    glue_run_id = getResolvedOptions(sys.argv, ["JOB_RUN_ID"])["JOB_RUN_ID"]

    acquired = acquire_billing_lock(billing_period, glue_run_id)
    if not acquired:
        existing = lock_table.get_item(Key={"billing_period": billing_period}).get("Item", {})
        print(f"Billing lock for {billing_period} held by run {existing.get('job_run_id')} — exiting cleanly")
        sys.exit(0)  # not an error — just a duplicate trigger; exit 0 so Glue marks SUCCEEDED

    print(f"Acquired billing lock for {billing_period} — proceeding with charges")
    # ... billing loop with idempotency keys (see failure mode 1) ...

main()

The content-hash idempotency key from failure mode 1 acts as a second safety net for the race between lock acquisition and actual Stripe calls: if two runs both pass the lock check in the narrow window before either commits (a near-impossible race given DynamoDB's atomic conditional writes, but included for completeness), Stripe deduplicates both calls on the same idempotency key and returns the existing charge. The Keybrake proxy's audit log surfaces the duplicate lock-acquisition attempt before the next billing reconciliation run.

Failure mode 3: Job Bookmark state not flushed on partial failure causes the next run to re-process already-charged S3 records

AWS Glue Job Bookmarks track which input files have been processed in previous job runs, enabling incremental processing without reprocessing already-handled data. When a Glue job with bookmarks enabled reads from S3, Glue records the ETag and last-modified timestamp of every S3 object it reads. On the next job run, Glue's bookmark state causes it to skip objects seen in previous runs and read only new files. The bookmark state is committed to Glue's internal bookmark store only on clean job success — a job that exits non-zero does not advance the bookmark.

The failure mode has a specific sequence. A monthly billing Glue job with bookmarks enabled reads S3 objects from s3://billing-data/2026-07/ — one object per customer. The job calls stripe.charges.create() for each customer (successfully), then writes billing records to Redshift (via a JDBC connection). The Redshift write fails on a connection timeout. The job exits non-zero. Glue marks the job run as FAILED. The bookmark state is not advanced — Glue still considers all the S3 objects from 2026-07/ as unprocessed.

On the next job run — whether triggered automatically by MaxRetries, manually by an engineer, or by the next scheduled trigger — Glue reads the same S3 objects again, because the bookmark still reports them as unseen. The billing script calls stripe.charges.create() for every customer again. Without an idempotency key, Stripe creates new charges. The Redshift write may succeed this time, writing one set of billing records. But the Stripe dashboard shows two charges per customer — one from the failed run and one from the successful run — and the Redshift billing table reflects only the charges from the second run, making the first set of charges invisible to billing reconciliation queries.

# The Job Bookmark failure sequence

# Run 1 (FAILED):
#   - Glue reads 500 S3 files from s3://billing-data/2026-07/ (bookmark: unseen)
#   - stripe.charges.create() called for all 500 customers — all SUCCEED
#   - Redshift JDBC write fails (connection timeout on large batch insert)
#   - Job exits non-zero → bookmark NOT advanced (all 500 files still "unseen")

# Run 2 (SUCCEEDED — MaxRetries or manual re-trigger):
#   - Glue bookmark: 2026-07/ files are STILL unseen (bookmark not advanced in run 1)
#   - Glue reads same 500 S3 files again
#   - stripe.charges.create() called for all 500 customers — all SUCCEED again
#   - Each customer charged TWICE (once in run 1, once in run 2)
#   - Redshift write succeeds — billing table shows 500 records (from run 2 only)
#   - Glue bookmark advances — 2026-07/ files marked as processed

# Result: 500 customers double-charged. Stripe has 1,000 charges.
# Redshift billing table has 500 records — only from run 2 charges.
# The 500 charges from run 1 are invisible to billing reconciliation.

The fix combines the content-hash idempotency key from failure mode 1 with a pre-flight DynamoDB check that operates at the individual customer level, not at the job level. The pre-flight check is the critical mechanism for the Job Bookmark scenario specifically: when run 2 begins and processes the same S3 files, the pre-flight check queries DynamoDB for an existing billing record for each customer and billing period. For customers where run 1 charged Stripe successfully before the Redshift failure, no DynamoDB record exists — the DynamoDB write was also part of the failed transaction. But with the idempotency key, Stripe returns the existing charge object from run 1 (within the 24-hour idempotency window) rather than creating a new charge. The idempotency key is what prevents the re-charge; the DynamoDB pre-flight protects beyond the 24-hour window.

# Write to DynamoDB immediately after each Stripe charge — before the Redshift batch write.
# This creates the pre-flight record that protects future runs from the Job Bookmark re-read.

# In the billing loop (from failure mode 1 fix):
charge = stripe.Charge.create(
    amount=amount_cents,
    currency="usd",
    customer=stripe_cus_id,
    description=f"Billing for {billing_period}",
    idempotency_key=idempotency_key
)

# Write billing record to DynamoDB IMMEDIATELY — not at end of loop
# This record survives a subsequent Redshift failure
billing_table.put_item(Item={
    "customer_id": customer_id,
    "billing_period": billing_period,
    "charge_id": charge.id,
    "amount_cents": amount_cents,
    "idempotency_key": idempotency_key,
    "charged_at": charge.created
})

# THEN write to Redshift (in a separate batch after all DynamoDB writes complete)
# If this fails and the job retries, the DynamoDB pre-flight skips already-billed customers

An additional safeguard for the Job Bookmark scenario: avoid using Glue's built-in bookmark as the only protection against re-processing billing data. The bookmark was designed for ETL pipelines where re-processing a file is harmless — a transform that produces the same output for the same input is idempotent by definition. Billing pipelines are not idempotent by default. Treat the Glue bookmark as a convenience feature for reducing read costs, not as a billing safety mechanism. The DynamoDB pre-flight check is the reliable gate.

The two-layer governance pattern for all three failure modes

Each failure mode above has a specific structural fix in the Glue job code. But the same two-layer pattern closes all three simultaneously and provides defense in depth for failure modes you haven't encountered yet.

Layer 1: content-hash idempotency keys. Before every stripe.charges.create() call, compute sha256(customer_id:amount_cents:billing_period:glue-billing)[:32] and pass it as the idempotency_key parameter. The key must be derived from the billing inputs — not from the Glue job run ID (changes on retry), the current timestamp (changes between runs), or any other value that is not stable across all execution paths that might re-invoke the charge. Pass billing_period as an explicit job parameter via the Glue job arguments, not computed at script runtime.

Layer 2: per-job vault keys via a spend-cap proxy. Instead of reading the Stripe secret key from Secrets Manager at job startup (shared across all customer charges in the run), issue a vault key from Keybrake before the billing loop — scoped to the total expected billing amount for the job run — and set it as the stripe.api_key with stripe.api_base pointed at the Keybrake proxy. Pass the vault key as a Glue job argument (via the --keybrake_vault_key parameter) so it is included in Glue's job run history and can be correlated with audit log entries. The proxy enforces the per-run USD cap, forwards the real Stripe key server-side, and logs every charge with its idempotency key, job run ID, and HTTP status — giving you a queryable audit trail across all three failure modes that Glue's job run history does not provide.

AWS Glue-specific audit gaps that make Stripe billing failures hard to detect

AWS Glue's job run history records start time, end time, DPU-hours consumed, and exit status for each job run. It does not record application-level events within the run — which customers were charged, which Stripe charge IDs were created, or whether a given charge was a retry of a previous charge. The audit gap is meaningful for billing investigations: when a customer reports a double charge, the Glue job run history shows the run succeeded, and you need to cross-reference Stripe's charge list with your DynamoDB billing table and CloudWatch logs to reconstruct the failure sequence.

CloudWatch Logs captures stdout from your Glue job (via print() statements), but log retention is typically 7 to 30 days. A double charge discovered from a monthly billing complaint will often be outside the CloudWatch retention window by the time it is investigated. An external audit table in DynamoDB — written at the time of each Stripe call, with the charge ID, idempotency key, Glue job run ID, and timestamp — is the only reliable post-incident record. Writing this record immediately after the Stripe call (not at the end of the billing loop) ensures the record exists even when the job later fails.

Glue's kill-switch for in-flight billing is coarse: you can cancel a job run from the Glue console or via StopJobRun, but in-flight Stripe calls in progress on Spark executor threads at the moment of cancellation are not interrupted — they complete before the executor acknowledges the stop signal. Cancelling a Glue job run that has processed 400 of 500 customers stops the remaining 100 customer charges from starting but does not prevent the 5 currently executing across threads from completing. A Keybrake vault key revocation takes effect in under a second at the proxy layer — all in-flight requests against that vault key receive a 401 immediately, regardless of what the Glue Spark executors are doing.

FAQ

Does using the Glue job run ID as the idempotency key work?

No. The Glue job run ID changes on each retry — jr_abc123 becomes jr_def456 on the first retry. An idempotency key derived from the run ID will be different on the retry, causing Stripe to treat it as a new charge request rather than a duplicate. The idempotency key must be derived from the billing inputs — customer ID, amount, billing period — which are stable across all retry attempts. The Glue job run ID is useful as a label in the DynamoDB billing record (for post-incident investigation), but not as the idempotency key itself.

Can I set MaxConcurrentRuns to 1 to prevent the trigger duplication failure mode?

MaxConcurrentRuns: 1 is the correct setting for billing jobs and prevents the obvious case of two runs executing simultaneously when Glue's concurrency check evaluates correctly. But it does not protect against the race window between trigger event arrival and the job run reaching RUNNING status, or against manual re-triggers from the Glue console during that window. Set MaxConcurrentRuns: 1 as a first defense, but also implement the DynamoDB conditional write billing lock — it is the only mechanism that is atomic at the billing-period level regardless of Glue's internal job state machine.

How do Glue Job Bookmarks interact with the pre-flight DynamoDB check?

They operate at different levels. The Glue Job Bookmark determines which S3 files are read by the job on each run. The DynamoDB pre-flight check determines whether a charge is executed for a given customer within a given billing period, regardless of where the customer data came from. If a bookmark failure causes the same S3 files to be re-read, the DynamoDB check catches customers already billed and skips them before any Stripe call is made. The bookmark reduces read costs and reduces the volume of data processed; the DynamoDB check prevents duplicate charges when the bookmark fails to advance.

Should I issue the Keybrake vault key inside the Glue job or from a Lambda that triggers the job?

Issue it from a Lambda (or Step Functions task) that triggers the Glue job. This separates the key issuance from the billing logic: the Lambda calls the Keybrake API, receives the vault key, starts the Glue job run via glue.start_job_run(Arguments={"--keybrake_vault_key": vault_key, "--billing_period": billing_period}), and records the vault key and job run ID together in DynamoDB for audit correlation. Issuing the vault key inside the Glue job script means the key creation is part of the billed computation and is harder to correlate with job run IDs in the audit log. The Lambda trigger pattern also means the vault key TTL can be set to match the expected job duration — if the vault key is issued with a 2-hour TTL and the job runs longer than 2 hours due to worker starvation or Spark job rescheduling, Stripe calls near the end of the job will be blocked by the proxy, which is the correct behavior for a billing run that exceeded its expected duration.

Does AWS Glue's built-in data quality checks catch double-billing?

AWS Glue Data Quality (DQDL rules evaluated via the EvaluateDataQuality transform) checks the shape and integrity of data flowing through the Glue job — null counts, uniqueness constraints, value ranges. It can detect if the output DynamoDB table has duplicate (customer_id, billing_period) rows after the job completes. It cannot detect duplicate Stripe charges, because the Stripe charge data is not a Glue data source unless you explicitly pull it into the pipeline. The correct tool for detecting Stripe-level duplicates is an audit query against your billing DynamoDB table crossed with stripe.charges.list() output — or the Keybrake audit log, which records every proxied charge with its idempotency key and job run ID.