dbt Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

dbt is a SQL-first transformation framework used in modern ELT pipelines to model, test, and document data transformations in a data warehouse. Teams wire it into Stripe billing in two structural patterns: dbt Python models (available since dbt 1.3) that call stripe.charges.create() directly inside Python functions executed in the warehouse's Python runtime; and downstream billing scripts that read from dbt-managed tables — usage aggregates, charge-ready queues — and call Stripe after dbt has transformed the raw data. Both patterns introduce Stripe double-charge failure modes that dbt's run history, test results, and lineage graph do not surface automatically.

This post covers three dbt-specific failure modes with dbt Python code and YAML, and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that eliminates all three without restructuring your dbt project.

Failure mode 1: dbt Python model re-run re-executes every stripe.charges.create() call

dbt 1.3 introduced Python models: Python files in the models/ directory that dbt compiles and executes in the warehouse's Python runtime (Snowpark for Snowflake, Spark for Databricks, or a local subprocess for dbt Core). Python models receive a dbt object and a session object, read upstream dbt model outputs as DataFrames, apply Python logic — including calls to external APIs — and return a DataFrame that dbt writes back to the warehouse as the model's materialization.

Teams use Python models for billing because the billing logic requires Python: calling the Stripe SDK, computing idempotency keys, handling per-customer CardError responses, writing results back to a billing table. A Python model that reads a billing_queue upstream model, iterates over customer rows, and calls stripe.charges.create() for each is a natural fit. The failure mode is what happens when that Python model fails partway through and dbt run is re-run.

dbt's execution model for Python models is atomic at the model level: a Python model either succeeds (and its results are written to the warehouse) or fails (and its results are discarded). There is no partial write — the model runs from the first line of the model() function to the last. If the model fails at row 847 of a 1,000-customer billing run — because of a database write error, a network timeout on the Stripe API, or a malformed customer record — re-running dbt run re-executes the entire Python model from the beginning. It re-reads the same billing queue, re-iterates all 1,000 customer rows, and calls stripe.charges.create() for customers 1 through 846 again — the customers who were already charged successfully in the first run.

# UNSAFE: dbt Python model that calls Stripe without idempotency keys
# Re-running the model after failure re-charges all customers processed in the first run
#
# models/billing/charge_customers.py

import stripe

def model(dbt, session):
    # Read upstream dbt model output as a DataFrame
    billing_df = dbt.ref("billing_queue")
    customers = billing_df.collect()  # materializes the DataFrame

    stripe.api_key = "sk_live_..."  # UNSAFE: unrestricted key, no spend cap

    results = []
    for row in customers:
        # UNSAFE: no idempotency key
        # If model fails after this row and is re-run, this customer is charged again
        charge = stripe.Charge.create(
            amount=row["amount_cents"],
            currency="usd",
            customer=row["stripe_customer_id"],
            description=f"Billing for {row['billing_period']}"
        )
        results.append({
            "customer_id": row["customer_id"],
            "charge_id": charge.id,
            "billing_period": row["billing_period"]
        })

    # If this write fails, dbt marks the model as failed
    # Re-running dbt re-executes from the first row — re-charging all customers
    return session.createDataFrame(results)

The fix is a content-hash idempotency key computed from stable business inputs — customer ID, amount in cents, and billing period — plus a pre-flight query to the billing results table written inside the Python model. When dbt run re-executes the Python model, the pre-flight check finds customers already billed in the previous (failed) run and skips their Stripe calls. The idempotency key provides a second layer of protection within the 24-hour Stripe deduplication window.

# SAFE: dbt Python model with idempotency keys + pre-flight check + Keybrake vault key
# Model re-runs skip already-billed customers; idempotency key as second guard
#
# models/billing/charge_customers.py

import hashlib
import stripe

VAULT_KEY = "vk_live_..."  # per-billing-period vault key from Keybrake

def model(dbt, session):
    dbt.config(materialized="table")  # full replace on each successful run

    billing_df = dbt.ref("billing_queue")
    customers = billing_df.collect()

    stripe.api_key  = VAULT_KEY
    stripe.api_base = "https://proxy.keybrake.com/stripe"  # spend-cap proxy

    # Pre-flight: read existing billing records from a persistent table
    # This table is written row-by-row inside this loop — it survives model failure
    try:
        existing_df = dbt.ref("billing_results_durable")
        already_billed = {
            row["customer_id"] + "|" + row["billing_period"]
            for row in existing_df.collect()
        }
    except Exception:
        already_billed = set()  # first run — table does not exist yet

    results = []
    for row in customers:
        key = row["customer_id"] + "|" + row["billing_period"]
        if key in already_billed:
            # Already billed in a previous (failed or successful) run — skip
            results.append({
                "customer_id": row["customer_id"],
                "charge_id": None,
                "billing_period": row["billing_period"],
                "skipped": True
            })
            continue

        # Content-hash idempotency key — stable across all model re-runs
        raw_key = f"{row['customer_id']}:{row['amount_cents']}:{row['billing_period']}:dbt-billing"
        idempotency_key = hashlib.sha256(raw_key.encode()).hexdigest()[:32]

        try:
            charge = stripe.Charge.create(
                amount=row["amount_cents"],
                currency="usd",
                customer=row["stripe_customer_id"],
                description=f"Billing for {row['billing_period']}",
                idempotency_key=idempotency_key
            )
            charge_id = charge.id
        except stripe.error.CardError as e:
            charge_id = None  # permanent failure — record it, do not raise
        except stripe.error.InvalidRequestError as e:
            charge_id = None

        results.append({
            "customer_id": row["customer_id"],
            "charge_id": charge_id,
            "billing_period": row["billing_period"],
            "amount_cents": row["amount_cents"],
            "idempotency_key": idempotency_key,
            "skipped": False
        })

    return session.createDataFrame(results)

The billing_results_durable model referenced in the pre-flight check is a separate dbt model (or an external table maintained outside dbt) that is written incrementally and survives the failure of the charge_customers model. When the charging loop writes results to the warehouse row-by-row — inside the Python model loop, not only in the final return — those writes survive model failure. The pre-flight check on the next re-run finds those records and skips the corresponding Stripe calls.

Failure mode 2: dbt post-hook triggers a billing Lambda — fires again on model re-run

dbt allows post-hook and pre-hook SQL or macro calls configured on any model. A common pattern for wiring dbt into a billing workflow: after a billing_queue model materializes, trigger a billing Lambda or function that reads from the freshly materialized table and calls Stripe. In dbt YAML this looks like:

# UNSAFE: dbt post-hook that triggers billing on every model materialization
# If the model is re-run, the post-hook fires again — Lambda is invoked twice
#
# models/billing/schema.yml

models:
  - name: billing_queue
    config:
      materialized: table
      post-hook:
        # UNSAFE: triggers billing Lambda every time the model materializes
        # On model failure + re-run: Lambda is invoked for the second time
        - "{{ run_query(\"CALL system$invoke_billing_lambda('\" + this.schema + \"',
                        '\" + this.name + \"')\") }}"

The failure mode sequence: dbt runs billing_queue, materializes the table successfully, then executes the post-hook, which invokes the billing Lambda. The Lambda reads from billing_queue, calls stripe.charges.create() for each customer in the table, and returns. Now a downstream model that depends on billing_queue fails — a schema mismatch, a test failure, a connection error. The dbt run is marked failed. The engineer re-runs dbt run (or dbt Cloud auto-retries the job). dbt re-runs billing_queue — because the run failed, the state is not clean, and dbt re-executes models that were not confirmed as part of a fully successful run. The post-hook fires again. The billing Lambda is invoked a second time. It reads the same billing_queue table (same customers, same amounts) and calls stripe.charges.create() again for every customer.

This failure mode is subtle because nothing in the post-hook configuration signals that the hook should only fire once per billing period. The post-hook fires on every model materialization — including re-materializations on retry. The Lambda itself sees no signal distinguishing a first invocation from a re-invocation: both invocations receive the same table name and the same customer data.

# SAFE: dbt post-hook with idempotency-aware billing Lambda
# Lambda uses content-hash idempotency keys + billing period lock to handle re-invocations
#
# models/billing/schema.yml

models:
  - name: billing_queue
    config:
      materialized: table
      post-hook:
        # Pass billing_period as a parameter so Lambda can scope its idempotency checks
        - "{{ run_query(\"CALL system$invoke_billing_lambda('\" + this.schema + \"',
                        '\" + this.name + \"', '\" + var('billing_period') + \"')\") }}"

# SAFE: billing Lambda handler — idempotency-first design handles double invocation
# A second invocation for the same billing_period is a no-op after pre-flight checks
#
# lambda_function.py

import hashlib
import boto3
import stripe
import snowflake.connector

VAULT_KEY = "vk_live_..."
dynamodb  = boto3.resource("dynamodb")
lock_table  = dynamodb.Table("billing_period_locks")
billing_table = dynamodb.Table("billing_records")

def handler(event, context):
    schema         = event["schema"]
    table_name     = event["table_name"]
    billing_period = event["billing_period"]

    # Billing period lock: only one Lambda invocation processes a given billing period.
    # Second invocation for the same period returns early.
    # Uses DynamoDB conditional write — atomic, survives concurrent invocations.
    try:
        lock_table.put_item(
            Item={"billing_period": billing_period, "locked_at": context.aws_request_id},
            ConditionExpression="attribute_not_exists(billing_period)"
        )
    except dynamodb.meta.client.exceptions.ConditionalCheckFailedException:
        # Another Lambda invocation already holds the lock for this period
        return {"status": "skipped", "reason": "billing_period already locked"}

    stripe.api_key  = VAULT_KEY
    stripe.api_base = "https://proxy.keybrake.com/stripe"

    conn   = snowflake.connector.connect(...)
    cursor = conn.cursor()
    cursor.execute(f"SELECT customer_id, stripe_customer_id, amount_cents FROM {schema}.{table_name}")
    customers = cursor.fetchall()

    results = []
    for customer_id, stripe_cus, amount_cents in customers:
        # Per-customer pre-flight check — protects if billing_period lock was released
        existing = billing_table.get_item(
            Key={"customer_id": customer_id, "billing_period": billing_period}
        ).get("Item")
        if existing:
            results.append({"customer_id": customer_id, "charge_id": existing["charge_id"], "skipped": True})
            continue

        raw_key = f"{customer_id}:{amount_cents}:{billing_period}:dbt-billing"
        idempotency_key = hashlib.sha256(raw_key.encode()).hexdigest()[:32]

        try:
            charge = stripe.Charge.create(
                amount=amount_cents,
                currency="usd",
                customer=stripe_cus,
                description=f"Billing for {billing_period}",
                idempotency_key=idempotency_key
            )
            charge_id = charge.id
        except stripe.error.CardError as e:
            charge_id = None

        billing_table.put_item(Item={
            "customer_id": customer_id,
            "billing_period": billing_period,
            "charge_id": charge_id,
            "amount_cents": amount_cents,
            "idempotency_key": idempotency_key
        })
        results.append({"customer_id": customer_id, "charge_id": charge_id})

    return {"status": "ok", "charged": len(results)}

The billing period lock is the key structural protection here. The DynamoDB conditional write on billing_period — attribute_not_exists(billing_period) — is atomic: exactly one Lambda invocation wins the insert; concurrent or re-invoked Lambdas receive a ConditionalCheckFailedException and exit early without calling Stripe. The per-customer pre-flight check and idempotency key provide a second layer of protection for the period when the lock is held by the first invocation but some individual Stripe calls have not yet completed.

Failure mode 3: dbt run --full-refresh rebuilds the billing queue and puts already-billed customers back in it

dbt incremental models are a common pattern for billing queues: the model materializes as a table on the first run, and on subsequent runs uses an is_incremental() filter to append only new records — customers who crossed a usage threshold since the last model run. The billing pipeline reads from this incremental table, charges Stripe for each customer in the queue, and marks those customers as billed in a separate table. The incremental model's next run uses the is_incremental() filter to exclude already-billed customers.

The failure mode is dbt run --full-refresh. This flag tells dbt to drop and recreate the incremental model's table from scratch, re-running the full model SQL without the is_incremental() filter. It is a routine operation used to fix schema changes (adding a column to the incremental model), correct data quality issues in historical records, or respond to a bug in the incremental filter logic. Engineers run it without considering billing consequences — to them it looks like a data quality fix, not a billing trigger.

The consequence: after a --full-refresh, the billing_queue incremental model contains every customer who has ever met the billing threshold — including customers billed in previous months, not just new unbilled customers. If the downstream billing pipeline reads from billing_queue and charges all rows in the table, every customer billed before the full-refresh is charged again. The billing pipeline has no visibility into the fact that a full-refresh happened; it sees a table full of customers and calls Stripe for all of them.

-- UNSAFE: incremental billing_queue model
-- After --full-refresh, all historical customers reappear as if they are unbilled
--
-- models/billing/billing_queue.sql

{{
  config(
    materialized='incremental',
    unique_key='customer_id'
  )
}}

SELECT
    customer_id,
    stripe_customer_id,
    SUM(usage_units) AS total_units,
    SUM(usage_units) * 0.001 * 100 AS amount_cents,  -- $0.001 per unit
    '{{ var("billing_period") }}' AS billing_period
FROM {{ ref("usage_events") }}
WHERE usage_date >= '{{ var("billing_period") }}-01'
  AND usage_date <  '{{ var("next_period_start") }}'

{% if is_incremental() %}
  -- This filter excludes already-billed customers on incremental runs
  -- But --full-refresh bypasses is_incremental() entirely: all customers appear
  AND customer_id NOT IN (SELECT customer_id FROM {{ this }})
{% endif %}

GROUP BY customer_id, stripe_customer_id

The fix is to move the "has this customer been billed for this period" check out of the dbt incremental model and into the billing pipeline itself — in a form that survives both --full-refresh and schema changes. The billing pipeline's DynamoDB pre-flight check (or equivalent) is the authoritative source of truth for which customers have been billed for which period, independent of dbt's incremental state. A --full-refresh on the dbt model has no effect on the DynamoDB billing table — the pre-flight check still finds the existing records and skips the Stripe calls.

-- SAFE: incremental billing_queue model — uses dbt as a transformation layer only
-- Billing deduplication is handled by the downstream pipeline's pre-flight check,
-- not by the is_incremental() filter. --full-refresh does not cause double billing.
--
-- models/billing/billing_queue.sql

{{
  config(
    materialized='incremental',
    unique_key='customer_id || \'|\' || billing_period'
  )
}}

SELECT
    customer_id,
    stripe_customer_id,
    SUM(usage_units) * 0.001 * 100 AS amount_cents,
    '{{ var("billing_period") }}' AS billing_period,
    -- Explicit flag: downstream pipeline MUST check this against billing_records table
    -- regardless of whether this model ran with or without --full-refresh
    'pending' AS billing_status
FROM {{ ref("usage_events") }}
WHERE usage_date >= '{{ var("billing_period") }}-01'
  AND usage_date <  '{{ var("next_period_start") }}'
GROUP BY customer_id, stripe_customer_id

# SAFE: billing pipeline that reads dbt billing_queue
# Pre-flight check against DynamoDB billing_records is the authoritative dedup guard
# --full-refresh on the dbt model does NOT cause double billing
#
# billing_runner.py

import hashlib
import boto3
import stripe
import snowflake.connector

VAULT_KEY      = "vk_live_..."
BILLING_PERIOD = "2026-07"  # explicit constant — not computed at runtime

dynamodb      = boto3.resource("dynamodb")
billing_table = dynamodb.Table("billing_records")

stripe.api_key  = VAULT_KEY
stripe.api_base = "https://proxy.keybrake.com/stripe"

conn   = snowflake.connector.connect(...)
cursor = conn.cursor()
cursor.execute("""
    SELECT customer_id, stripe_customer_id, amount_cents, billing_period
    FROM analytics.billing_queue
    WHERE billing_period = %s
""", (BILLING_PERIOD,))

for customer_id, stripe_cus, amount_cents, billing_period in cursor:
    # Pre-flight check: authoritative guard against dbt --full-refresh and re-runs
    existing = billing_table.get_item(
        Key={"customer_id": customer_id, "billing_period": billing_period}
    ).get("Item")
    if existing:
        continue  # already billed — skip regardless of dbt model state

    raw_key = f"{customer_id}:{amount_cents}:{billing_period}:dbt-billing"
    idempotency_key = hashlib.sha256(raw_key.encode()).hexdigest()[:32]

    try:
        charge = stripe.Charge.create(
            amount=amount_cents,
            currency="usd",
            customer=stripe_cus,
            description=f"Billing for {billing_period}",
            idempotency_key=idempotency_key
        )
    except stripe.error.CardError:
        continue

    # DynamoDB write immediately after Stripe call
    billing_table.put_item(Item={
        "customer_id": customer_id,
        "billing_period": billing_period,
        "charge_id": charge.id,
        "amount_cents": amount_cents,
        "idempotency_key": idempotency_key
    })

The structural principle: treat dbt as a transformation layer responsible for aggregating and shaping usage data, not as the billing state machine. Billing state — which customers have been charged for which period — lives in a durable store (DynamoDB, PostgreSQL) that is written by the billing pipeline and read by the billing pipeline. The dbt model's incremental state is a query optimization tool, not a billing idempotency mechanism. This separation means that any dbt operation — --full-refresh, schema change, model rebuild — has no effect on billing outcomes.

The two-layer governance pattern for all three failure modes

Each dbt failure mode has a distinct structural root cause. The same two-layer pattern closes all three and provides defense in depth for dbt pipeline behaviors not yet encountered.

Layer 1: content-hash idempotency keys + external pre-flight check. Before every stripe.charges.create() call — whether in a dbt Python model, a post-hook Lambda, or a downstream billing script — query a durable external store (DynamoDB, PostgreSQL billing table) for an existing record keyed on (customer_id, billing_period). If found, skip. If not, compute sha256(customer_id:amount_cents:billing_period:dbt-billing)[:32] and pass as idempotency_key. Write the billing record to the external store immediately after the Stripe call succeeds. The external store must be outside dbt's managed schema — a table that dbt models overwrite is not a durable store. billing_period must be a constant (a dbt var, a Lambda environment variable, an explicit CLI argument) — not derived from CURRENT_DATE() or datetime.now() at execution time.

Layer 2: per-period vault keys via a spend-cap proxy. Issue a Keybrake vault key before each billing period's pipeline run — scoped to the total expected billing amount for the period across all customers — and use it as the Stripe API key in every billing call within that run. The proxy enforces the per-period USD cap, logs every proxied request with its idempotency key and HTTP status, and provides a kill switch that takes effect in under a second regardless of whether the billing call originates from a dbt Python model, a Lambda invoked via a post-hook, or a standalone billing script.

dbt-specific audit gaps that make Stripe billing failures hard to detect

dbt's run artifacts — run_results.json, the manifest, the catalog — record which models ran, their execution time, row counts, and pass/fail status. They do not record application-level events inside Python model functions: which customers were processed, which Stripe charge IDs were returned, or whether a given model run was a re-run of a previous failed run. When a customer reports a double charge, dbt's run history shows two model runs (both may show as "success" if the second re-run completed without error) with no signal that the second run re-charged customers processed in the first.

dbt Cloud's job run history adds metadata like trigger source (scheduled vs. manual vs. API), run duration, and which models were run. It does not add billing-level visibility. The only durable record of which customer was charged when is in the external billing table written by the pipeline — the DynamoDB records written inside the Python model loop or the Lambda handler. The Keybrake proxy audit log adds a second durable record: every proxied Stripe request appears in the log with its idempotency key, response status, timestamp, and vault key — giving you a complete cross-run audit trail that dbt's own run history does not provide.

Stopping an in-flight billing run that is executing inside a dbt Python model requires either cancelling the dbt run (via the dbt Cloud API or Ctrl-C on the CLI) or killing the warehouse session running the Python model. Neither approach stops Stripe calls that are already in flight at the moment of cancellation — a warehouse cancel kills the Python process, but a Stripe HTTP request that has already been sent by the Python SDK is not recalled. Revoking the Keybrake vault key takes effect at the proxy layer in under a second, blocking all requests from that key before they reach Stripe — regardless of whether the cancellation arrives before or after the Python process is terminated.

FAQ

Should I set materialized='view' on my billing_queue model to prevent data from being cached and re-used?

A view materialization means the model SQL re-runs on every downstream query. For a billing queue this is usually not what you want — you want a snapshot of customers ready to be billed at the time the billing run starts, not a view that reflects upstream changes mid-run. Use materialized='table' for billing queues and rely on the external DynamoDB pre-flight check for idempotency. A view does not help with double-charge protection; it just moves where the data is read from.

Can I use dbt's unique_key on an incremental model to prevent duplicate customers in the billing queue?

unique_key on an incremental model tells dbt how to merge new records with existing ones — it prevents duplicate rows in the dbt table, not duplicate Stripe charges. A customer can appear once in the billing_queue table (deduplicated by unique_key) and still be charged twice if the downstream billing pipeline runs twice for the same period. Billing deduplication requires external state written at the moment of the Stripe call, not SQL-level row deduplication in the dbt model.

How does the billing_period variable interact with dbt's scheduled runs?

If billing_period is computed inside the dbt model from CURRENT_DATE() or a warehouse function, and the billing job is scheduled at midnight on the first of the month, a job that starts at 11:59 PM and runs past midnight produces models where some nodes use the last day of the old month and others use the first day of the new month. This creates a mismatched billing period in the idempotency key. Pass billing_period as an explicit dbt var from the job scheduler — dbt run --vars '{"billing_period": "2026-07"}' — so the value is constant for the entire run, including all re-run attempts.

What happens to a Keybrake vault key if a dbt Cloud job is cancelled mid-run?

The vault key remains valid until its TTL expires or you revoke it explicitly. If you need to cancel a billing run and ensure no further Stripe charges are issued — even from in-flight warehouse sessions that haven't received the cancellation signal yet — revoke the vault key immediately after cancelling the dbt job. Revocation takes effect at the proxy in under a second: any Stripe request that reaches the proxy after revocation receives a 401 and is not forwarded to Stripe. This is the correct sequence: cancel dbt Cloud job → revoke vault key → investigate → re-run with a new vault key.

Should the billing post-hook be replaced with a dbt Cloud webhook instead?

dbt Cloud webhooks fire after a job completes successfully — not after each model run. This makes them more appropriate for triggering billing than post-hooks, which fire after each model materialization including re-runs. A webhook-triggered billing Lambda sees exactly one invocation per successful dbt Cloud job run. However, the billing period DynamoDB lock and per-customer pre-flight check are still required — dbt Cloud can fire multiple webhooks for the same job (retry via webhook delivery failure, duplicate delivery in rare cases), and the "exactly once per successful job" guarantee is at the webhook level, not the billing level. Treat the webhook as a trigger, not as an idempotency mechanism.