Monte Carlo Data Observability and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Monte Carlo is a data observability platform that monitors warehouse table freshness, row-count volume, and value distribution, then fires alerts and webhooks when anomalies are detected. Teams wire Monte Carlo into Stripe billing pipelines as a circuit breaker: if the usage_events table is stale or anomalous, halt the billing Lambda until the incident resolves. Three Monte Carlo-specific behaviors introduce Stripe double-charge failure modes that Monte Carlo’s own incident log, lineage graph, and anomaly dashboards do not surface as billing risk.

This post covers those three failure modes with Monte Carlo API usage, dbt model YAML, content-hash idempotency keys, per-run vault keys via a spend-cap proxy, and pre-flight database checks — the two-layer governance pattern that eliminates all three without changing how your Monte Carlo monitors are configured.

Failure mode 1: mid-run circuit breaker exits billing without committing partial progress — retry re-charges already-billed customers

A common Monte Carlo integration pattern: the billing Lambda polls the Monte Carlo API for open incidents before each page of customers. If an incident is open on the usage_events table, the Lambda exits cleanly (return code 0) and waits for the next trigger. The intent is defensive — stop billing on bad data.

The problem is exit ordering. The billing Lambda processes customers in pages of 100. After each page it calls stripe.charges.create() for each customer in the page, then polls Monte Carlo before proceeding to the next page. If Monte Carlo opens a freshness or volume incident while the Lambda is partway through — say, after successfully charging page 1 (customers 1–100) and page 2 (customers 101–200) — the poll check on page 3 finds an open incident. The Lambda exits with code 0. Clean exit, no error. But it does not commit partial progress to the billing database before exiting: the 200 customers in pages 1 and 2 were charged in Stripe but the Lambda never wrote their charge_id values to billing_records, because the billing function writes to billing_records only at the end of a full successful run.

When the incident resolves and the billing Lambda retries, it reads the billing database to find customers with billed_at IS NULL. All 1,000 customers qualify — including the 200 already charged in the partial first run. The retry charges all 1,000. Customers 1–200 receive their second Stripe charge of the billing period.

import stripe, os, requests, time

MONTE_CARLO_API_KEY = os.environ["MONTE_CARLO_API_KEY"]
MC_API_BASE = "https://api.getmontecarlo.com/v1"

def has_open_incident(table_name: str) -> bool:
    resp = requests.get(
        f"{MC_API_BASE}/incidents",
        headers={"x-mcd-id": MONTE_CARLO_API_KEY},
        params={"status": "active", "asset": table_name},
        timeout=5,
    )
    resp.raise_for_status()
    return len(resp.json().get("incidents", [])) > 0

# UNSAFE billing loop: exits cleanly on MC incident but never commits partial progress
def run_billing_unsafe(customers: list, billing_period: str):
    stripe.api_key = os.environ["STRIPE_SECRET_KEY"]  # UNSAFE: unrestricted, shared

    for page_start in range(0, len(customers), 100):
        # Poll Monte Carlo before each page
        if has_open_incident("usage_events"):
            print("Monte Carlo incident detected — exiting cleanly.")
            return  # DANGER: 0..page_start customers were already charged in Stripe
                    # but billing_records has no rows for them yet.
                    # Retry will re-charge them because billed_at IS NULL in the DB.

        page = customers[page_start : page_start + 100]
        for customer in page:
            stripe.Charge.create(
                amount=customer["amount_cents"],
                currency="usd",
                customer=customer["stripe_customer_id"],
                description=f"Usage billing for {billing_period}",
                # no idempotency_key — retry creates ch_B for the same billing event
            )
        # billing_records written only here, at end of each page — but only if we reach it.
        # If MC check above triggers exit before this page's write, charges are orphaned.
        commit_page_to_billing_db(page, billing_period)

Monte Carlo’s incident log records the incident correctly. The Stripe dashboard shows two charges per customer for the affected period. Monte Carlo’s billing function exit is logged as a clean termination — not a failure. Nothing in Monte Carlo’s UI links the incident to the subsequent duplicate charges.

The incident timing makes this harder to reproduce: Monte Carlo’s monitors run on a configurable schedule (typically every 15–60 minutes). A billing run that takes less than 15 minutes might never trigger a mid-run poll hit. But a billing run across 5,000 customers that takes 45 minutes will almost certainly encounter at least one monitor cycle during its execution — and any incident that opens during that window (on any monitored table, if the circuit breaker is not scoped to the specific billing table) triggers the exit.

Failure mode 2: incident-resolution webhook and scheduled billing trigger fire concurrently on the same billing period

Teams commonly wire two triggers to their billing Lambda: a scheduled daily or monthly trigger and a Monte Carlo incident-resolution webhook (“when the data quality incident on usage_events resolves, re-run billing in case it was delayed by the incident”). Both triggers are correct in isolation — the schedule covers the normal case and the webhook covers the delayed case. Together they create a concurrent billing scenario whenever the two fire within the same billing period.

The collision is more common than it looks. The scheduled billing trigger fires at 02:00 UTC on the first of the month. At 01:40, a freshness monitor detects that usage_events is 30 minutes stale (ETL ran 30 minutes late). An incident opens. The billing Lambda’s startup check sees the open incident and exits. At 02:10, the ETL completes and Monte Carlo auto-resolves the incident. The incident-resolution webhook fires the billing Lambda. Now: the Lambda starts billing at 02:10, which is in the same billing period as the scheduled trigger that fired at 02:00. The scheduled trigger also retries (because its first attempt exited cleanly via the circuit breaker). Both retry attempts fire within the same billing window.

# AWS Lambda configuration with two triggers — creates concurrent billing runs:

# Trigger 1: EventBridge schedule (first of each month, 02:00 UTC)
# Trigger 2: Monte Carlo webhook via API Gateway → Lambda

# Both triggers invoke the same Lambda function.
# Lambda has no mechanism to detect which trigger source fired it
# (unless event source is explicitly checked — often omitted in early implementations).

def handler(event, context):
    source = event.get("source", "unknown")
    billing_period = get_current_billing_period()

    # MISSING: no check for whether billing_period is already in progress
    # Both the schedule trigger and the MC webhook trigger reach this point.
    run_billing(billing_period)  # charges all customers — called twice concurrently


# The billing_records table is not checked before starting:
# Run A (schedule retry): starts at 02:10:01 UTC, reads 1,000 customers, begins charging
# Run B (MC webhook):     starts at 02:10:03 UTC, reads 1,000 customers, begins charging
#
# Both runs read billing_records before Run A has committed any rows.
# Both see 1,000 customers with billed_at IS NULL.
# Both call stripe.charges.create() for all 1,000.
# Stripe receives two independent charges per customer (no shared idempotency key).
# 2,000 total charges for 1,000 customers.

Monte Carlo’s incident log shows one freshness incident that opened at 01:40 and resolved at 02:10 — consistent with a normal ETL delay. The incident resolution timestamp (02:10) matches the webhook trigger timestamp. The Lambda execution logs show two concurrent invocations starting within 2 seconds of each other. Stripe’s API shows two charges per customer, both at 02:10 UTC, both with different Request-Id headers (neither run used an idempotency key). Nothing in Monte Carlo’s dashboard links the webhook automation to the duplicate charges.

Failure mode 3: dbt model JOIN fan-out creates duplicate usage rows invisible to Monte Carlo’s row-count anomaly detector

Monte Carlo monitors the row count of usage_events_daily, the dbt model that aggregates usage for billing. Row-count anomaly detection works on percentage deviation from historical baseline — Monte Carlo fires an incident when today’s row count deviates from the trailing 14-day average by more than a configurable threshold (typically 20–30%).

A dbt model change introduces a fan-out: an engineer adds a JOIN with subscription_tiers to include each customer’s tier label on their usage rows. The JOIN is LEFT JOIN subscription_tiers ON usage_events.customer_id = subscription_tiers.customer_id. Most customers have exactly one active tier and the JOIN is 1:1. But 200 of 1,000 customers are in the middle of a plan migration — they have two active rows in subscription_tiers (legacy plan and new plan, both with active = true). The JOIN returns two rows for those customers, inflating usage_events_daily from 1,000 rows to 1,200.

A 20% row count increase would normally trigger a Monte Carlo volume anomaly. But this month also added 190 new customers, so the row count increase is 20% from the fan-out plus 19% from new customer growth — a combined 39% increase. Monte Carlo’s detector sees the 39% increase and fires an anomaly. The engineer reviews it and marks it as expected (“new customer cohort this month, plus a migration batch”) and resolves the incident without investigating the fan-out. The billing Lambda proceeds with 1,200 rows. 200 customers with dual-tier rows receive two charges: one per row in usage_events_daily.

-- BEFORE: dbt model produces one row per customer per billing period
-- models/usage_events_daily.sql

SELECT
    customer_id,
    billing_period,
    SUM(event_cost_cents) AS amount_cents
FROM {{ ref('raw_usage_events') }}
GROUP BY customer_id, billing_period


-- AFTER: fan-out introduced by subscription_tiers JOIN
-- models/usage_events_daily.sql

SELECT
    ue.customer_id,
    ue.billing_period,
    SUM(ue.event_cost_cents) AS amount_cents,
    st.tier_label,                   -- new column for billing function enrichment
    st.tier_id                       -- new column for billing function enrichment
FROM {{ ref('raw_usage_events') }} ue
LEFT JOIN {{ ref('subscription_tiers') }} st
    ON ue.customer_id = st.customer_id
    -- MISSING: AND st.active = true AND st.is_primary = true
    -- Without this filter, customers with multiple active tiers get one row per tier.
GROUP BY ue.customer_id, ue.billing_period, st.tier_label, st.tier_id
--        ^-- GROUP BY now includes tier columns, breaking the one-row-per-customer guarantee

# Billing function: charges one Stripe fee per row in usage_events_daily
# UNSAFE: no dedup check — assumes one row per customer per billing period

def run_billing(rows: list, billing_period: str):
    stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

    for row in rows:
        customer_id = row["customer_id"]
        amount_cents = row["amount_cents"]
        stripe_customer_id = row["stripe_customer_id"]
        tier_label = row.get("tier_label", "default")  # new field from dbt change

        stripe.Charge.create(
            amount=amount_cents,
            currency="usd",
            customer=stripe_customer_id,
            description=f"Usage billing for {billing_period} ({tier_label})",
            # no idempotency_key — two rows for the same customer produce two charges
        )

# Customer cus_001 has two rows in usage_events_daily this month:
# {"customer_id": "cus_001", "amount_cents": 4900, "tier_label": "legacy", "tier_id": "t_1"}
# {"customer_id": "cus_001", "amount_cents": 4900, "tier_label": "growth", "tier_id": "t_2"}
#
# run_billing() creates ch_A ($49) and ch_B ($49) for cus_001.
# Stripe records two separate charges. Neither is an error.
# Monte Carlo's lineage shows usage_events_daily ← subscription_tiers: correct.
# Monte Carlo's row-count monitor saw a 39% increase and was manually resolved: correct.
# Nothing alerts on the billing semantics of the fan-out.

Monte Carlo’s lineage graph correctly shows subscription_tiers as an upstream dependency of usage_events_daily. It shows the column-level lineage: tier_label and tier_id flow from subscription_tiers into usage_events_daily. What Monte Carlo does not show is the cardinality change: the model now produces a variable number of rows per customer depending on subscription_tiers contents. The billing function assumes one row per customer — a constraint that Monte Carlo’s monitoring infrastructure has no way to express or enforce without a custom monitor that checks row count per customer_id, not total row count.

The fix: content-hash idempotency key + vault key + pre-flight check

All three failure modes share the same root cause: the billing function fires multiple times for the same customer in the same billing period, and nothing prevents the subsequent calls from creating duplicate Stripe charges. The fix has two independent layers that close all three modes without modifying Monte Carlo’s monitor configuration or alert routing.

Layer 1: content-hash idempotency key stable across billing run retries and concurrent invocations

The Stripe idempotency key must be derived from the business event’s stable fields — fields that do not change across billing retries, concurrent webhook + schedule triggers, or fan-out row duplicates. The key must be scoped to (customer_id, billing_period) and not include any row-level metadata that differs between fan-out rows (tier_label, tier_id, row position, or Lambda invocation ID).

import hashlib, stripe, os, psycopg2

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

def make_idempotency_key(customer_id: str, billing_period: str) -> str:
    # Derived from stable business identity: customer + billing period only.
    # Must NOT include: tier_label, tier_id, Lambda request ID, trigger source,
    # row index, run timestamp, or any Monte Carlo incident metadata.
    # All of these change across retries, concurrent invocations, and fan-out rows.
    raw = f"{customer_id}:{billing_period}:monte-carlo-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def process_usage_row(conn, row: dict) -> dict:
    customer_id = row["customer_id"]
    stripe_customer_id = row["stripe_customer_id"]
    amount_cents = row["amount_cents"]
    billing_period = row["billing_period"]

    cur = conn.cursor()

    # Pre-flight: skip if already billed this period (survives retries, fan-out duplicates,
    # and concurrent webhook + schedule trigger races beyond Stripe's 24h idempotency window)
    cur.execute(
        "SELECT 1 FROM billing_records WHERE customer_id = %s AND billing_period = %s",
        (customer_id, billing_period)
    )
    if cur.fetchone():
        return {"status": "skipped", "customer_id": customer_id}

    idempotency_key = make_idempotency_key(customer_id, billing_period)

    response = stripe.Charge.create(
        amount=amount_cents,
        currency="usd",
        customer=stripe_customer_id,
        description=f"Usage billing for {billing_period}",
        idempotency_key=idempotency_key,
    )

    cur.execute(
        "INSERT INTO billing_records (customer_id, billing_period, charge_id, billed_at) "
        "VALUES (%s, %s, %s, NOW()) ON CONFLICT (customer_id, billing_period) DO NOTHING",
        (customer_id, billing_period, response["id"])
    )
    conn.commit()
    return {"status": "charged", "charge_id": response["id"]}

The idempotency key closes failure mode 1 (mid-run exit retry) and failure mode 2 (concurrent invocations) within Stripe’s 24-hour idempotency window. For fan-out rows (failure mode 3), the same key is generated for both fan-out rows of the same customer — Stripe returns the existing charge for the second call rather than creating a new one. The pre-flight check closes all three failure modes beyond the 24-hour window and adds a defense-in-depth layer within the window.

Layer 2: per-billing-period vault key capped at expected total × 1.10

Issue a vault key before the billing run capped at 110% of the expected total charges for the billing period. The vault key replaces the raw Stripe restricted key as the stripe.api_key for the billing function. Any run that exceeds the cap — because of a concurrent second run, a mid-run exit retry that double-charges, or a fan-out that doubles every customer’s charge — receives a 402 spend_cap_exceeded response from the proxy after the cap is hit. No further charges clear until a new vault key is issued.

import requests, os

PROXY_BASE = "https://proxy.keybrake.com"
PROXY_KEY = os.environ["KEYBRAKE_API_KEY"]

def issue_billing_vault_key(billing_period: str, expected_total_usd: float) -> str:
    cap = round(expected_total_usd * 1.10, 2)
    resp = requests.post(
        f"{PROXY_BASE}/vault/keys",
        headers={"Authorization": f"Bearer {PROXY_KEY}"},
        json={
            "label": f"monte-carlo-billing-{billing_period}",
            "vendor": "stripe",
            "daily_usd_cap": cap,
            "allowed_endpoints": ["/v1/charges"],
            "expires_at": f"{billing_period}-31T23:59:59Z",
        },
    )
    resp.raise_for_status()
    return resp.json()["vault_key"]

# Handler with vault key and per-run advisory lock:
def handler(event, context):
    billing_period = get_current_billing_period()
    expected_total = query_expected_total(billing_period)
    vault_key = issue_billing_vault_key(billing_period, expected_total)
    stripe.api_key = vault_key  # vault key, not raw Stripe key

    conn = psycopg2.connect(os.environ["DATABASE_URL"])
    try:
        results = run_billing_with_lock(conn, billing_period)
    except RuntimeError as e:
        print(f"BILLING_LOCK_CONTENTION: {e}")
        return {"statusCode": 409, "body": str(e)}
    return {"statusCode": 200, "charged": len([r for r in results if r["status"] == "charged"])}

Governance gaps specific to Monte Carlo

Four Monte Carlo-specific behaviors reduce the effectiveness of billing governance patterns designed for other pipeline tools:

• Monte Carlo’s circuit breaker signal is advisory, not transactional. The billing function checks Monte Carlo’s API for open incidents, but there is no transactional link between the Monte Carlo incident state and the billing function’s database writes. An exit triggered by an open Monte Carlo incident leaves no record in the billing database about which customers were already charged before the exit. The billing database — not Monte Carlo’s incident log — is the authoritative source of truth for what has been charged. The pre-flight check reads from billing_records, not from Monte Carlo.
• Monte Carlo’s incident-resolution webhook does not carry billing-period context. The webhook payload includes the incident ID, the affected asset (table name), and the resolution timestamp. It does not include the billing period that was delayed by the incident, or whether a billing run for that period already started and partially completed. A billing function that treats every incident resolution as “safe to bill now” will fire even when a concurrent run from the scheduled trigger is already in progress.
• Monte Carlo’s row-count anomaly detector measures deviation from baseline, not semantic correctness. A dbt fan-out that increases row count by 20% triggers a Monte Carlo alert. A fan-out that increases row count by 18% (within the detection band) or is co-incident with genuine volume growth (masking the fan-out signal) passes silently. Monte Carlo does not validate the cardinality constraint that the billing function relies on — one row per customer per billing period. A custom Monte Carlo monitor that checks COUNT(*) / COUNT(DISTINCT customer_id) ratio detects fan-outs regardless of total volume change.
• Monte Carlo’s lineage graph shows dependencies but not cardinality contracts. The lineage graph shows that usage_events_daily depends on subscription_tiers after the JOIN is added. It does not show that the relationship is supposed to be 1:1 per customer. Engineers reviewing the lineage after a duplicate-charge incident will see a correct graph — the dependency is real — and may not immediately identify the fan-out as the cause of the billing error.

Comparison: protection by governance layer

Advisory lock for concurrent invocation prevention

The pre-flight check closes the duplicate-charge window for customers already in billing_records, but a concurrent invocation race condition exists during the first run’s commit window — between when Run A has charged a customer and when it commits the billing_records row. Run B’s pre-flight check during that window sees no row for the customer. The idempotency key closes this at the Stripe level, but acquiring an advisory lock before starting the billing loop prevents unnecessary Stripe calls and eliminates the race entirely:

import psycopg2, os

def run_billing_with_lock(conn, billing_period: str) -> list:
    cur = conn.cursor()

    lock_key = hash(f"billing-{billing_period}") & 0x7FFFFFFF
    cur.execute("SELECT pg_try_advisory_lock(%s)", (lock_key,))
    acquired = cur.fetchone()[0]

    if not acquired:
        raise RuntimeError(
            f"Another billing run for {billing_period} is already active. "
            "Exiting to prevent concurrent duplicate charges. "
            "This is expected when the MC incident-resolution webhook fires "
            "while the scheduled billing trigger is already running."
        )

    try:
        rows = load_usage_rows(billing_period)
        results = [process_usage_row(conn, row) for row in rows]
        return results
    finally:
        cur.execute("SELECT pg_advisory_unlock(%s)", (lock_key,))
        conn.commit()

# The advisory lock ensures that when the MC webhook and the scheduled trigger
# fire within seconds of each other (failure mode 2), whichever invocation
# acquires the lock first completes the billing run, and the second invocation
# raises RuntimeError immediately — before making any Stripe calls.
# The RuntimeError is logged as a 409 response, not a billing failure.

Custom Monte Carlo monitor for fan-out detection

The three-layer fix (idempotency key + vault key + pre-flight check) handles fan-out at the billing layer. But fan-outs should be caught upstream — in Monte Carlo — before the billing function sees the expanded dataset. Add a custom Monte Carlo SQL rule monitor on usage_events_daily that checks the row-per-customer ratio:

-- Custom Monte Carlo SQL rule monitor
-- Add via Monte Carlo UI: Monitors → SQL Rule → Usage Events Fan-Out Check

SELECT
    COUNT(*) AS total_rows,
    COUNT(DISTINCT customer_id) AS distinct_customers,
    ROUND(COUNT(*)::numeric / NULLIF(COUNT(DISTINCT customer_id), 0), 2) AS rows_per_customer
FROM usage_events_daily
WHERE billing_period = DATE_TRUNC('month', CURRENT_DATE - INTERVAL '1 day')::date

-- Monitor condition: rows_per_customer > 1.0
-- If any billing period has more than one row per distinct customer,
-- open a Monte Carlo incident before the billing Lambda reads usage_events_daily.
-- This catches fan-outs regardless of whether total row count falls within the
-- volume anomaly detection band.

This monitor fires an incident before the billing run starts, allowing the billing Lambda’s circuit breaker to exit cleanly without any charges. With the advisory lock and idempotency key in place, the billing Lambda can safely retry once the fan-out is corrected in the dbt model and the incident resolves — the pre-flight check skips any customers the corrected run has already processed if a partial run occurred before the monitor fired.

pytest enforcement suite

import hashlib, pytest
from unittest.mock import patch, MagicMock

# 1. Idempotency key is stable across trigger sources and fan-out rows
def test_idempotency_key_excludes_tier_metadata():
    # Both fan-out rows for the same customer must produce the same key
    key1 = make_idempotency_key("cus_001", "2026-07")
    key2 = make_idempotency_key("cus_001", "2026-07")  # same customer, same period
    assert key1 == key2

    # Confirm tier_label does not affect the key (fan-out rows differ only in tier_label)
    raw_without_tier = f"cus_001:2026-07:monte-carlo-billing"
    expected = hashlib.sha256(raw_without_tier.encode()).hexdigest()[:32]
    assert key1 == expected

# 2. Pre-flight skips Stripe call for second fan-out row of same customer/period
def test_preflight_skips_fanout_second_row(mock_db, mock_stripe):
    mock_db.cursor.return_value.fetchone.return_value = (1,)  # billing_records row exists

    result = process_usage_row(
        conn=mock_db,
        row={
            "customer_id": "cus_001",
            "stripe_customer_id": "cus_stripe_001",
            "amount_cents": 4900,
            "billing_period": "2026-07",
            "tier_label": "growth",  # second fan-out row
            "tier_id": "t_2",
        },
    )

    assert result["status"] == "skipped"
    mock_stripe.Charge.create.assert_not_called()

# 3. Mid-run exit retry: already-billed customers skipped by pre-flight
def test_mid_run_exit_retry_skips_already_billed(mock_db, mock_stripe):
    billed_customers = [
        {"customer_id": f"cus_{i}", "stripe_customer_id": f"cus_stripe_{i}",
         "amount_cents": 4900, "billing_period": "2026-07",
         "tier_label": "default", "tier_id": "t_1"}
        for i in range(200)
    ]
    mock_db.cursor.return_value.fetchone.return_value = (1,)  # all in billing_records

    results = [process_usage_row(mock_db, row) for row in billed_customers]

    assert all(r["status"] == "skipped" for r in results)
    mock_stripe.Charge.create.assert_not_called()

# 4. Concurrent invocation: advisory lock blocks second run immediately
def test_advisory_lock_blocks_concurrent_webhook_trigger(mock_conn):
    mock_conn.cursor.return_value.fetchone.side_effect = [(True,), (False,)]

    with pytest.raises(RuntimeError, match="already active"):
        run_billing_with_lock(mock_conn, "2026-07")

# 5. Vault key spend cap blocks excess charges from concurrent second run
def test_vault_key_cap_blocks_concurrent_run(mock_proxy):
    mock_proxy.post.side_effect = (
        [MagicMock(status_code=200, json=lambda: {"id": f"ch_{i}"})] * 1000
        + [MagicMock(status_code=402, json=lambda: {"error": "spend_cap_exceeded"})] * 1000
    )

    run_a = [make_charge_via_proxy(f"cus_{i}", 4900, "2026-07", mock_proxy) for i in range(1000)]
    assert all(r["status_code"] == 200 for r in run_a)

    run_b = [make_charge_via_proxy(f"cus_{i}", 4900, "2026-07", mock_proxy) for i in range(1000)]
    assert all(r["status_code"] == 402 for r in run_b)

FAQ

Q: Should I disable Monte Carlo’s incident-resolution webhook trigger for billing entirely?
A: Disabling it is the safest option if you cannot add the advisory lock and pre-flight check immediately. The scheduled trigger already covers the normal billing path. The webhook trigger is useful for recovering from incidents that cause a long delay past the scheduled billing window — but it creates the concurrent invocation risk whenever it fires close to the scheduled trigger. If you keep both triggers, the advisory lock is mandatory: it ensures that only one billing run per billing period can start, regardless of how many Lambda invocations are launched simultaneously.

Q: How do I detect a mid-run Monte Carlo circuit breaker exit that left orphaned Stripe charges?
A: Query Stripe for all charges in the current billing period and join to your billing_records table. Any charge_id in Stripe that has no corresponding row in billing_records represents a charge made during a partial run that exited before committing. These are not duplicate charges yet — they are charges the billing function doesn’t know it made. Insert them into billing_records as if they were normal charges, then allow the retry to run. The pre-flight check will skip those customers correctly.

Q: Can I use Monte Carlo’s SQL rule monitor to catch fan-outs in real time before billing runs?
A: Yes, and it is the recommended upstream defense. The SQL rule monitor on COUNT(*) / COUNT(DISTINCT customer_id) > 1.0 fires a Monte Carlo incident before the billing run starts, and the billing Lambda’s circuit breaker exits cleanly. The key is scheduling the monitor to run before your billing Lambda’s trigger window — not after. If the monitor runs every hour and billing fires on the first of the month at 02:00, configure the monitor to run at 01:45 so the incident is open before the Lambda starts.

Q: How should I scope the vault key cap when billing period totals vary widely month over month?
A: Use the previous month’s total plus 10% as the cap. For months where you expect higher usage (new customer cohorts, plan upgrades), raise the cap manually before the billing run. The cap is a circuit breaker for runaway charges — it should be set high enough to never block legitimate charges, and low enough to catch a 2x overcharge from a fan-out or concurrent run. A 10% buffer over the expected total satisfies both constraints in most billing pipelines. If the cap is hit during a legitimate billing run, issue a second vault key for the remaining amount and resume.

Q: Is Monte Carlo’s lineage graph sufficient to catch schema changes that affect billing semantics?
A: No. Lineage shows data flow dependencies, not semantic contracts. Monte Carlo will correctly show that usage_events_daily depends on subscription_tiers after a JOIN is added. It will not show that the JOIN is expected to be 1:1 per customer, or that the billing function downstream assumes one row per customer per billing period. Encoding that contract requires a custom SQL rule monitor as described above. Treat Monte Carlo lineage as necessary but not sufficient for billing pipeline safety.