SageMaker Pipelines Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Amazon SageMaker Pipelines is AWS's managed ML workflow orchestrator, increasingly used for post-inference billing pipelines — charge collection after a usage-metering Processing Job, subscription renewal after a model evaluation step, or payment fan-out driven by a batch inference output. Three deployment-level failure modes surface specifically when SageMaker Processing Jobs call Stripe, and all three produce duplicate charges that do not surface in SageMaker's execution status logs.

This post covers three SageMaker-specific failure modes — RetryPolicy re-execution from line 1, ParallelStep fan-out with an unrestricted shared key, and EventBridge at-least-once delivery launching concurrent billing executions — each of which produces multiple stripe.charges.create() calls against already-billed customers, and the two-layer governance pattern that closes all three: content-hash idempotency keys plus per-execution vault keys via a spend-cap proxy, without changes to your pipeline graph.

Failure mode 1: RetryPolicy re-invokes the Processing Job container from line 1 after stripe.charges.create() has already succeeded

SageMaker Pipelines supports RetryPolicy on ProcessingStep, TrainingStep, and TransformStep. When configured, SageMaker re-runs the underlying Processing Job if the step exits with a failure status — the job container is relaunched from scratch with the original input arguments, starting from line 1 of the entrypoint script.

The critical failure window opens when stripe.charges.create() returns a charge object successfully but a subsequent operation raises an unhandled exception that causes the container to exit with a non-zero code. A common example in billing pipelines: the Processing Job calls stripe.charges.create() — returning ch_A — then attempts to write the charge record to a downstream DynamoDB table. If the DynamoDB write raises a ProvisionedThroughputExceededException and the retry backoff logic inside the container exhausts without recovering, the container exits with code 1. SageMaker marks the step as Failed and applies the RetryPolicy: a new Processing Job is launched with the identical input arguments. When the new container starts, stripe.charges.create() fires again from line 1 with no idempotency key — producing ch_B. Both charges succeed. SageMaker's execution history shows one failed attempt and one successful step completion; the duplicate charge is invisible in the pipeline execution record.

# billing_processor.py — UNSAFE: RetryPolicy re-executes from line 1
import os
import stripe
import boto3

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]  # shared, unrestricted
dynamodb = boto3.resource("dynamodb")

def main():
    # Arguments passed via SageMaker Processing Job environment variables
    customer_id = os.environ["CUSTOMER_ID"]
    amount_cents = int(os.environ["AMOUNT_CENTS"])
    billing_period = os.environ["BILLING_PERIOD"]

    # RetryPolicy starts here — stripe.charges.create() fires again on any
    # downstream exception that caused the previous container to exit non-zero
    charge = stripe.charges.create(
        amount=amount_cents,
        currency="usd",
        customer=customer_id,
        description=f"Subscription {billing_period}",
    )

    # If this raises (DynamoDB throughput exhausted, network timeout),
    # the container exits with code 1 → SageMaker RetryPolicy fires →
    # new container starts → stripe.charges.create() fires again → ch_B
    table = dynamodb.Table(os.environ["CHARGES_TABLE"])
    table.put_item(Item={
        "customer_id": customer_id,
        "billing_period": billing_period,
        "charge_id": charge.id,
    })

if __name__ == "__main__":
    main()

Two fixes work in combination. The first is a content-hash idempotency key derived from the Processing Job's stable input arguments before the Stripe call. The key must be computed from values that remain identical across the original run and every retry — customer_id, amount_cents, and billing_period form a stable triple that does not change between the failed container and the retried container, because SageMaker passes the same environment variables to both. Adding a service namespace prefix prevents collisions with idempotency keys from other parts of the system. On retry, stripe.charges.create() receives the same idempotency key and Stripe returns the original charge object rather than creating a new one. The second fix is separating permanent errors from transient ones: stripe.error.CardError and stripe.error.InvalidRequestError cannot be resolved by retrying with the same arguments and must be handled explicitly — the container should exit cleanly (code 0) after recording the declined or invalid state rather than raising, which would consume a retry attempt while creating no additional Stripe calls.

# billing_processor.py — SAFE: content-hash idempotency key + vault key
import hashlib
import os
import requests
import stripe
import boto3

dynamodb = boto3.resource("dynamodb")

def get_vault_key(label: str, allowed_cents: int) -> str:
    resp = requests.post(
        "https://proxy.keybrake.com/keys",
        headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
        json={
            "vendor": "stripe",
            "allowed_endpoints": ["POST /v1/charges"],
            "daily_usd_cap": round((allowed_cents / 100) * 1.10, 2),
            "expires_in_seconds": 3600,
            "label": label,
        },
        timeout=10,
    )
    resp.raise_for_status()
    return resp.json()["vault_key"]

def main():
    customer_id = os.environ["CUSTOMER_ID"]
    amount_cents = int(os.environ["AMOUNT_CENTS"])
    billing_period = os.environ["BILLING_PERIOD"]

    # Idempotency key derived from stable inputs — identical on every retry
    idempotency_key = hashlib.sha256(
        f"{customer_id}:{amount_cents}:{billing_period}:sagemaker-billing".encode()
    ).hexdigest()[:32]

    # Per-job vault key scoped to this customer's charge only
    vault_key = get_vault_key(
        label=f"sm-billing-{customer_id}-{billing_period}",
        allowed_cents=amount_cents,
    )

    stripe_client = stripe.StripeClient(
        vault_key,
        base_url="https://proxy.keybrake.com/stripe/v1",
    )

    try:
        charge = stripe_client.charges.create(
            params={
                "amount": amount_cents,
                "currency": "usd",
                "customer": customer_id,
                "description": f"Subscription {billing_period}",
            },
            options=stripe.RequestOptions(idempotency_key=idempotency_key),
        )
    except stripe.error.CardError as e:
        # Permanent — exit cleanly with a recorded declined status
        table = dynamodb.Table(os.environ["CHARGES_TABLE"])
        table.put_item(Item={
            "customer_id": customer_id,
            "billing_period": billing_period,
            "status": "declined",
            "code": e.code,
        })
        return  # exit 0 — no retry needed
    except stripe.error.InvalidRequestError as e:
        table = dynamodb.Table(os.environ["CHARGES_TABLE"])
        table.put_item(Item={
            "customer_id": customer_id,
            "billing_period": billing_period,
            "status": "invalid",
            "error": str(e),
        })
        return

    table = dynamodb.Table(os.environ["CHARGES_TABLE"])
    table.put_item(Item={
        "customer_id": customer_id,
        "billing_period": billing_period,
        "charge_id": charge.id,
        "status": "charged",
    })

if __name__ == "__main__":
    main()

Failure mode 2: ParallelStep fan-out shares one unrestricted STRIPE_SECRET_KEY across all concurrent Processing Job instances with no per-job spend cap

SageMaker Pipelines' ParallelStep (available in SageMaker SDK v2.x as part of the pipeline step type) allows multiple steps to execute concurrently within a pipeline execution. A common billing architecture fans out one ProcessingStep per customer cohort inside a ParallelStep, batching the monthly charge collection across hundreds or thousands of customers in parallel rather than sequentially.

The failure mode appears at the key scoping layer. Each Processing Job instance launched by the ParallelStep fan-out reads STRIPE_SECRET_KEY from the same SSM Parameter Store path or the same Secrets Manager secret. This is a single unrestricted key shared across every concurrent job in the fan-out: there is no per-job spend cap, no per-cohort endpoint allowlist, and no circuit breaker that stops a runaway charging loop before it propagates. If a data preparation step upstream produces a malformed amount_cents value — for example, a DataFrame column that was multiplied by 100 twice, producing 100× the intended charge amount — all N concurrent Processing Jobs read the same wrong value from the shared pipeline parameter and all N jobs call stripe.charges.create() with the inflated amount simultaneously. The error propagates to the full customer cohort before any single job completes and surfaces a failure. Stripe processes all N charges at the wrong amount; there is nothing in the SageMaker pipeline to stop them.

# pipeline_definition.py — UNSAFE: ParallelStep shares one unrestricted key
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep
from sagemaker.workflow.parallel_step import ParallelStep
from sagemaker.processing import ScriptProcessor
from sagemaker.workflow.parameters import ParameterString

billing_period = ParameterString(name="BillingPeriod", default_value="2026-07")

# All cohort steps pull the SAME STRIPE_SECRET_KEY from SSM — no per-cohort cap
cohort_steps = []
for cohort_id in ["cohort-a", "cohort-b", "cohort-c"]:
    processor = ScriptProcessor(
        image_uri="...",
        role=role_arn,
        instance_count=1,
        instance_type="ml.m5.large",
        env={
            "STRIPE_SECRET_KEY": stripe_key_from_ssm,  # same key for all cohorts
            "COHORT_ID": cohort_id,
        },
    )
    step = ProcessingStep(
        name=f"BillingStep-{cohort_id}",
        processor=processor,
        code="billing_processor.py",
    )
    cohort_steps.append(step)

# All cohort billing steps run concurrently — shared unrestricted key means
# a data error in amount_cents hits all cohorts before any job surfaces a failure
parallel_billing = ParallelStep(
    name="ParallelBillingFanOut",
    steps=cohort_steps,
)

pipeline = Pipeline(
    name="monthly-billing-pipeline",
    parameters=[billing_period],
    steps=[parallel_billing],
)

The fix is issuing per-cohort vault keys in a dedicated ProcessingStep that runs before the ParallelStep fan-out. Each vault key is scoped to a single cohort's total charge amount with a cap of cohort_total_cents × 1.10, a short TTL matching the expected billing run duration, and an endpoint allowlist restricted to POST /v1/charges. The vault keys are written to an S3 object by the issuance step and read by each cohort's Processing Job at startup. If a data error inflates the amount for one cohort, the vault key for that cohort was issued with the correct capped amount before the data error was applied — the proxy blocks the inflated charge before it reaches Stripe and records the violation in the audit log. The remaining cohorts continue unaffected.

# pipeline_definition.py — SAFE: per-cohort vault keys from IssueVaultKeysStep
import json, hashlib, requests, os, boto3
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep
from sagemaker.workflow.parallel_step import ParallelStep
from sagemaker.processing import ScriptProcessor, ProcessingOutput

# vault_key_issuer.py (runs as the IssueVaultKeysStep container)
# -----------------------------------------------------------------
# import json, os, requests, boto3
# s3 = boto3.client("s3")
# cohorts = json.loads(os.environ["COHORTS_JSON"])
# billing_period = os.environ["BILLING_PERIOD"]
# vault_keys = {}
# for cohort in cohorts:
#     resp = requests.post(
#         "https://proxy.keybrake.com/keys",
#         headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
#         json={
#             "vendor": "stripe",
#             "allowed_endpoints": ["POST /v1/charges"],
#             "daily_usd_cap": round((cohort["total_cents"] / 100) * 1.10, 2),
#             "expires_in_seconds": 7200,
#             "label": f"sm-{cohort['id']}-{billing_period}",
#         },
#         timeout=10,
#     )
#     vault_keys[cohort["id"]] = resp.json()["vault_key"]
# s3.put_object(
#     Bucket=os.environ["VAULT_KEYS_BUCKET"],
#     Key=f"vault-keys/{billing_period}.json",
#     Body=json.dumps(vault_keys),
# )
# -----------------------------------------------------------------

issuer_processor = ScriptProcessor(
    image_uri="...", role=role_arn,
    instance_count=1, instance_type="ml.m5.large",
    env={"KEYBRAKE_API_KEY": keybrake_key_from_ssm},
)
issue_vault_keys_step = ProcessingStep(
    name="IssueVaultKeys",
    processor=issuer_processor,
    code="vault_key_issuer.py",
    outputs=[ProcessingOutput(output_name="vault-keys", source="/opt/ml/processing/output")],
)

cohort_steps = []
for cohort_id in ["cohort-a", "cohort-b", "cohort-c"]:
    processor = ScriptProcessor(
        image_uri="...", role=role_arn,
        instance_count=1, instance_type="ml.m5.large",
        env={
            "COHORT_ID": cohort_id,
            "VAULT_KEYS_BUCKET": vault_keys_bucket,
            # STRIPE_SECRET_KEY is NOT passed — vault key loaded at runtime from S3
        },
    )
    step = ProcessingStep(
        name=f"BillingStep-{cohort_id}",
        processor=processor,
        code="billing_processor.py",
        depends_on=[issue_vault_keys_step],  # vault keys must be issued first
    )
    cohort_steps.append(step)

parallel_billing = ParallelStep(name="ParallelBillingFanOut", steps=cohort_steps)

pipeline = Pipeline(
    name="monthly-billing-pipeline",
    steps=[issue_vault_keys_step, parallel_billing],
)

Failure mode 3: EventBridge at-least-once delivery or concurrent StartPipelineExecution calls launch two billing pipeline executions for the same period

SageMaker Pipelines is commonly triggered by EventBridge rules — a rate(30 days) or cron(0 0 1 * ? *) rule that calls sagemaker:StartPipelineExecution via an IAM-authorized Lambda or Step Functions target. EventBridge guarantees at-least-once delivery: under transient target invocation failures (Lambda throttles, Step Functions API errors), EventBridge retries the trigger event. If the first StartPipelineExecution call succeeded but the EventBridge target returned a failure response before acknowledging the event, EventBridge retries the event delivery and a second StartPipelineExecution call is issued with identical input parameters.

The result is two concurrent pipeline executions with distinct Execution ARNs and identical billing inputs — the same BillingPeriod parameter, the same customer cohort data, and the same Stripe credentials — each progressing independently through the pipeline graph. SageMaker Pipelines does not deduplicate concurrent executions for the same pipeline with identical parameters; the second execution is a fully independent run. Both executions reach their billing ProcessingStep, both Processing Jobs call stripe.charges.create() with the same customer IDs and amounts, and both succeed — producing two Stripe charge objects per customer with no idempotency link between them. This failure mode is particularly dangerous because neither execution appears failed in SageMaker's execution history; both show Succeeded status.

# trigger_lambda.py — UNSAFE: no dedup before StartPipelineExecution
import boto3

sm = boto3.client("sagemaker")

def handler(event, context):
    billing_period = event.get("billing_period", current_billing_period())

    # EventBridge at-least-once: this can be called twice for the same event
    # → two concurrent pipeline executions → two stripe.charges.create() calls
    # per customer with no idempotency link between them
    response = sm.start_pipeline_execution(
        PipelineName="monthly-billing-pipeline",
        PipelineParameters=[
            {"Name": "BillingPeriod", "Value": billing_period},
        ],
    )
    return response["PipelineExecutionArn"]

There are two layers of defence at the pipeline trigger level. The first is a deterministic PipelineExecutionDisplayName set to a string derived from the billing period — for example billing-2026-07. SageMaker enforces execution display name uniqueness within a short deduplication window; a second call with the same display name within that window either raises a ConflictException or creates a separate execution with an appended suffix. This is not a reliable hard dedup guarantee across all SageMaker versions, so the second layer is a pre-flight check in the Lambda trigger: before calling StartPipelineExecution, query SageMaker for any execution of the billing pipeline in Executing or Succeeded status with a BillingPeriod parameter matching the current period, and abort if one is found. The content-hash idempotency key at the Processing Job level remains in place as the last line of defence: even if two concurrent executions reach stripe.charges.create(), the idempotency key prevents Stripe from creating a second charge object.

# trigger_lambda.py — SAFE: pre-flight dedup + deterministic execution name
import boto3
from datetime import datetime, timezone

sm = boto3.client("sagemaker")

def current_billing_period() -> str:
    now = datetime.now(timezone.utc)
    return f"{now.year}-{now.month:02d}"

def billing_execution_already_running_or_done(billing_period: str) -> bool:
    """Return True if a billing execution for this period is already active or complete."""
    paginator = sm.get_paginator("list_pipeline_executions")
    for page in paginator.paginate(PipelineName="monthly-billing-pipeline"):
        for execution in page["PipelineExecutionSummaries"]:
            if execution["PipelineExecutionStatus"] not in ("Executing", "Succeeded"):
                continue
            # Check parameters of this execution
            try:
                detail = sm.describe_pipeline_execution(
                    PipelineExecutionArn=execution["PipelineExecutionArn"]
                )
                params = {
                    p["Name"]: p["Value"]
                    for p in detail.get("PipelineParameters", [])
                }
                if params.get("BillingPeriod") == billing_period:
                    return True
            except sm.exceptions.ResourceNotFound:
                continue
    return False

def handler(event, context):
    billing_period = event.get("billing_period", current_billing_period())

    # Pre-flight: abort if billing for this period is already running or done
    if billing_execution_already_running_or_done(billing_period):
        return {
            "status": "skipped",
            "reason": f"Billing execution for {billing_period} already active or complete",
        }

    response = sm.start_pipeline_execution(
        PipelineName="monthly-billing-pipeline",
        PipelineExecutionDisplayName=f"billing-{billing_period}",  # deterministic
        PipelineParameters=[
            {"Name": "BillingPeriod", "Value": billing_period},
        ],
    )
    return {"status": "started", "arn": response["PipelineExecutionArn"]}

Why all three modes need the proxy layer, not just idempotency keys

Idempotency keys prevent Stripe from creating duplicate charge objects when the same key is presented twice. They do not provide visibility into what was attempted. When a SageMaker Processing Job retry fires stripe.charges.create() twice — once successfully and once blocked by the idempotency layer — SageMaker's execution history shows the step as Succeeded and the Processing Job CloudWatch Logs show one completed run. There is no record in SageMaker or CloudWatch that a second Stripe API call was attempted. The idempotent duplicate is invisible in your system until you cross-reference Stripe's API logs against your DynamoDB charge records — a reconciliation that must be done manually and retroactively.

A spend-cap proxy sits at the network layer between your Processing Jobs and Stripe's API. Every vault key is scoped to a single customer's or cohort's billing event with a cap set at amount_cents × 1.10. When the retry fires, the proxy checks the accumulated spend on that vault key, sees the cap is exhausted, and returns a 402 Policy Violation before forwarding to Stripe. The proxy records both the successful request and the blocked duplicate in the audit log with timestamps, idempotency keys, and response status codes — giving you a complete accounting of every Stripe API call your Processing Jobs attempted across every pipeline execution, not just what Stripe processed.

Gap analysis: four additional SageMaker billing risks

1. Using the SageMaker Execution ARN as an idempotency key breaks on EventBridge duplicate delivery

SageMaker assigns a new Execution ARN to every call to StartPipelineExecution, including those triggered by EventBridge retry delivery of the same event. If your Processing Job entrypoint derives its Stripe idempotency key from the Execution ARN — via an environment variable passed from the pipeline parameter — two concurrent executions of the same billing period produce two different Execution ARNs and therefore two different idempotency keys. Both charges proceed unimpeded at Stripe. The correct key derivation uses stable billing event identity: customer_id + amount_cents + billing_period — values that are identical across both executions regardless of their Execution ARNs.

2. SageMaker Pipelines step caching returns a stale vault key after a charge refund

SageMaker Pipelines step caching reuses the output of a completed step if the input configuration is identical to a previous execution. If your IssueVaultKeysStep is cache-enabled and the pipeline is re-run with the same BillingPeriod parameter after a billing period's charges have been refunded or disputed, SageMaker returns the cached vault key output from the original run — vault keys that have already been consumed and whose spend caps are exhausted. The billing Processing Jobs then fail with 402 Policy Violation from the proxy when attempting legitimate re-billing. Disable caching on your IssueVaultKeysStep by setting cache_config=CacheConfig(enable_caching=False); vault key issuance must always produce fresh keys for each execution.

3. Spot Managed instance interruption leaves no checkpoint: processing resumes from line 1

SageMaker Processing Jobs support Spot Managed instances via managed_spot_training=True on the Processor configuration. When a Spot instance is interrupted (capacity withdrawn by AWS), SageMaker terminates the running container without a checkpoint. If RetryPolicy is configured and the interrupted job had already called stripe.charges.create() before the interruption, the retry launches a new container that starts from line 1 — calling stripe.charges.create() again. Spot interruption is not distinguishable from application failure at the SageMaker retry layer; both trigger a retry. Content-hash idempotency keys are the primary protection: the Stripe idempotency key is stable between the interrupted run and the retry because it is derived from inputs, not from the instance ID or start time.

4. SageMaker Feature Store put_record() SDK retries compound billing retry risk

SageMaker Feature Store is sometimes used as the charge record store in ML-native billing pipelines, with put_record() called after the Stripe charge to write the charge ID and amount as feature values. The SageMaker Python SDK's Feature Store client applies its own automatic retry logic on transient ValidationError and ServiceUnavailable responses. If the charge is written to Feature Store successfully but the Feature Store write acknowledgment is not received by the container before a network interruption causes the container to exit, two separate retry layers operate independently: the Feature Store SDK retries the put_record() call (no risk — Feature Store writes are idempotent on primary key), while the SageMaker Pipelines RetryPolicy retries the entire step container (risk — stripe.charges.create() fires again). These are additive, not exclusive — Feature Store's own retry does not suppress the container-level retry. Both layers must be addressed independently.

Enforcement tests

# test_sagemaker_billing.py
import hashlib
import pytest
from unittest.mock import MagicMock, patch

def billing_idempotency_key(customer_id: str, amount_cents: int, billing_period: str) -> str:
    return hashlib.sha256(
        f"{customer_id}:{amount_cents}:{billing_period}:sagemaker-billing".encode()
    ).hexdigest()[:32]

def test_idempotency_key_is_stable_across_retries():
    """Same inputs always produce the same idempotency key."""
    key1 = billing_idempotency_key("cus_123", 2999, "2026-07")
    key2 = billing_idempotency_key("cus_123", 2999, "2026-07")
    assert key1 == key2
    assert len(key1) == 32

def test_idempotency_key_differs_across_customers():
    """Different customers always produce different keys."""
    key1 = billing_idempotency_key("cus_AAA", 2999, "2026-07")
    key2 = billing_idempotency_key("cus_BBB", 2999, "2026-07")
    assert key1 != key2

def test_pre_flight_check_skips_stripe_call_when_charge_exists():
    """Pre-flight DB check returns early without calling Stripe."""
    mock_table = MagicMock()
    mock_table.get_item.return_value = {
        "Item": {"customer_id": "cus_123", "billing_period": "2026-07", "charge_id": "ch_A"}
    }
    with patch("boto3.resource") as mock_boto:
        mock_boto.return_value.Table.return_value = mock_table
        with patch("stripe.StripeClient") as mock_stripe:
            # processor logic: if item found, return early
            item = mock_table.get_item(
                Key={"customer_id": "cus_123", "billing_period": "2026-07"}
            ).get("Item")
            if item:
                result = {"status": "already_charged", "charge_id": item["charge_id"]}
            mock_stripe.assert_not_called()
    assert result["status"] == "already_charged"

def test_card_error_returns_dict_not_raises():
    """CardError is caught and returned as a dict so the container exits 0."""
    import stripe
    mock_client = MagicMock()
    mock_client.charges.create.side_effect = stripe.error.CardError(
        "Your card was declined.", None, "card_declined"
    )
    try:
        mock_client.charges.create(params={}, options=None)
        result = {"status": "charged"}
    except stripe.error.CardError as e:
        result = {"status": "declined", "code": e.code}
    assert result["status"] == "declined"
    assert result["code"] == "card_declined"

def test_vault_key_cap_is_amount_plus_ten_percent():
    """Vault key daily cap is always amount_cents * 1.10."""
    amount_cents = 2999
    expected_cap = round((amount_cents / 100) * 1.10, 2)
    assert expected_cap == 32.99

    amount_cents_large = 99900
    expected_cap_large = round((amount_cents_large / 100) * 1.10, 2)
    assert expected_cap_large == 1098.9

FAQ

Can I use the SageMaker pipeline execution ARN as my Stripe idempotency key?

No. Each call to StartPipelineExecution produces a unique Execution ARN, including duplicate invocations from EventBridge at-least-once delivery. Using the ARN as the idempotency key means two concurrent executions of the same billing period each produce a distinct Stripe idempotency key — both charges proceed. Use values that are identical across both executions: customer_id, amount_cents, and billing_period derived from the pipeline parameter, not from the execution identity.

How do vault keys work with SageMaker IAM execution roles?

SageMaker Processing Jobs run under the IAM execution role you specify on the Processor. Vault keys from Keybrake are HTTP bearer tokens, not IAM credentials — they are passed as environment variables to the container, not as AWS IAM role references. Your execution role still needs IAM permissions to read from S3 (vault key outputs from the IssueVaultKeysStep), write to DynamoDB (charge records), and access SSM/Secrets Manager (the Keybrake API key used to issue vault keys). The vault key itself requires no IAM configuration; it is validated by the Keybrake proxy, not by AWS.

What happens if the IssueVaultKeysStep fails before the billing fan-out starts?

If the vault key issuance step fails, SageMaker marks the IssueVaultKeysStep as Failed. Because the ParallelStep fan-out has depends_on=[issue_vault_keys_step], SageMaker does not execute any of the billing Processing Jobs. The pipeline execution stops with a clear failure at the issuance step — no Stripe calls are made. The Keybrake API is highly available; transient failures can be retried by adding a RetryPolicy on the issuance step itself with a small retry count, since vault key issuance has no side effects at Stripe.

How do I size vault key TTLs for large ParallelStep fan-outs?

Set the vault key TTL to the expected duration of the entire billing fan-out plus a safety margin. For example, if your ParallelStep runs 50 concurrent Processing Jobs that each take 3–5 minutes, the fan-out completes within 10 minutes in the typical case but could take up to 20 minutes if some jobs experience instance startup delays. Issue vault keys with expires_in_seconds: 7200 (2 hours) to cover the startup delay, actual billing run, and DynamoDB write time for the slowest job in the cohort. Vault keys that expire before the job completes cause the Stripe call to fail with a 401 Unauthorized from the proxy — which triggers the SageMaker RetryPolicy and the full retry path. Set TTLs conservatively.

Does the proxy work with SageMaker VPC mode and PrivateLink?

SageMaker Processing Jobs running in VPC mode (with network_config=NetworkConfig(enable_network_isolation=False, subnets=[...], security_group_ids=[...])) can reach the Keybrake proxy at proxy.keybrake.com if the VPC has outbound HTTPS access via a NAT Gateway or a VPC endpoint. With enable_network_isolation=True, all outbound network access is blocked — neither the Keybrake proxy nor the direct Stripe API is reachable. Do not enable network isolation for billing Processing Jobs that must call Stripe. For fully private deployments, contact us about a private proxy endpoint.

How does SageMaker Pipelines step caching interact with vault keys?

Disable caching on the IssueVaultKeysStep by setting CacheConfig(enable_caching=False). Step caching is appropriate for deterministic preprocessing steps that produce the same output for the same input, but vault keys have short TTLs and are consumed — a cached vault key output from a previous pipeline execution will reference keys that are either expired or already exhausted from the original billing run. If caching is enabled and the pipeline is re-run with identical parameters (e.g., a re-billing after a refund), the cached vault keys will return 401 or 402 errors when the billing Processing Jobs attempt to use them. Caching billing-related steps is always incorrect.