Kedro Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Kedro is an open-source Python framework from QuantumBlack (McKinsey) that brings software engineering discipline to data science and ML pipelines — wrapping Python functions as typed node objects, connecting them by matching input/output names into a Pipeline DAG, and managing data I/O through a DataCatalog. Teams use it for feature engineering, model training pipelines, and increasingly for orchestrating AI agent workflows where the same input→node→output model maps cleanly onto agent steps that call external APIs. Its execution model — full pipeline re-run on any failure (no native resume), parallel modular pipeline instances via ParallelRunner, and a cross-cutting Hooks system for lifecycle callbacks — introduces three specific Stripe billing failure modes: a pipeline re-run after a downstream node failure re-executes the billing node from the start; concurrent namespaced pipeline instances running under ParallelRunner share one process-level stripe.api_key with no per-instance spend cap; and an on_node_error Hook implementing automatic node-level retry re-invokes the billing node's callable directly without a stable idempotency key after a transient timeout.

This post covers all three failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-execution vault keys via a spend-cap proxy — that eliminates all three without restructuring your Kedro pipeline.

Failure mode 1: pipeline re-run after a downstream node failure re-executes the billing node from the start

Kedro's runner implementations — SequentialRunner, ParallelRunner, ThreadRunner — execute pipeline nodes in dependency order. When any node raises an exception, the runner propagates the exception to the caller and terminates the run. Kedro has no built-in mechanism to resume a partially completed pipeline from the failed node: there is no checkpoint system, no per-node state persistence, and no way to tell the runner to skip nodes whose outputs already exist in the catalog. When the downstream node — for example, save_charge_record, which writes the charge ID returned by the billing node to a database dataset — raises a DatasetError or a database timeout after the billing node has already called stripe.charges.create() and received a charge ID, the runner raises and the pipeline is marked as failed. An operator re-running the pipeline with kedro run --pipeline=billing or a scheduler retrying the failed run triggers a completely fresh execution from the first node. The billing node runs again, calls stripe.charges.create() a second time with the same inputs, and Stripe creates a second charge for the same customer.

# nodes/billing.py — UNSAFE: no idempotency key, pipeline re-run double-charges
import stripe
import os

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]  # unrestricted live key

def charge_customer(
    customer_id: str,
    amount_cents: int,
    billing_period: str,
) -> dict:
    # Pipeline re-run starts here — no idempotency key, Stripe creates ch_B on retry
    charge = stripe.charges.create(
        amount=amount_cents,
        currency="usd",
        customer=customer_id,
        description=f"Subscription {billing_period}",
    )
    return {"charge_id": charge.id, "status": charge.status}

def save_charge_record(charge_customer: dict, db_conn: object) -> bool:
    # If this raises (DB timeout, constraint violation), caller re-runs entire pipeline
    db_conn.execute(
        "INSERT INTO charges (customer_id, charge_id, period) VALUES (?, ?, ?)",
        (charge_customer["customer_id"], charge_customer["charge_id"], "2026-07"),
    )
    return True

# pipeline.py — billing pipeline definition
from kedro.pipeline import node, pipeline

def create_pipeline(**kwargs):
    return pipeline([
        node(
            func=charge_customer,
            inputs=["customer_id", "amount_cents", "billing_period"],
            outputs="charge_customer",
            name="charge_customer_node",
        ),
        node(
            func=save_charge_record,
            inputs=["charge_customer", "params:db_conn"],
            outputs="save_status",
            name="save_charge_record_node",
        ),
    ])

# $ kedro run --pipeline=billing
# save_charge_record raises → pipeline fails → re-run → charge_customer runs again
# → stripe.charges.create() called twice → ch_A and ch_B both created

The fix is to derive a content-hash idempotency key from the billing inputs inside charge_customer and include it with every stripe.charges.create() call. The inputs — customer_id, amount_cents, billing_period — are Kedro catalog parameters that are identical on every run of the same logical billing job. The hash is therefore stable across every pipeline re-run. Stripe's idempotency layer returns the original charge object on the second call without creating a new charge.

# nodes/billing.py — SAFE: content-hash idempotency key survives pipeline re-runs
import stripe
import hashlib
import os

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

def _billing_idempotency_key(
    customer_id: str, amount_cents: int, billing_period: str
) -> str:
    raw = f"{customer_id}:{amount_cents}:{billing_period}:kedro-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def charge_customer(
    customer_id: str,
    amount_cents: int,
    billing_period: str,
) -> dict:
    idempotency_key = _billing_idempotency_key(customer_id, amount_cents, billing_period)
    charge = stripe.charges.create(
        amount=amount_cents,
        currency="usd",
        customer=customer_id,
        description=f"Subscription {billing_period}",
        idempotency_key=idempotency_key,  # stable across all pipeline re-runs
    )
    return {
        "charge_id": charge.id,
        "customer_id": customer_id,
        "status": charge.status,
        "idempotency_key": idempotency_key,
    }

def save_charge_record(charge_customer: dict, db_conn: object) -> bool:
    db_conn.execute(
        "INSERT OR IGNORE INTO charges (customer_id, charge_id, period, idem_key) "
        "VALUES (?, ?, ?, ?)",
        (
            charge_customer["customer_id"],
            charge_customer["charge_id"],
            "2026-07",
            charge_customer["idempotency_key"],
        ),
    )
    return True

The INSERT OR IGNORE in save_charge_record is a companion fix: on a pipeline re-run where the charge already exists in the database (perhaps the timeout occurred during the database write after the row was partially committed), the insert silently succeeds rather than raising a unique-constraint violation that would cause the pipeline to fail again. The combination of Stripe-level idempotency and database-level deduplication makes the pipeline safe to re-run any number of times.

Failure mode 2: ParallelRunner concurrent namespaced pipeline instances share one unrestricted stripe.api_key

Kedro's modular pipeline system lets you instantiate the same pipeline template multiple times with namespace prefixes, then assemble them into a combined pipeline that runs all instances in a single kedro run call. A common pattern for batch billing is to create one billing pipeline instance per customer cohort, assemble them under separate namespaces, and run the combined pipeline with ParallelRunner so all cohorts are billed concurrently. Under ParallelRunner, independent nodes (nodes with no shared dependencies) are dispatched to a multiprocessing.pool.Pool, each in a child process, or to threads via ThreadRunner. In all cases, the child processes or threads that execute charge_customer read stripe.api_key from the environment — the same unrestricted live key inherited by every process from the parent. There is no per-namespace or per-instance key isolation. If the catalog parameters for one namespace contain a data-entry error — for example, cohort_a.amount_cents: 49999 where 4999 was intended — the billing node for cohort A fires a $499.99 charge against every customer in that cohort. Because all cohorts execute concurrently and share one unrestricted key, there is no circuit breaker: by the time the first error surfaces in cohort A's results, cohorts B through N have already submitted their own charges against the same unrestricted key, and there is no spend cap to halt them.

# pipeline_registry.py — UNSAFE: namespaced instances share one unrestricted key
from kedro.pipeline import pipeline as make_pipeline
from billing_pipeline import create_billing_pipeline

COHORTS = ["cohort_a", "cohort_b", "cohort_c"]

def register_pipelines():
    # Each namespace instance shares stripe.api_key from environment — no per-cohort cap
    billing_pipelines = [
        make_pipeline(create_billing_pipeline(), namespace=cohort)
        for cohort in COHORTS
    ]
    combined = sum(billing_pipelines, start=make_pipeline([]))
    return {"billing": combined}

# $ kedro run --pipeline=billing --runner=ParallelRunner
# All 3 cohort billing nodes run concurrently, sharing one unrestricted key
# Data-entry error in cohort_a.amount_cents → wrong charge to all cohort A customers
# No cap stops cohort_b and cohort_c from also charging with the same key

# conf/base/parameters_billing.yml — cohort parameters (UNSAFE: no vault key per cohort)
cohort_a:
  customer_ids: ["cus_a1", "cus_a2", "cus_a3"]
  amount_cents: 49999  # typo: should be 4999
  billing_period: "2026-07"

cohort_b:
  customer_ids: ["cus_b1", "cus_b2"]
  amount_cents: 4999
  billing_period: "2026-07"

The fix is to issue a per-cohort vault key before each namespaced pipeline instance runs and inject it as a catalog parameter for that namespace. Each vault key is scoped to POST /v1/charges and capped at the cohort's expected total billing amount plus a 10% buffer. A data-entry error that produces a $499.99 charge where $49.99 was expected exhausts cohort A's vault key cap on the very first Stripe call — the proxy blocks the oversized charge and returns a 402 error. Because each namespace has its own vault key, the cap failure for cohort A does not affect cohort B or cohort C: each cohort's billing node independently fails fast against its own per-cohort cap. The consistent data-entry error fails every customer in cohort A before a single charge is created.

# pipeline_registry.py — SAFE: per-cohort vault keys issued before parallel dispatch
import httpx
import os

KEYBRAKE_ADMIN_KEY = os.environ["KEYBRAKE_ADMIN_KEY"]
KEYBRAKE_PROXY_URL = os.environ.get("KEYBRAKE_PROXY_URL", "https://proxy.keybrake.com")

def issue_vault_key(cohort: str, customer_count: int, amount_cents: int) -> str:
    """Issue a vault key capped at cohort total × 1.10."""
    resp = httpx.post(
        f"{KEYBRAKE_PROXY_URL}/keys",
        headers={"Authorization": f"Bearer {KEYBRAKE_ADMIN_KEY}"},
        json={
            "label": f"kedro-{cohort}-{billing_period}",
            "vendor": "stripe",
            "daily_usd_cap": round(customer_count * amount_cents * 1.10 / 100, 2),
            "allowed_endpoints": ["POST /v1/charges"],
            "expires_in_seconds": 3600,
        },
    )
    resp.raise_for_status()
    return resp.json()["vault_key"]

def register_pipelines():
    cohort_configs = {
        "cohort_a": {"customer_count": 3, "amount_cents": 4999},
        "cohort_b": {"customer_count": 2, "amount_cents": 4999},
        "cohort_c": {"customer_count": 5, "amount_cents": 9999},
    }

    # Issue all vault keys before ParallelRunner dispatches any node
    # If issuance fails for any cohort, abort before any charge is attempted
    vault_keys = {
        cohort: issue_vault_key(cohort, cfg["customer_count"], cfg["amount_cents"])
        for cohort, cfg in cohort_configs.items()
    }

    billing_pipelines = [
        make_pipeline(
            create_billing_pipeline(),
            namespace=cohort,
            parameters={f"{cohort}.vault_key": vault_keys[cohort]},
        )
        for cohort in cohort_configs
    ]
    combined = sum(billing_pipelines, start=make_pipeline([]))
    return {"billing": combined}

# nodes/billing.py — SAFE: uses per-cohort vault key from catalog parameter
import stripe
import hashlib

def charge_customer(
    customer_id: str,
    amount_cents: int,
    billing_period: str,
    vault_key: str,  # injected from catalog parameter, unique per cohort
) -> dict:
    idempotency_key = hashlib.sha256(
        f"{customer_id}:{amount_cents}:{billing_period}:kedro-billing".encode()
    ).hexdigest()[:32]

    # Each cohort's billing node uses its own vault key — capped at cohort total × 1.10
    client = stripe.Stripe(
        api_key=vault_key,
        base_url=f"{KEYBRAKE_PROXY_URL}/stripe",
    )
    charge = client.charges.create(
        amount=amount_cents,
        currency="usd",
        customer=customer_id,
        description=f"Subscription {billing_period}",
        idempotency_key=idempotency_key,
    )
    return {"charge_id": charge.id, "customer_id": customer_id, "status": charge.status}

Using a stripe.Stripe(api_key=vault_key) client instance inside the node function rather than mutating stripe.api_key at module level is important for thread safety. Under ThreadRunner (which uses Python threads in a single process rather than subprocesses), module-level mutation of the stripe singleton is not safe across concurrent threads. Per-call client instances are thread-safe by design and make key scoping explicit in the function signature.

Failure mode 3: on_node_error Hook implementing retry re-fires the billing node without an idempotency key

Kedro's Hooks system provides lifecycle callbacks at the pipeline and node level. The on_node_error hook is called whenever a node raises during execution and receives the exception, the node object, the catalog, and the resolved inputs for that node's run. A common production pattern is a retry hook that implements automatic node-level retry for transient errors — APIConnectionError, requests.exceptions.Timeout, or any IOError — by re-invoking the node's callable directly with the same inputs. When this hook wraps the billing node and a network timeout occurs after stripe.charges.create() has already submitted the HTTP request but before the server's 200 response arrives at the client, the hook classifies the Timeout as a transient error and re-calls the node function. The re-call reaches Stripe without a stable idempotency key — either because the original function never included one, or because the hook reconstructed the call with different state than the original invocation — and Stripe creates a second charge. Unlike the pipeline re-run scenario, this failure happens within a single kedro run invocation: both charges are created before the pipeline's final status is determined.

# hooks/retry_hook.py — UNSAFE: retries billing node on timeout without idempotency key
import time
import logging
from kedro.framework.hooks import hook_impl
from kedro.pipeline.node import Node

logger = logging.getLogger(__name__)

class RetryOnTransientErrorHook:
    """Retry any node up to 3 times on transient network errors."""

    TRANSIENT_ERRORS = (TimeoutError, ConnectionError, OSError)
    MAX_RETRIES = 3

    @hook_impl
    def on_node_error(
        self,
        error: Exception,
        node: Node,
        catalog,
        inputs: dict,
        is_async: bool,
        run_id: str,
    ):
        if not isinstance(error, self.TRANSIENT_ERRORS):
            return  # re-raise non-transient errors

        logger.warning("Transient error in node %s, retrying...", node.name)
        for attempt in range(self.MAX_RETRIES):
            try:
                time.sleep(2 ** attempt)
                # Re-calls node function with original inputs — no idempotency key
                result = node.func(**inputs)
                # Kedro does not allow hooks to return results; result is discarded
                logger.info("Node %s recovered on attempt %d", node.name, attempt + 1)
                return
            except self.TRANSIENT_ERRORS:
                if attempt == self.MAX_RETRIES - 1:
                    raise
                continue

# settings.py — hook registered globally, applies to ALL nodes including billing
from hooks.retry_hook import RetryOnTransientErrorHook

HOOKS = (RetryOnTransientErrorHook(),)

The failure is subtle for two reasons. First, the on_node_error hook does not have the ability to update the node's output in the catalog: Kedro hooks are side-effect callbacks, not result interceptors. The hook can re-run the function and observe the result, but cannot inject it back into the pipeline. This means the retry in the hook is partially futile from a pipeline-correctness standpoint — the runner still considers the node failed and will propagate the original error — but the Stripe call is made regardless, creating a second charge as a side effect of the retry attempt. Second, the retry happens inside a single pipeline run, so idempotency keys based on the run ID will not help: both the original call and the hook's retry call share the same run ID.

The fix is two-part. First, update the billing node to always derive a stable content-hash idempotency key from billing inputs — independent of run ID or any runtime-generated value — so that even if the hook retries the node, Stripe's idempotency cache returns the same charge object on every attempt. Second, explicitly exclude billing nodes from the retry hook by filtering on node name or a tag, since billing node retries carry inherent risk that is best handled at the Stripe level via idempotency, not at the infrastructure level via generic retry.

# hooks/retry_hook.py — SAFE: skips billing nodes; billing node handles its own safety
import time
import logging
from kedro.framework.hooks import hook_impl
from kedro.pipeline.node import Node

logger = logging.getLogger(__name__)

BILLING_NODE_NAMES = {"charge_customer_node"}  # excluded from automatic retry

class RetryOnTransientErrorHook:
    """Retry non-billing nodes on transient errors. Billing nodes are excluded."""

    TRANSIENT_ERRORS = (TimeoutError, ConnectionError, OSError)
    MAX_RETRIES = 3

    @hook_impl
    def on_node_error(
        self,
        error: Exception,
        node: Node,
        catalog,
        inputs: dict,
        is_async: bool,
        run_id: str,
    ):
        if node.name in BILLING_NODE_NAMES:
            # Billing node runs exactly once per pipeline execution.
            # The content-hash idempotency key in the node handles Stripe safety.
            # A pipeline-level re-run (new kedro run) is the correct retry path.
            logger.warning(
                "Billing node %s failed — not retrying via hook. "
                "Re-run the pipeline to retry with idempotency key protection.",
                node.name,
            )
            return  # propagate original error to runner

        if not isinstance(error, self.TRANSIENT_ERRORS):
            return  # re-raise non-transient errors

        for attempt in range(self.MAX_RETRIES):
            try:
                time.sleep(2 ** attempt)
                node.func(**inputs)
                logger.info("Node %s recovered on attempt %d", node.name, attempt + 1)
                return
            except self.TRANSIENT_ERRORS:
                if attempt == self.MAX_RETRIES - 1:
                    raise
                continue

With the billing node excluded from automatic retry and the content-hash idempotency key present in the node function, the failure path converges safely. A timeout after stripe.charges.create() submits the charge causes the hook to propagate the error, the runner marks the pipeline failed, and an operator re-runs the pipeline. The re-run executes the billing node again with the same inputs, producing the same idempotency key, and Stripe returns the original charge object. The pipeline completes without creating a second charge, and save_charge_record deduplicates against the existing database row.

Governance approach comparison

Gap analysis: four Kedro billing edge cases the main patterns don't cover

1. kedro run --from-nodes=charge_customer_node partial re-run re-charges customers after a downstream failure. Kedro's CLI supports running a pipeline from a named node, which is useful for re-running only the failed portion of a pipeline. When an operator uses --from-nodes=charge_customer_node to re-run from the billing node after a downstream failure, the billing node executes again regardless of whether it completed successfully in the prior run. A content-hash idempotency key handles this correctly as long as the billing inputs are identical. The risk arises when the operator also changes a parameter (for example, adjusting billing_period to isolate a specific month) before the partial re-run: the modified input produces a different idempotency key, and Stripe creates a new charge. The fix: never change billing inputs on a partial re-run intended as a retry. Use --from-nodes=save_charge_record_node when only the downstream node needs re-running, and inject the original charge ID from the database as a catalog parameter to skip the Stripe call entirely.

2. Kedro MemoryDataset caching stale charge IDs across pipeline reruns in long-running processes. When Kedro pipelines are embedded in long-running services (FastAPI, Celery, a custom daemon), catalog datasets marked as MemoryDataset persist across pipeline runs within the same process. A MemoryDataset holding the output of charge_customer from a previous run will cause the runner to skip the billing node entirely on the next run (the dataset is already populated), returning the cached charge ID from the prior billing period. Downstream nodes act on the stale charge ID, leading to incorrect accounting — revenue booked twice for the old period, none for the new period. The fix: explicitly clear or version MemoryDataset entries for billing nodes between pipeline runs, or use PartitionedDataset with period-keyed partitions so each billing run writes to a distinct partition.

3. kedro run --env=production credential overlay accidentally using a test vault key against the live Stripe environment. Kedro's configuration loader resolves credentials in order: base/ then {env}/, with later entries overriding earlier ones. If a vault_key credential is defined in conf/base/credentials.yml (pointing to a test proxy key) and not overridden in conf/production/credentials.yml, the production run uses the test key. The test key may be scoped to a test Stripe account with test-mode charge limits: all charges succeed in Stripe's test mode regardless of amount, bypassing the spend cap enforcement that the production vault key would provide. The fix: assert that the vault_key credential in the active environment is a production key (prefix check: vk_live_ vs vk_test_) inside a before_pipeline_run hook, and abort the run if a test key is detected in a production environment.

4. KedroSession.run() in a Jupyter notebook triggering concurrent pipeline runs via cell re-execution. Kedro's KedroSession context manager is increasingly used in Jupyter notebooks for interactive pipeline development. When a notebook cell containing session.run(pipeline_name="billing") is executed while a prior execution is still running (possible in VS Code's multi-kernel mode or JupyterLab with non-blocking cell execution), two independent pipeline instances run concurrently. Both reach the billing node with the same inputs and the same content-hash idempotency key. Stripe's idempotency layer handles this correctly if the keys are identical and the first request has already returned — Stripe returns the cached result to the second caller. However, if both requests arrive at Stripe simultaneously before either has returned, Stripe may process both and return an IdempotencyError to one of them, causing that pipeline run to fail. The fix: use KedroSession with a notebook-level lock or ensure only one billing pipeline run is active at any time by checking for in-progress runs before dispatching a new one.

Enforcement: five pytest tests for Kedro billing pipelines

import hashlib
import pytest
from unittest.mock import MagicMock, patch, call
from kedro.runner import SequentialRunner
from kedro.io import DataCatalog, MemoryDataset
from kedro.pipeline import node, pipeline as make_pipeline

from nodes.billing import charge_customer, _billing_idempotency_key
from hooks.retry_hook import RetryOnTransientErrorHook

def make_key(customer_id, amount_cents, billing_period):
    raw = f"{customer_id}:{amount_cents}:{billing_period}:kedro-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def test_idempotency_key_stable_across_pipeline_reruns():
    """Same billing inputs always produce the same idempotency key."""
    key1 = _billing_idempotency_key("cus_001", 4999, "2026-07")
    key2 = _billing_idempotency_key("cus_001", 4999, "2026-07")
    assert key1 == key2
    assert len(key1) == 32

def test_idempotency_key_distinct_per_customer_and_period():
    """Different customers and periods produce distinct idempotency keys."""
    key_a = _billing_idempotency_key("cus_001", 4999, "2026-07")
    key_b = _billing_idempotency_key("cus_002", 4999, "2026-07")
    key_c = _billing_idempotency_key("cus_001", 4999, "2026-08")
    assert key_a != key_b
    assert key_a != key_c
    assert key_b != key_c

def test_vault_key_cap_blocks_oversized_charge():
    """Vault key issued at amount × 1.10 rejects a 100× unit-bug charge."""
    from pipeline_registry import issue_vault_key

    with patch("httpx.post") as mock_post:
        mock_post.return_value = MagicMock(
            status_code=200,
            json=lambda: {"vault_key": "vk_live_test_abc"},
        )
        issue_vault_key("cohort_a", customer_count=3, amount_cents=4999)
        call_json = mock_post.call_args.kwargs["json"]
        # cap = 3 customers × $49.99 × 1.10 = $164.97
        assert call_json["daily_usd_cap"] == pytest.approx(3 * 4999 * 1.10 / 100, rel=0.01)
        assert call_json["allowed_endpoints"] == ["POST /v1/charges"]

def test_retry_hook_excludes_billing_node():
    """RetryOnTransientErrorHook does not retry charge_customer_node on timeout."""
    hook = RetryOnTransientErrorHook()
    call_count = 0

    def failing_billing_fn(**kwargs):
        nonlocal call_count
        call_count += 1
        raise TimeoutError("network timeout after charge submitted")

    node_mock = MagicMock()
    node_mock.name = "charge_customer_node"
    node_mock.func = failing_billing_fn

    with pytest.raises(TimeoutError):
        hook.on_node_error(
            error=TimeoutError("network timeout"),
            node=node_mock,
            catalog=MagicMock(),
            inputs={"customer_id": "cus_001", "amount_cents": 4999},
            is_async=False,
            run_id="run_abc",
        )

    assert call_count == 0  # hook does not retry billing node — zero extra calls

def test_billing_node_sends_idempotency_key_to_stripe():
    """charge_customer passes stable idempotency key to stripe.charges.create."""
    with patch("stripe.charges.create") as mock_create:
        mock_create.return_value = MagicMock(id="ch_test_123", status="succeeded")

        with patch("stripe.Stripe") as mock_stripe_cls:
            mock_client = MagicMock()
            mock_client.charges.create.return_value = MagicMock(
                id="ch_test_123", status="succeeded"
            )
            mock_stripe_cls.return_value = mock_client

            result = charge_customer(
                customer_id="cus_001",
                amount_cents=4999,
                billing_period="2026-07",
                vault_key="vk_live_test_abc",
            )
            call_kwargs = mock_client.charges.create.call_args.kwargs
            assert call_kwargs["idempotency_key"] == make_key("cus_001", 4999, "2026-07")
            assert result["charge_id"] == "ch_test_123"

Frequently asked questions

Can I use Kedro's --from-nodes flag to skip the billing node on a re-run?

Yes — kedro run --pipeline=billing --from-nodes=save_charge_record_node starts execution from the downstream node, skipping charge_customer_node entirely. For this to work, the output of the billing node — the charge_customer dict — must already exist in a persistent catalog dataset (not a MemoryDataset) from the prior run. A JSONDataset or database dataset persisting the charge result lets you resume the pipeline from any point after billing. This is the safest retry path when you are certain the charge was already created and only the downstream step needs re-running.

How do per-cohort vault keys interact with Kedro's configuration layers?

Issue vault keys programmatically at pipeline assembly time (in register_pipelines() or a before_pipeline_run hook), not from static config files. Vault keys are short-lived credentials that expire after the pipeline's execution window; storing them in conf/ would require regenerating them before every run, defeating the purpose of the config layer. Pass them as runtime parameters injected into the catalog via catalog.add("vault_key", MemoryDataset(vault_key)) inside a before_pipeline_run hook so they are available to billing nodes without appearing in any config file that might be committed to version control.

What happens if the Keybrake proxy is unreachable when a Kedro billing node executes?

The stripe.Stripe(base_url=proxy_url) client will fail with a connection error because the proxy is the configured base URL. This causes the billing node to raise, the runner marks the pipeline failed, and no charge is created. This is the correct behavior: billing should not proceed when the governance proxy is unavailable. Do not configure a fallback to api.stripe.com directly — the fallback bypasses the spend cap and defeats the entire governance layer. Re-run the pipeline after the proxy recovers; the content-hash idempotency key ensures the charge is only created once.

How should I handle vault key issuance failures in register_pipelines()?

Fail immediately and loudly: if issue_vault_key() raises for any cohort, let the exception propagate and abort the pipeline assembly before any node executes. This is the safest behavior — it prevents a partial run where some cohorts have vault keys and others fall back to an unrestricted key. Log the failure with the cohort name and the HTTP status code from the proxy. In orchestrators like Airflow or Prefect, the pipeline assembly failure will cause the task to fail immediately, triggering the scheduler's retry logic or an alert, before any Stripe charge is attempted.

Does ThreadRunner vs ParallelRunner change the key isolation approach?

The vault key approach is the same for both runners, but the isolation mechanism differs. Under ParallelRunner (subprocess pool), each worker process has its own memory space, so module-level stripe.api_key mutations in one worker do not affect others. Under ThreadRunner (thread pool in one process), module-level state is shared across all threads — a thread setting stripe.api_key = vault_key_a can race with another thread reading it as vault_key_b. For ThreadRunner, always use per-call stripe.Stripe(api_key=vault_key) client instances rather than mutating the module-level singleton. The vault key cap enforcement at the proxy layer is identical in both cases.

Can I use Kedro tags to identify and protect billing nodes across multiple pipelines?

Yes — Kedro nodes accept a tags argument: node(func=charge_customer, ..., tags=["billing"]). Your RetryOnTransientErrorHook can check node.tags instead of node.name to exclude any node tagged billing from automatic retry. This approach scales better than maintaining a hardcoded set of node names: as you add more billing nodes across multiple pipelines, tagging them with billing automatically excludes them from retry without updating the hook. Tags also allow kedro run --tags=billing to run only billing nodes in isolation for testing or manual re-execution.