Google Cloud Workflows Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Google Cloud Workflows is a serverless orchestration service that chains Cloud Functions, Cloud Run services, and Google Cloud APIs into durable, YAML-defined workflows. AI agent teams use Workflows to build billing pipelines with built-in retry, concurrent fan-out via parallel for, and scheduled execution via Cloud Scheduler. When those workflows include Stripe billing — per-customer subscription renewals fanned out with a parallel for loop, monthly billing triggered by a Cloud Scheduler job, or usage-based charges invoked by an AI agent step — Workflows introduces three specific failure modes that neither IAM restrictions nor Stripe restricted keys prevent on their own.

This post covers all three failure modes with Workflows YAML and Python Cloud Functions code, and the two-layer governance pattern — content-hash idempotency keys plus per-iteration vault keys via a spend-cap proxy — that eliminates all three without changing your workflow graph.

Failure mode 1: try/retry re-invokes the Cloud Function step from the beginning after a downstream failure that follows a successful Stripe charge

Google Cloud Workflows supports a try/retry block that re-executes the failing step when it raises an exception matching the configured HTTP status codes or exception types. This is correct behavior for transient failures — a Cloud Function cold-start timeout, a Firestore unavailability, a Cloud Run connection reset — where re-invoking the step from the beginning is safe. The failure mode occurs when a Cloud Function called from the workflow invokes stripe.charges.create(), that call succeeds and returns a charge object, and then a subsequent operation within the same function (writing the charge record to Firestore or Cloud SQL) raises an exception. The Cloud Function throws; the Workflows runtime sees the exception, matches it against the retry block, and re-invokes the function by repeating the HTTP call. The re-invocation starts the Cloud Function handler from the top. stripe.charges.create() fires again with no idempotency key, and Stripe creates a second charge for the same customer.

With max_retries: 3 and backoff.multiplier: 2, a Firestore write that intermittently raises a deadline-exceeded error after a successful Stripe charge can produce up to three charges before Workflows exhausts the retry budget. The Workflows execution graph shows each retry attempt as a separate HTTP call node, each marked as failed because the function threw, but Stripe shows two or three charges in the succeeded state. The mismatch is invisible until a customer disputes a duplicate payment.

# charge_customer/main.py — UNSAFE: no idempotency key
# downstream Firestore failure triggers duplicate charge on Workflows retry
import stripe
import os
from google.cloud import firestore

def charge_customer(request):
    data = request.get_json()
    customer_id = data["customerId"]
    amount_cents = data["amountCents"]
    billing_period = data["billingPeriod"]

    stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

    # UNSAFE: no idempotency key.
    # If db.collection().add() raises below, Workflows retries this function.
    # On retry, stripe.Charge.create() fires again → ch_B created.
    charge = stripe.Charge.create(
        amount=amount_cents,
        currency="usd",
        customer=customer_id,
        description=f"Billing {billing_period}",
    )

    # Firestore write: if this raises (deadline exceeded, unavailable),
    # the function throws → Workflows retry fires →
    # stripe.Charge.create() fires again on the next invocation.
    db = firestore.Client()
    db.collection("charges").add({
        "customerId": customer_id,
        "billingPeriod": billing_period,
        "chargeId": charge["id"],
        "status": "succeeded",
    })

    return {"chargeId": charge["id"]}, 200

The fix is a content-hash idempotency key derived from the billing inputs — customer ID, amount, and billing period — combined with a workflow-specific salt. This key is computed before the Stripe call and is identical across every Cloud Function re-invocation with the same inputs. When Workflows retries after a Firestore failure, stripe.Charge.create() fires with the same idempotency key, and Stripe returns the original charge object without creating a new one. The Firestore write becomes safe to retry.

Additionally, catch permanent Stripe errors (card declined, invalid customer) explicitly and return a non-2xx response with a structured error body rather than raising an exception. Workflows can then route these to a except branch that transitions to a failure step, rather than triggering the retry logic. Use an HTTP return status in the 4xx range for permanent errors so Workflows' default retry predicate — which retries on connection errors and 5xx responses — does not pick them up.

# charge_customer/main.py — SAFE: content-hash idempotency key + permanent-error handling
import stripe
import os
import hashlib
from google.cloud import firestore

def billing_idempotency_key(customer_id: str, amount_cents: int, billing_period: str) -> str:
    raw = f"{customer_id}:{amount_cents}:{billing_period}:gcw-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def charge_customer(request):
    data = request.get_json()
    customer_id = data["customerId"]
    amount_cents = data["amountCents"]
    billing_period = data["billingPeriod"]
    vault_key = data["vaultKey"]  # per-iteration vault key from issue_vault_keys step

    # Idempotency key is stable across every re-invocation with the same inputs
    idempotency_key = billing_idempotency_key(customer_id, amount_cents, billing_period)

    # Point the Stripe SDK at the Keybrake proxy using the per-iteration vault key
    stripe.api_key = vault_key
    stripe.api_base = "https://proxy.keybrake.com/stripe"

    try:
        charge = stripe.Charge.create(
            amount=amount_cents,
            currency="usd",
            customer=customer_id,
            description=f"Billing {billing_period}",
            idempotency_key=idempotency_key,
        )
    except stripe.error.CardError as e:
        # Permanent: return 422 so Workflows does not retry
        return {"error": "card_declined", "message": e.user_message}, 422
    except stripe.error.InvalidRequestError as e:
        return {"error": "invalid_request", "message": str(e)}, 422

    db = firestore.Client()
    db.collection("charges").add({
        "customerId": customer_id,
        "billingPeriod": billing_period,
        "chargeId": charge["id"],
        "status": "succeeded",
    })

    return {"chargeId": charge["id"]}, 200

In the Workflows YAML, configure the retry block to cover only transient HTTP and connection errors, and add an except branch that routes 422 responses to a dedicated failure step without triggering the retry predicate:

# billing_workflow.yaml — SAFE retry configuration
- charge_step:
    try:
      call: http.post
      args:
        url: ${charge_function_url}
        auth:
          type: OIDC
        body:
          customerId: ${customer.customerId}
          amountCents: ${customer.amountCents}
          billingPeriod: ${billing_period}
          vaultKey: ${customer.vaultKey}
      result: charge_result
    retry:
      predicate: ${retry_predicate}
      max_retries: 3
      backoff:
        initial_delay: 2
        max_delay: 60
        multiplier: 2
    except:
      as: e
      steps:
        - check_permanent:
            switch:
              - condition: ${e.code == 422}
                next: record_billing_failure
        - reraise:
            raise: ${e}

- retry_predicate:
    # Retry only on connection errors and 5xx responses, not on 4xx permanent errors
    params: [e]
    steps:
      - check:
          return: ${e.code == null or e.code >= 500}

Failure mode 2: parallel for fan-out shares one unrestricted Secret Manager secret across all concurrent iterations with no per-iteration spend cap

Google Cloud Workflows' parallel for block executes loop iterations concurrently. Each iteration runs as an independent branch of the workflow, but all branches read the same Secret Manager secret at the start of the workflow — the same STRIPE_SECRET_KEY resolved once and passed as a variable throughout. There is no per-iteration dollar cap in Workflows: the concurrency_limit parameter controls how many branches execute simultaneously, but does not limit how much money each iteration is permitted to charge. A data bug that passes amount_cents: 49999 (intended: 4999) to all 400 customers triggers $19,999.60 in charges across the concurrent parallel branches before any result returns to the parent workflow. Because all iterations use the same unrestricted key, a cap hit in one iteration does not abort the others — each branch executes its Cloud Function call independently, and only the overall parallel for step aggregates results when all branches complete.

# UNSAFE billing_workflow.yaml — parallel for with shared unrestricted key
main:
  params: [args]
  steps:
    - get_stripe_key:
        call: googleapis.secretmanager.v1.projects.secrets.versions.access
        args:
          name: projects/my-project/secrets/stripe-secret-key/versions/latest
        result: secret_response
    - set_key:
        assign:
          - stripe_key: ${base64.decode(secret_response.payload.data)}
          - customers: ${args.customers}
          - billing_period: ${args.billingPeriod}

    # UNSAFE: all iterations share the same stripe_key with no per-customer cap.
    # A data error in amount_cents charges all customers the wrong amount.
    - fan_out:
        parallel:
          for:
            value: customer
            in: ${customers}
            steps:
              - charge:
                  call: http.post
                  args:
                    url: ${charge_function_url}
                    body:
                      customerId: ${customer.customerId}
                      amountCents: ${customer.amountCents}
                      billingPeriod: ${billing_period}
                      stripeKey: ${stripe_key}  # same key for all 400 customers

The fix is to add an issue_vault_keys step before the parallel for block. This step calls a Cloud Function that fetches the customer list, issues one vault key per customer from the Keybrake proxy — each capped at the customer's expected charge amount plus a 10% buffer, scoped to POST /v1/charges, with a TTL that covers the parallel fan-out duration — and returns the enriched customer list with each item carrying its own vault key. The parallel for block then iterates over this enriched list. Each iteration receives a unique vault key in the step's input and uses that key in the Cloud Function call. A data bug that passes an oversized amount is caught by the per-iteration vault key cap on the first Stripe call for that customer — the proxy returns a 429 with the x-keybrake-cap-hit header before the charge is created. Because each iteration has its own key, the cap hit for one customer does not propagate to others.

# issue_vault_keys/main.py — Cloud Function called before parallel for
import os
import json
import urllib.request

KEYBRAKE_API_KEY = os.environ["KEYBRAKE_API_KEY"]
KEYBRAKE_PROXY_URL = os.environ.get("KEYBRAKE_PROXY_URL", "https://proxy.keybrake.com")

def issue_vault_keys(request):
    data = request.get_json()
    customers = data["customers"]
    billing_period = data["billingPeriod"]
    enriched = []

    for customer in customers:
        customer_id = customer["customerId"]
        amount_cents = customer["amountCents"]
        # Cap = expected charge + 10% buffer; scope to charges POST only
        daily_usd_cap = int((amount_cents * 1.1) / 100) + 1

        body = json.dumps({
            "vendor": "stripe",
            "daily_usd_cap": daily_usd_cap,
            "allowed_endpoints": ["POST /v1/charges"],
            "expires_in_seconds": 3600,
            "label": f"gcw/{billing_period}/{customer_id}",
        }).encode()

        req = urllib.request.Request(
            f"{KEYBRAKE_PROXY_URL}/vault/keys",
            data=body,
            headers={
                "Authorization": f"Bearer {KEYBRAKE_API_KEY}",
                "Content-Type": "application/json",
            },
            method="POST",
        )
        with urllib.request.urlopen(req) as resp:
            vault_key = json.loads(resp.read())["vault_key"]

        enriched.append({**customer, "vaultKey": vault_key})

    return {"customers": enriched, "billingPeriod": billing_period}, 200

# SAFE billing_workflow.yaml — issue vault keys before parallel for
main:
  params: [args]
  steps:
    - issue_keys_step:
        call: http.post
        args:
          url: ${issue_vault_keys_url}
          auth:
            type: OIDC
          body:
            customers: ${args.customers}
            billingPeriod: ${args.billingPeriod}
        result: keys_response
    - set_enriched:
        assign:
          - enriched_customers: ${keys_response.body.customers}
          - billing_period: ${args.billingPeriod}

    # SAFE: each iteration receives its own vault key capped at customer's amount + 10%
    - fan_out:
        parallel:
          concurrency_limit: 50
          for:
            value: customer
            in: ${enriched_customers}
            steps:
              - charge:
                  call: http.post
                  args:
                    url: ${charge_function_url}
                    auth:
                      type: OIDC
                    body:
                      customerId: ${customer.customerId}
                      amountCents: ${customer.amountCents}
                      billingPeriod: ${billing_period}
                      vaultKey: ${customer.vaultKey}

Failure mode 3: Cloud Scheduler at-least-once delivery fires two concurrent Workflows executions for the same billing period

Cloud Scheduler delivers jobs with at-least-once semantics. A monthly billing Cloud Scheduler job that targets a Workflows execution via an HTTP trigger can, under internal retry or delivery anomaly conditions, fire the trigger twice for the same scheduled time. Each trigger starts a new Workflows execution with a system-generated execution ID. Both executions run concurrently, each independently proceeding through the vault key issuance and parallel for fan-out steps. A second path for double-start: an operator manually triggers the workflow via the Cloud Console or gcloud workflows run while the scheduled execution is still active — intending to process a single missed customer — but inadvertently starting a full second billing run. Both executions reach the parallel for step, fetch the same customer list, issue separate vault keys per customer, and call stripe.Charge.create() for the same customers. With content-hash idempotency keys, Stripe returns the original charge for each customer — no double charge at the payment layer. But both executions complete successfully in the Workflows console, creating duplicate billing run records in Firestore and incorrect business metrics that surface in monthly reconciliation.

The fix has two parts. First, add a pre-flight check as the first step in the workflow: a call to a Cloud Function that queries Firestore for a completed billing run record for the current period. If the record exists, the workflow returns immediately with a skip result and makes no Stripe calls. Second, use Cloud Workflows' sys.log to write a uniqueness token at workflow start, and check for it before proceeding. The application-level pre-flight check is more reliable than trying to enforce execution uniqueness at the Cloud Scheduler level, because Cloud Scheduler does not natively deduplicate concurrent executions by job ID.

# preflight_check/main.py — first step in the workflow
import os
from google.cloud import firestore

def preflight_check(request):
    data = request.get_json()
    billing_period = data["billingPeriod"]

    db = firestore.Client()
    doc = db.collection("billing_runs").document(billing_period).get()
    if doc.exists and doc.to_dict().get("status") == "completed":
        return {"alreadyBilled": True, "billingPeriod": billing_period}, 200

    return {"alreadyBilled": False, "billingPeriod": billing_period}, 200

# SAFE billing_workflow.yaml — preflight check before any billing steps
main:
  params: [args]
  steps:
    - preflight:
        call: http.post
        args:
          url: ${preflight_check_url}
          auth:
            type: OIDC
          body:
            billingPeriod: ${args.billingPeriod}
        result: preflight_result
    - check_already_billed:
        switch:
          - condition: ${preflight_result.body.alreadyBilled == true}
            next: skip_billing
    - issue_keys_step:
        call: http.post
        args:
          url: ${issue_vault_keys_url}
          auth:
            type: OIDC
          body:
            customers: ${args.customers}
            billingPeriod: ${args.billingPeriod}
        result: keys_response
    - set_enriched:
        assign:
          - enriched_customers: ${keys_response.body.customers}
          - billing_period: ${args.billingPeriod}
    - fan_out:
        parallel:
          concurrency_limit: 50
          for:
            value: customer
            in: ${enriched_customers}
            steps:
              - charge:
                  call: http.post
                  args:
                    url: ${charge_function_url}
                    auth:
                      type: OIDC
                    body:
                      customerId: ${customer.customerId}
                      amountCents: ${customer.amountCents}
                      billingPeriod: ${billing_period}
                      vaultKey: ${customer.vaultKey}
    - mark_completed:
        call: http.post
        args:
          url: ${mark_completed_url}
          auth:
            type: OIDC
          body:
            billingPeriod: ${billing_period}
        result: mark_result
    - return_success:
        return: ${mark_result.body}
    - skip_billing:
        return:
          status: skipped
          billingPeriod: ${args.billingPeriod}

Approach comparison

Gap analysis: four scenarios the pattern does not fully cover

1. Workflow variable persistence across retried steps. When Workflows retries a step that sets variables (using an assign step before an HTTP call), the variable assignment re-runs on each retry. If the vault key was issued in a prior step and stored as a workflow variable, and the billing step is retried, the same vault key is reused — this is correct behavior. However, if the vault key step itself is retried (because issue_vault_keys returned 5xx), a second vault key is issued for the same customer. Both keys are valid until their TTL expires. To prevent ambiguity, the charge_customer function should accept and use the vault key passed in the step input, not re-fetch it.

2. Cloud Scheduler minimum retry interval. Cloud Scheduler retries failed job executions with a configurable retry count and minimum backoff interval. If the Workflows trigger HTTP endpoint returns 5xx (for example, because Workflows is transiently unavailable), Cloud Scheduler retries the trigger. Each retry can start a new Workflows execution. The pre-flight check covers this case as long as the Firestore billing-run record is written atomically before the workflow reaches the billing steps — but if the workflow crashes between the pre-flight check and the first issue_vault_keys call, a Scheduler retry will pass the pre-flight check and proceed to billing. Use Firestore transaction semantics to write a in_progress record before the first billing step and upgrade it to completed after the last step.

3. parallel for partial failure and retry of the whole step. If one branch of the parallel for raises an unhandled exception, Workflows marks the entire parallel for step as failed and, if the parent workflow has a retry block on that step, re-executes all branches — including the ones that already succeeded. A customer whose branch succeeded in the first execution will have their Cloud Function called again. With content-hash idempotency keys, the Stripe call returns the original charge. But if your Cloud Function writes to Firestore before returning, the duplicate write may conflict. Use Firestore conditional writes (create_time precondition) to detect and skip already-written records.

4. Workflow callback and event triggers. Google Cloud Workflows supports event-driven triggers via Eventarc. If a billing workflow is triggered by a Pub/Sub message via Eventarc, and the message is delivered more than once (Pub/Sub guarantees at-least-once delivery), multiple executions start for the same event. The pre-flight check guards against this, but the dedup key must be the billing period derived from the Pub/Sub message payload, not the Eventarc execution ID, which changes per delivery.

Enforcement test suite

# test_billing_workflow.py
import hashlib
import pytest
from unittest.mock import patch, MagicMock

def billing_idempotency_key(customer_id, amount_cents, billing_period):
    raw = f"{customer_id}:{amount_cents}:{billing_period}:gcw-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

def test_idempotency_key_stable_across_retries():
    """Same inputs must produce same key on every Cloud Function re-invocation."""
    key1 = billing_idempotency_key("cus_ABC", 4999, "2026-07")
    key2 = billing_idempotency_key("cus_ABC", 4999, "2026-07")
    assert key1 == key2

def test_idempotency_key_unique_per_customer():
    """Different customers must produce different keys within the same billing period."""
    key1 = billing_idempotency_key("cus_ABC", 4999, "2026-07")
    key2 = billing_idempotency_key("cus_XYZ", 4999, "2026-07")
    assert key1 != key2

def test_idempotency_key_unique_per_period():
    """Same customer in different billing periods must produce different keys."""
    key1 = billing_idempotency_key("cus_ABC", 4999, "2026-07")
    key2 = billing_idempotency_key("cus_ABC", 4999, "2026-08")
    assert key1 != key2

def test_card_error_returns_422_not_raises():
    """CardError must return 422 response, not raise — prevents Workflows retry."""
    import stripe
    from charge_customer.main import charge_customer

    mock_request = MagicMock()
    mock_request.get_json.return_value = {
        "customerId": "cus_ABC",
        "amountCents": 4999,
        "billingPeriod": "2026-07",
        "vaultKey": "vk_test_abc123",
    }

    with patch("stripe.Charge.create") as mock_create:
        mock_create.side_effect = stripe.error.CardError(
            "Your card was declined.", param=None, code="card_declined"
        )
        response, status_code = charge_customer(mock_request)

    assert status_code == 422
    assert response["error"] == "card_declined"

def test_preflight_skips_already_billed_period():
    """Preflight check must return alreadyBilled=True for completed periods."""
    from preflight_check.main import preflight_check
    from unittest.mock import patch, MagicMock

    mock_request = MagicMock()
    mock_request.get_json.return_value = {"billingPeriod": "2026-07"}

    mock_doc = MagicMock()
    mock_doc.exists = True
    mock_doc.to_dict.return_value = {"status": "completed"}

    with patch("google.cloud.firestore.Client") as mock_fs:
        mock_fs.return_value.collection.return_value.document.return_value.get.return_value = mock_doc
        response, status_code = preflight_check(mock_request)

    assert status_code == 200
    assert response["alreadyBilled"] is True

Frequently asked questions

Can I use the Workflows execution ID as the idempotency key instead of a content hash?

No. The execution ID changes every time Workflows starts a new execution — including on Cloud Scheduler retry, manual re-run, and any other trigger that produces a fresh execution. Two executions for the same billing period will have different IDs, so both will call Stripe with different idempotency keys and both will succeed. The content-hash approach derives the key from the business data (customer ID, amount, period), so it is stable across any number of executions that process the same billing inputs.

Does Cloud Workflows' built-in retry deduplicate at the workflow level?

No. Workflows retries individual steps, not entire executions. When a step's retry block fires, it re-executes that step's HTTP call. The step has no memory of what happened in prior attempts — there is no built-in idempotency at the step level. The step's Cloud Function is called again from the top of its handler. Idempotency keys and the pre-flight check must be implemented in your application code, not in the Workflows runtime configuration.

How long should vault key TTLs be set for a large parallel for fan-out?

Set the TTL to cover the expected duration of the parallel for block plus a buffer. For a fan-out of 500 customers with each Cloud Function call taking up to 5 seconds and concurrency_limit: 50, the total duration is approximately ceil(500 / 50) × 5s = 50 seconds. Set the TTL to at least 300 seconds (5 minutes) to account for cold starts, retry backoff, and queue depth. Vault keys that expire mid-fan-out will cause 401 errors on the Keybrake proxy for in-progress iterations, which Workflows will attempt to retry per the step's retry configuration.

What happens if the Keybrake proxy is unreachable during a fan-out iteration?

The Stripe SDK call will fail with a connection error. Workflows will retry the step per the retry block — and because the idempotency key is the same across retries, when the proxy becomes reachable the Stripe call will either succeed (first attempt) or return the original charge (subsequent attempts, if the first attempt reached Stripe before the proxy went down). The proxy is a passthrough — it does not store Stripe responses, only enforces the spend cap. Network isolation from the proxy does not lose the idempotency key.

How does the per-iteration vault key work with Workflows variable scoping?

In Google Cloud Workflows, variables in a parallel for loop body are scoped to that iteration — each iteration sees its own copy of the loop variable (customer in the examples above). The vault key is a field on each element of the enriched_customers list, so each iteration reads its own key from customer.vaultKey. There is no shared mutable state between iterations in the Workflows runtime for loop-body variables, so vault key isolation is enforced by the data structure, not by any Workflows-specific configuration.

Does this pattern work with Cloud Run jobs instead of Cloud Functions?

Yes. The Workflows YAML calls HTTP endpoints — whether those endpoints are served by Cloud Functions, Cloud Run services, or Cloud Run jobs (via the Jobs API) is transparent to the Workflows runtime. The idempotency key derivation and vault key issuance are application-level patterns implemented in your handler code, not tied to the compute substrate. Replace http.post calls to Cloud Function URLs with Cloud Run service URLs or the Cloud Run Jobs execution API as appropriate for your architecture.