Keybrake

Kestra · AI agents · API key security

Kestra AI agent API key: scoping vendor calls in YAML workflow tasks

Kestra is an open-source workflow orchestration platform — declare your flows in YAML, run Python and shell tasks inside Docker containers, and trigger on schedules, webhooks, or events. When AI agent workflows use Kestra to dispatch vendor API calls, the platform's YAML-first approach creates spending risks that are harder to spot than in code-native orchestrators: EachParallel fans out tasks for each item in a list with no per-execution vendor spend cap, automatic task retries multiply each failed vendor call into multiple charges, and re-triggering a flow from a webhook event re-executes every task — including those that already made the vendor call. This page covers the vault-key pattern using Kestra's KV Store that bounds per-execution spend without restructuring your YAML flows.

TL;DR

Kestra's KV Store lets you read and write arbitrary key-value data inside flows. Issue a vault key in a setup task at the start of the flow, store it in the KV Store under a run-scoped key, and read it in every downstream task that makes vendor calls. All parallel tasks in an EachParallel batch read the same vault key and its per-execution cap. Revoke from the Keybrake dashboard without rotating the real API key stored in Kestra's secret manager.

How Kestra AI agent workflows call vendor APIs

In a Kestra workflow, vendor API calls live inside Python tasks (using io.kestra.plugin.scripts.python.Script or io.kestra.plugin.core.http.Request). An AI billing workflow that processes subscription renewals might use EachParallel to fan out charges:

id: billing-workflow
namespace: production.agents

inputs:
  - id: customer_ids
    type: JSON

tasks:
  - id: charge-customers
    type: io.kestra.plugin.core.flow.EachParallel
    value: "{{ inputs.customer_ids }}"
    tasks:
      - id: charge-customer
        type: io.kestra.plugin.scripts.python.Script
        containerImage: python:3.11-slim
        script: |
          import stripe, os

          stripe.api_key = "{{ secret('STRIPE_SECRET_KEY') }}"  # full-access key
          result = stripe.PaymentIntent.create(
              amount=2999,
              currency="usd",
              customer="{{ taskrun.value }}",
          )
          print(f"Created: {result.id}")

This is standard Kestra. The EachParallel spawns one task execution per customer ID simultaneously. Each Python task runs in its own Docker container, reads the Stripe secret from Kestra's secret manager, and calls Stripe independently. The problem: the full-access Stripe secret is shared across all parallel containers, there's no cap on total charges across all parallel tasks, and Kestra's automatic retry (configured at the task level) means each container failure produces another Stripe API call.

Three gaps Kestra's native tooling doesn't fill for vendor spend control

GapWhat happens in practiceKestra's answer
No per-execution spend cap A billing workflow receives a JSON input with 5,000 customer IDs (upstream query returned all customers, not just the day's renewal cohort). Kestra's EachParallel spawns 5,000 Docker containers simultaneously up to the cluster concurrency limit. Each container calls Stripe independently. The workflow execution completes successfully from Kestra's perspective while Stripe records 5,000 unintended charges. None. Kestra's execution logs track task start/end times, output values, and error states — not dollar amounts spent on vendor calls within task containers.
No task-level vendor revoke You can kill a Kestra execution from the UI or via the API (PUT /api/v1/executions/{executionId}/kill). Kestra sends a kill signal to running containers. But containers already past the Stripe API call will complete that call before the kill signal reaches them. Rotating the Stripe secret to stop in-flight containers breaks every other Kestra workflow using that secret. Execution kill via Kestra UI or REST API. Running containers receive a kill signal but may complete their current operation — including the vendor API call — before exiting.
No per-call audit with execution context Kestra's execution log records task outputs (what you print or return from the task script) but doesn't automatically parse Stripe response data or cross-reference Stripe Request-Id values with Kestra execution IDs and task run IDs. Execution logs and outputs accessible via Kestra UI and API. No automatic vendor call cost correlation — you'd need to manually log and aggregate across thousands of separate task outputs.

The EachParallel risk: concurrent containers and simultaneous vendor calls

Kestra's EachParallel is Kestra's fan-out primitive — it creates one child task execution per item in the input list and runs them all concurrently. On a Kestra deployment with a Kubernetes executor and 100 available pods, 100 Python containers execute in the first wave, each making its own Stripe API call. Unlike code-native orchestrators where you control concurrency via thread pools or worker counts, Kestra's container-per-task model means each task is fully isolated — there's no shared memory where you could put a counter to track total spend.

A vault key issued once and read from the KV Store in each container enforces spend atomically at the proxy layer. The proxy aggregates spend across all concurrent containers using the same vault key. Once the cap is reached, all further calls from any container return 429, regardless of how many containers are running simultaneously.

Scoping vault keys per Kestra execution using KV Store

Issue the vault key in a setup task before the EachParallel, store it in the KV Store under an execution-scoped key, and read it in each parallel task:

id: billing-workflow
namespace: production.agents

inputs:
  - id: customer_ids
    type: JSON
  - id: budget_usd
    type: FLOAT
    defaults: 300.0

tasks:
  - id: setup-vault-key
    type: io.kestra.plugin.scripts.python.Script
    containerImage: python:3.11-slim
    script: |
      import httpx, os, json

      execution_id = "{{ execution.id }}"
      response = httpx.post(
          "https://proxy.keybrake.com/vault/keys",
          headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
          json={
              "vendor": "stripe",
              "daily_usd_cap": {{ inputs.budget_usd }},
              "allowed_endpoints": ["POST /v1/payment_intents"],
              "expires_in": "4h",
              "agent_run_label": f"kestra/{execution_id}",
          },
      )
      vault_key = response.json()["vault_key"]
      # Write to KV Store for downstream tasks
      print(json.dumps({"vault_key": vault_key}))

  - id: store-vault-key
    type: io.kestra.plugin.core.kv.Set
    key: "vault-key-{{ execution.id }}"
    value: "{{ outputs['setup-vault-key'].vars.vault_key }}"
    namespace: production.agents
    ttl: PT4H

  - id: charge-customers
    type: io.kestra.plugin.core.flow.EachParallel
    value: "{{ inputs.customer_ids }}"
    tasks:
      - id: charge-customer
        type: io.kestra.plugin.scripts.python.Script
        containerImage: python:3.11-slim
        script: |
          import stripe, os

          vault_key = "{{ kv('vault-key-' + execution.id) }}"
          stripe.api_key = vault_key          # scoped vault key from KV Store
          stripe.api_base = "https://proxy.keybrake.com/stripe"
          customer_id = "{{ taskrun.value }}"
          stripe.PaymentIntent.create(
              amount=2999,
              currency="usd",
              customer=customer_id,
              idempotency_key=f"{customer_id}-2999",  # stable on retry
          )

The vault key is issued once in setup-vault-key and stored in Kestra's KV Store under an execution-scoped key. All parallel containers in EachParallel read the same vault key from the KV Store. The KV Store TTL matches the vault key TTL — both expire after 4 hours. The real Stripe secret stays in Kestra's secret manager, never in the KV Store or container environment. The proxy audit log records each call with agent_run_label: "kestra/{execution_id}" — queryable by execution ID or time window.

How Keybrake fits

Keybrake is the proxy layer between your Kestra task containers and Stripe, Twilio, or Resend. You issue the vault key in a setup task, store it in the KV Store, and read it in each task container. The real Stripe secret stays in Kestra's secret manager — not in the KV Store, not in container environment variables, not in execution logs. Each workflow execution gets its own vault key with its own dollar cap, endpoint allowlist, and expiry. Parallel EachParallel tasks that exceed the cap return 429s — these surface as task execution failures with traceback data in the Kestra UI, not silent charges distributed across hundreds of isolated containers.

Get early access

Related questions

Is the vault key in KV Store secure enough, or should I use Kestra's secret manager?

Kestra's KV Store is designed for operational state — it's accessible to any task in the namespace that knows the key. Kestra's secret manager (the SECRET() function in YAML) is designed for sensitive credentials, with access controls and audit logging. For maximum security, issue the vault key in the setup task and pass it directly as a task output to downstream tasks (using {{ outputs['setup-vault-key'].vars.vault_key }}) rather than writing to KV Store. Alternatively, write to KV Store but use a scoped namespace path not accessible to other workflows. The key advantage of the vault key versus the real secret: even if the vault key leaks from KV Store, it expires after the configured TTL and can be revoked immediately from Keybrake — the real Stripe secret is never at risk.

How should I handle automatic retries in Kestra without retry-storming on cap exhaustion?

Kestra's automatic retry fires when a task exits with a non-zero status code. When a container receives a 429 from Keybrake cap exhaustion, your Python code should handle the HTTP 429 error and exit with status 0 (logging the cap-hit gracefully) rather than raising an unhandled exception that triggers a retry. Only transient errors — connection timeouts, temporary service unavailability — should exit non-zero and trigger the automatic retry. Use structured exception handling: catch stripe.error.RateLimitError and inspect the response to distinguish Keybrake cap exhaustion (includes X-Keybrake-Cap-Hit: true header) from Stripe's own rate limiting (which you may want to retry).

Does this pattern work for Kestra flows triggered by webhooks or event triggers?

Yes. The vault key is issued per execution, and every Kestra execution — whether triggered by a schedule, a webhook, a manual run, or an event trigger — gets its own unique execution.id. The KV Store key vault-key-{{ execution.id }} is scoped to that specific execution. If a webhook triggers the flow twice in rapid succession (e.g. two webhook events arrive within seconds), each resulting execution gets its own vault key and its own cap. This prevents two concurrent webhook-triggered executions from sharing a cap and prematurely exhausting each other's budget.

Further reading