Huey Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Huey is a lightweight Python task queue backed by Redis, SQLite, or in-memory storage, widely adopted in Flask and Django applications for background billing, subscription renewals, and webhook processing. Three deployment-level failure modes surface specifically when Huey tasks call Stripe — and all three produce duplicate charges that are invisible in worker logs.

This post covers three Huey-specific failure modes — retry re-execution from line 1, periodic-task-plus-manual-dispatch dedup gaps, and immediate=True eager mode silently enabling upstream retry middleware — each of which produces multiple stripe.charges.create() calls against already-billed customers, and the two-layer governance pattern that closes all three: content-hash idempotency keys plus per-task vault keys via a spend-cap proxy, without changing your task topology.

Failure mode 1: Huey task retry re-executes the billing callable from line 1 after stripe.charges.create() has already succeeded

Huey's retry behavior is configured with retries=N and an optional retry_delay=seconds on the @huey.task() decorator. When a task raises any exception, Huey re-enqueues the task message with the original arguments and decrements the retry counter. The re-enqueued task starts from the first line of the decorated function — not from the line that raised.

The critical failure window opens when stripe.charges.create() returns a charge object successfully but a subsequent operation raises before the task function returns. A common example: the Stripe API call succeeds, returning ch_A, but db.record_charge(charge.id, customer_id) raises a psycopg2.OperationalError because the database connection pool is momentarily exhausted. Huey catches the unhandled exception and re-enqueues the task at the configured retry_delay. When the retry executes, stripe.charges.create() fires again with the same arguments and no idempotency key — producing ch_B. Both charges succeed. The worker log records one FAILED entry and one SUCCESS entry for the same logical billing event. The Stripe dashboard shows two charges created a few seconds apart with identical amounts and customer IDs.

# tasks.py — UNSAFE: retry re-executes stripe.charges.create() from line 1
import os
import stripe
from huey import RedisHuey

huey = RedisHuey("billing", host="localhost")
stripe.api_key = os.environ["STRIPE_SECRET_KEY"]  # shared, unrestricted

@huey.task(retries=3, retry_delay=10)
def charge_customer(customer_id: str, amount_cents: int, billing_period: str):
    # Retry starts here — stripe.charges.create() fires again on any exception
    charge = stripe.charges.create(
        amount=amount_cents,
        currency="usd",
        customer=customer_id,
        description=f"Subscription {billing_period}",
    )
    # If this raises (DB pool exhaustion, network error), Huey re-enqueues the task
    # and stripe.charges.create() fires again → ch_B created alongside ch_A
    db.record_charge(charge.id, customer_id, billing_period)
    return charge.id

Two fixes work in combination. The first is a content-hash idempotency key derived from the task's stable input arguments before the Stripe call. The key must be computed from values that do not change between the original execution and any retry — customer_id, amount_cents, and billing_period form a stable triple, and adding a service namespace prefix prevents collisions with idempotency keys from other parts of the system. On retry, stripe.charges.create() receives the same idempotency key and Stripe returns the original charge object rather than creating a new one. The second fix is separating permanent errors from transient ones: stripe.error.CardError and stripe.error.InvalidRequestError cannot be resolved by retrying with the same arguments, so they must be caught and returned as structured result dicts rather than re-raised. Re-raising them causes Huey to retry the task — which fails identically and exhausts retries while potentially generating additional partial charges if the billing API partially succeeded before the error was returned.

# tasks.py — SAFE: content-hash idempotency key + permanent errors returned as dicts
import hashlib
import os
import requests
import stripe
from huey import RedisHuey

huey = RedisHuey("billing", host="localhost")

def get_vault_key(label: str, allowed_cents: int) -> str:
    resp = requests.post(
        "https://proxy.keybrake.com/keys",
        headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
        json={
            "vendor": "stripe",
            "allowed_endpoints": ["POST /v1/charges"],
            "daily_usd_cap": round((allowed_cents / 100) * 1.10, 2),
            "expires_in_seconds": 3600,
            "label": label,
        },
        timeout=10,
    )
    resp.raise_for_status()
    return resp.json()["vault_key"]

@huey.task(retries=3, retry_delay=10)
def charge_customer(customer_id: str, amount_cents: int, billing_period: str):
    # Idempotency key derived from stable inputs — identical on every retry
    idempotency_key = hashlib.sha256(
        f"{customer_id}:{amount_cents}:{billing_period}:huey-billing".encode()
    ).hexdigest()[:32]

    # Per-task vault key scoped to this customer's charge only
    vault_key = get_vault_key(
        label=f"huey-billing-{customer_id}-{billing_period}",
        allowed_cents=amount_cents,
    )

    stripe_client = stripe.StripeClient(
        vault_key,
        base_url="https://proxy.keybrake.com/stripe/v1",
    )

    try:
        charge = stripe_client.charges.create(
            params={
                "amount": amount_cents,
                "currency": "usd",
                "customer": customer_id,
                "description": f"Subscription {billing_period}",
            },
            options=stripe.RequestOptions(idempotency_key=idempotency_key),
        )
    except stripe.error.CardError as e:
        # Permanent — retrying will not fix a declined card
        return {"status": "declined", "customer_id": customer_id, "code": e.code}
    except stripe.error.InvalidRequestError as e:
        # Permanent — retrying with the same args will fail the same way
        return {"status": "invalid", "customer_id": customer_id, "error": str(e)}

    db.record_charge(charge.id, customer_id, billing_period)
    return {"status": "charged", "charge_id": charge.id}

Failure mode 2: @huey.periodic_task and manual task.schedule() for the same billing period create two independent task instances with no dedup

Huey's @huey.periodic_task decorator triggers a task on a crontab-style schedule by creating a task message in the queue at the scheduled time. Unlike APScheduler or Celery Beat, Huey's periodic task mechanism executes the decorated function as a regular task — the periodic trigger is just a message enqueued by Huey's internal scheduler loop, indistinguishable to workers from a manually enqueued message.

The dedup gap appears when a billing event that should happen exactly once is covered by both a periodic task and a manual dispatch. The common scenario: a @huey.periodic_task(crontab(day_of_month="1")) triggers monthly billing on the first of each month. An operations engineer, seeing that billing hasn't fully completed (due to a temporary worker outage earlier that day), manually calls charge_customer.schedule(args=(customer_id, amount_cents, "2026-07"), delay=60) to catch up the missed customers. Both the periodic job's re-queued messages and the manually-scheduled messages reach stripe.charges.create() for the same customers. Huey has no concept of task identity across invocation paths — the periodic message and the manual message have different task IDs and are treated as completely independent units of work.

# tasks.py — UNSAFE: periodic billing + manual backfill creates two task instances
from huey import RedisHuey, crontab
import stripe, os

huey = RedisHuey("billing", host="localhost")
stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

@huey.task(retries=3, retry_delay=10)
def charge_customer(customer_id: str, amount_cents: int, billing_period: str):
    charge = stripe.charges.create(
        amount=amount_cents, currency="usd", customer=customer_id,
        description=f"Subscription {billing_period}",
    )
    db.record_charge(charge.id, customer_id, billing_period)
    return charge.id

@huey.periodic_task(crontab(day_of_month="1", hour="0", minute="0"))
def run_monthly_billing():
    customers = db.get_active_subscribers()
    for c in customers:
        # Periodic enqueue — one task message per customer
        charge_customer(c["id"], c["amount_cents"], current_billing_period())

# Elsewhere, in an ops script or admin panel:
# charge_customer.schedule(args=(cid, amt, period), delay=60)
# → second task message, second stripe.charges.create() for the same customer

The fix is enforcing at-most-once execution at the application layer using Stripe's idempotency key, since Huey provides no dedup guarantee across enqueue paths. A content-hash idempotency key derived from (customer_id, billing_period) means that regardless of whether the charge attempt comes from the periodic task or the manual backfill, Stripe returns the existing charge object rather than creating a new one. The second fix is checking for an existing charge record in the database before calling Stripe at all: if db.find_charge(customer_id, billing_period) returns a result, the task returns immediately without any Stripe API call, eliminating the window entirely at the source rather than relying on Stripe's idempotency layer as the last line of defence.

# tasks.py — SAFE: idempotency key + pre-flight DB check closes the dedup gap
import hashlib, os, requests, stripe
from huey import RedisHuey, crontab

huey = RedisHuey("billing", host="localhost")

def get_vault_key(label: str, allowed_cents: int) -> str:
    resp = requests.post(
        "https://proxy.keybrake.com/keys",
        headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
        json={
            "vendor": "stripe",
            "allowed_endpoints": ["POST /v1/charges"],
            "daily_usd_cap": round((allowed_cents / 100) * 1.10, 2),
            "expires_in_seconds": 3600,
            "label": label,
        },
        timeout=10,
    )
    resp.raise_for_status()
    return resp.json()["vault_key"]

@huey.task(retries=3, retry_delay=10)
def charge_customer(customer_id: str, amount_cents: int, billing_period: str):
    # Pre-flight: if this billing period was already charged, skip entirely
    existing = db.find_charge(customer_id, billing_period)
    if existing:
        return {"status": "already_charged", "charge_id": existing["charge_id"]}

    idempotency_key = hashlib.sha256(
        f"{customer_id}:{billing_period}:huey-billing".encode()
    ).hexdigest()[:32]

    vault_key = get_vault_key(
        label=f"huey-billing-{customer_id}-{billing_period}",
        allowed_cents=amount_cents,
    )

    stripe_client = stripe.StripeClient(
        vault_key,
        base_url="https://proxy.keybrake.com/stripe/v1",
    )

    try:
        charge = stripe_client.charges.create(
            params={
                "amount": amount_cents,
                "currency": "usd",
                "customer": customer_id,
                "description": f"Subscription {billing_period}",
            },
            options=stripe.RequestOptions(idempotency_key=idempotency_key),
        )
    except stripe.error.CardError as e:
        return {"status": "declined", "customer_id": customer_id, "code": e.code}
    except stripe.error.InvalidRequestError as e:
        return {"status": "invalid", "customer_id": customer_id, "error": str(e)}

    db.record_charge(charge.id, customer_id, billing_period)
    return {"status": "charged", "charge_id": charge.id}

@huey.periodic_task(crontab(day_of_month="1", hour="0", minute="0"))
def run_monthly_billing():
    customers = db.get_active_subscribers()
    period = current_billing_period()
    for c in customers:
        charge_customer(c["id"], c["amount_cents"], period)

Failure mode 3: immediate=True eager mode accidentally in production lets upstream retry middleware fire a second charge

Huey supports an immediate mode — set by passing immediate=True to the RedisHuey constructor or by setting the HUEY_IMMEDIATE environment variable — that executes all tasks synchronously in the calling thread at the moment they are enqueued, rather than sending messages to a broker. This mode is designed for test environments and local development: it eliminates the need for a running Huey consumer process and makes task execution deterministic.

The production failure mode appears when immediate=True configuration leaks into a deployed environment. This is more common than it sounds: a staging environment variable is included in a Kubernetes ConfigMap template that is also used for production with a different environment name, or a .env.staging file is mistakenly deployed to a production container during a CI pipeline misconfiguration. When immediate=True is active in production, every call to charge_customer(customer_id, amount_cents, billing_period) — which an engineer expects to enqueue a message — instead executes the billing function synchronously in the caller's thread. If the HTTP request that triggered the billing enqueue is behind a load balancer or API gateway with connection-level retry logic (AWS ALB's default two retries on 502/504, nginx upstream retry, or a client SDK with built-in retries), the retried request calls charge_customer(...) again, which immediately executes stripe.charges.create() a second time in the request thread without any idempotency protection.

# app.py — UNSAFE: immediate=True leaks into production
import os
from huey import RedisHuey

# HUEY_IMMEDIATE accidentally set to "1" in the deployed ConfigMap
huey = RedisHuey(
    "billing",
    host=os.environ.get("REDIS_HOST", "localhost"),
    immediate=os.environ.get("HUEY_IMMEDIATE", "0") == "1",  # True in prod by accident
)

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

@huey.task()
def charge_customer(customer_id: str, amount_cents: int, billing_period: str):
    # In immediate mode: executes synchronously in the HTTP request thread
    # ALB retries the request → this function runs again → second charge
    charge = stripe.charges.create(
        amount=amount_cents,
        currency="usd",
        customer=customer_id,
        description=f"Subscription {billing_period}",
    )
    db.record_charge(charge.id, customer_id, billing_period)
    return charge.id

# Flask view — engineer expects this to enqueue a task
@app.route("/billing/charge", methods=["POST"])
def trigger_charge():
    charge_customer(request.json["customer_id"], request.json["amount_cents"], ...)
    return {"queued": True}, 202
    # In immediate mode: the charge fires NOW, not later — and if ALB retries → double charge

There are three layers of defence. The first is detecting eager mode at startup and refusing to process billing tasks when it is active: add an assertion in your billing task initialization that not huey.immediate, and configure your monitoring to alert on the presence of HUEY_IMMEDIATE=1 in production pods via Kubernetes admission webhooks or a startup check. The second is the same content-hash idempotency key that protects against retry failure modes — even if the billing callable executes twice due to eager mode, Stripe returns the original charge object on the second call rather than creating a new one. The third is per-task vault keys with spend caps: even without an idempotency key, the vault key's daily cap is set to the single customer's charge amount, so the proxy blocks the second call before it reaches Stripe, and the Keybrake audit log captures both attempts with a clear rejection record showing the duplicate was stopped.

# app.py — SAFE: immediate mode guard + idempotency + vault key
import hashlib, os, requests, stripe
from huey import RedisHuey

huey = RedisHuey(
    "billing",
    host=os.environ.get("REDIS_HOST", "localhost"),
    immediate=os.environ.get("HUEY_IMMEDIATE", "0") == "1",
)

# Fail fast at startup in production — never process billing with eager mode active
if os.environ.get("APP_ENV") == "production" and huey.immediate:
    raise RuntimeError(
        "HUEY_IMMEDIATE=1 is set in production — billing tasks would execute "
        "synchronously and are not safe to run. Unset HUEY_IMMEDIATE and restart."
    )

def get_vault_key(label: str, allowed_cents: int) -> str:
    resp = requests.post(
        "https://proxy.keybrake.com/keys",
        headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
        json={
            "vendor": "stripe",
            "allowed_endpoints": ["POST /v1/charges"],
            "daily_usd_cap": round((allowed_cents / 100) * 1.10, 2),
            "expires_in_seconds": 3600,
            "label": label,
        },
        timeout=10,
    )
    resp.raise_for_status()
    return resp.json()["vault_key"]

@huey.task(retries=3, retry_delay=10)
def charge_customer(customer_id: str, amount_cents: int, billing_period: str):
    # Pre-flight: skip if already charged this period
    existing = db.find_charge(customer_id, billing_period)
    if existing:
        return {"status": "already_charged", "charge_id": existing["charge_id"]}

    idempotency_key = hashlib.sha256(
        f"{customer_id}:{billing_period}:huey-billing".encode()
    ).hexdigest()[:32]

    vault_key = get_vault_key(
        label=f"huey-billing-{customer_id}-{billing_period}",
        allowed_cents=amount_cents,
    )

    stripe_client = stripe.StripeClient(
        vault_key,
        base_url="https://proxy.keybrake.com/stripe/v1",
    )

    try:
        charge = stripe_client.charges.create(
            params={
                "amount": amount_cents,
                "currency": "usd",
                "customer": customer_id,
                "description": f"Subscription {billing_period}",
            },
            options=stripe.RequestOptions(idempotency_key=idempotency_key),
        )
    except stripe.error.CardError as e:
        return {"status": "declined", "customer_id": customer_id, "code": e.code}
    except stripe.error.InvalidRequestError as e:
        return {"status": "invalid", "customer_id": customer_id, "error": str(e)}

    db.record_charge(charge.id, customer_id, billing_period)
    return {"status": "charged", "charge_id": charge.id}

Why all three modes need the proxy layer, not just idempotency keys

Idempotency keys fix Stripe's side of the problem — they prevent Stripe from creating duplicate charge objects when the same key is presented twice. But they do not solve the audit problem: if your billing task fires stripe.charges.create() twice, your application's own audit trail shows one charge record (written on the first successful execution) while the Stripe dashboard received two API calls. Without a proxy layer recording every outbound Stripe request, the duplicate attempt is invisible in your system — you only discover it by reconciling Stripe's API logs against your internal records after the fact.

A spend-cap proxy sits at the network layer between your Huey workers and Stripe's API. Every vault key is scoped to a single customer's billing event with a cap of amount_cents × 1.10. When the retry or duplicate fires, the proxy checks the accumulated spend on that vault key, sees that the cap is already exhausted, and returns a 402 Policy Violation before forwarding the request to Stripe. The proxy records both the successful request and the blocked duplicate in the audit log, giving you a complete picture of what your agents and workers attempted — not just what Stripe processed.

What Huey's audit trail looks like without the proxy

Huey's built-in result store (Redis or SQLite) records task outcomes: success or failure, result value, and timestamps. It does not record what the task did to external systems. A Huey task that calls stripe.charges.create() twice — once successfully and once as a retry that is blocked by Stripe's idempotency layer — appears in the Huey result store as a single successful task with the original charge ID as its result value. There is no indication in the Huey task history that a second Stripe API call was made. You cannot reconstruct the duplicate attempt from Huey logs alone; you need either the Stripe API logs (which are only available via the Stripe Dashboard for paid plans) or a proxy-layer audit log that captures every outbound request before it reaches Stripe.

This audit gap matters for compliance-sensitive workloads. If your billing system processes payments under PCI DSS or SOC 2 obligations, demonstrating that each customer was charged exactly once requires a complete record of every Stripe API call attempted by your system, not just the successful task completions stored in Huey's result backend. A proxy audit log — recording method, endpoint, vault key, idempotency key, response status, and timestamp for every forwarded request — satisfies this requirement without modifying your Huey task code or adding Stripe webhook listeners.

Configuring vault key scope for Huey billing workloads

Huey billing tasks typically fall into three categories that warrant different vault key policies: subscription renewals (predictable per-customer amounts on a fixed schedule), usage-based billing fan-outs (variable amounts from a metered usage calculation, dispatched in bulk), and one-off manual charges triggered by admin actions or webhook events. Each category has a different risk profile and should be scoped accordingly.

For subscription renewals dispatched by a periodic task, issue vault keys in a pre-flight batch before enqueuing tasks, with each key capped at the specific customer's monthly amount. Pass vault keys as task arguments rather than reading a shared key inside the task. For usage-based fan-outs, cap at the calculated usage amount for that billing cycle rather than a fixed subscription price; the variable cap means a calculation error is caught at the proxy before it propagates across the entire customer cohort. For manual admin charges, issue short-lived vault keys (30–60 minutes) scoped to a single POST /v1/charges call and require the key to be consumed within the session — any attempt to reuse the key after the window closes is rejected and logged, providing a complete audit trail of admin-initiated billing actions.

# Pre-flight vault key issuance before periodic billing fan-out
import requests, os
from typing import List, Dict

def issue_billing_vault_keys(customers: List[Dict], billing_period: str) -> Dict[str, str]:
    """Issue one vault key per customer before any task is enqueued."""
    vault_keys = {}
    for c in customers:
        resp = requests.post(
            "https://proxy.keybrake.com/keys",
            headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
            json={
                "vendor": "stripe",
                "allowed_endpoints": ["POST /v1/charges"],
                "daily_usd_cap": round((c["amount_cents"] / 100) * 1.05, 2),
                "expires_in_seconds": 7200,  # 2h window for billing run
                "label": f"periodic-billing-{c['id']}-{billing_period}",
                "metadata": {
                    "billing_period": billing_period,
                    "customer_id": c["id"],
                    "source": "huey-periodic-task",
                },
            },
            timeout=10,
        )
        resp.raise_for_status()
        vault_keys[c["id"]] = resp.json()["vault_key"]
    return vault_keys

@huey.periodic_task(crontab(day_of_month="1", hour="0", minute="0"))
def run_monthly_billing():
    customers = db.get_active_subscribers()
    period = current_billing_period()

    # Issue all vault keys before touching Stripe
    vault_keys = issue_billing_vault_keys(customers, period)

    for c in customers:
        charge_customer(
            customer_id=c["id"],
            amount_cents=c["amount_cents"],
            billing_period=period,
            vault_key=vault_keys[c["id"]],
        )

Related Huey configuration patterns worth reviewing

Beyond the three failure modes covered above, two additional Huey configuration choices affect billing safety. The first is the choice of storage backend. Huey supports Redis, SQLite, and an in-memory MemoryHuey. The MemoryHuey backend is lossy — enqueued task messages are not persisted to disk, so a worker crash after a Stripe charge succeeded but before the task result was recorded means the charge has no local record. Use Redis with persistence (appendonly yes in redis.conf) or a durable SQLite file on a mounted volume for any workload that touches Stripe. The second is Huey's utc=True flag, which affects how periodic task schedules are evaluated. If your billing period logic uses UTC timestamps for idempotency key derivation but your @huey.periodic_task schedule is evaluated in local time, the billing period string may differ between the periodic task's execution context and a manual task.schedule() call issued from a different timezone context — breaking the idempotency key match. Always derive billing_period from UTC timestamps, and always set utc=True on your Huey instance for billing workloads.

If you are migrating from Celery to Huey, note that Huey's retry semantics differ from Celery's self.retry() mechanism. In Celery, tasks must explicitly call self.retry() to trigger a retry — uncaught exceptions do not automatically retry unless autoretry_for is configured. In Huey, retries=N catches any unhandled exception and re-enqueues automatically, meaning all three failure modes described above apply out of the box to any Huey task with retries enabled, without any explicit retry call in the task body. Billing tasks migrated from Celery to Huey without idempotency keys gain new double-charge risk from the more aggressive default retry behavior.