Azure Service Bus and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Azure Service Bus is a fully managed enterprise message broker with at-least-once delivery semantics in PEEK_LOCK mode. When Service Bus messages drive billing — a usage event is enqueued, a consumer receives it and calls stripe.charges.create() — three Service Bus-specific behaviors introduce Stripe double-charge failure modes that Azure Monitor queue metrics do not surface as billing risk.

This post covers those three failure modes with azure-servicebus Python code, content-hash idempotency keys stable across lock expirations and dead-letter resubmissions, per-billing-period vault keys via a spend-cap proxy, and pre-flight database checks — the two-layer governance pattern that closes all three without changing queue configuration, LockDuration, or MaxDeliveryCount.

Failure mode 1: Service Bus lock duration expires while stripe.charges.create() is in-flight — the lock renewal call fails independently of the billing thread

Every message received in PEEK_LOCK mode (ServiceBusReceiveMode.PEEK_LOCK, the default for ServiceBusReceiver) holds a lock token valid for the queue’s LockDuration. LockDuration is a queue-level setting configurable from 5 seconds to 5 minutes (300 seconds), with a default of 60 seconds. During this window the message is invisible to all other consumers. If the consumer does not call receiver.complete_message(msg) within the lock window — or extend the lock with receiver.renew_message_lock(msg) — Service Bus releases the lock token, the message becomes visible to all competing consumers, and the delivery count is incremented.

The azure-servicebus Python SDK requires explicit lock renewal: unlike the Google Pub/Sub lease manager thread (which renews automatically inside StreamingPullFuture), ServiceBusReceiver does not renew locks automatically. Engineers who need lock renewal must run a separate thread or use AutoLockRenewer from azure.servicebus.aio. AutoLockRenewer polls a background task that calls renew_message_lock() on a schedule (default: it calculates a renewal interval from the remaining lock time). This background task makes its own network request to the Service Bus endpoint — a request that can fail independently of the stripe.charges.create() call it is meant to protect.

The failure scenario: a billing consumer receives a usage event, starts a Stripe charge that takes 75 seconds under load (stripe-python’s max_network_retries=2 with a 30-second per-attempt timeout), and the AutoLockRenewer background task calls renew_message_lock() at T=45 seconds. That renewal call hits a transient Azure Service Bus throttling event (ServerBusyError — Service Bus throttles renew_message_lock calls per namespace per second) and fails. The lock expires at T=60. Service Bus releases the lock and delivers the message to a second competing consumer in the same consumer group. The second consumer’s billing callback calls stripe.charges.create() for the same customer — creating ch_B alongside ch_A that the first consumer’s Stripe call is still waiting to complete.

from azure.servicebus import ServiceBusClient, ServiceBusReceiveMode
import stripe
import os
import json

connection_string = os.environ["AZURE_SERVICEBUS_CONNECTION_STRING"]
queue_name = os.environ["AZURE_SERVICEBUS_QUEUE_NAME"]

with ServiceBusClient.from_connection_string(connection_string) as client:
    with client.get_queue_receiver(
        queue_name,
        receive_mode=ServiceBusReceiveMode.PEEK_LOCK,
    ) as receiver:
        for msg in receiver:
            event = json.loads(str(msg))
            customer_id = event["customer_id"]
            billing_period = event["billing_period"]

            stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

            # This call can take 60-90 seconds under load with stripe-python's
            # default max_network_retries=2 (up to 3 attempts × 30s timeout).
            # The queue's LockDuration (default 60 seconds) will expire during
            # this call unless renew_message_lock() is called before T=60.
            # If AutoLockRenewer's background renewal call hits ServerBusyError
            # (Service Bus throttling), the lock expires and Service Bus delivers
            # the message to the next competing consumer.
            # The second consumer calls stripe.charges.create() for the same
            # customer — no idempotency_key means ch_B is created alongside ch_A.
            charge = stripe.Charge.create(
                amount=event["amount_cents"],
                currency="usd",
                customer=event["stripe_customer_id"],
                description=f"Usage billing for {billing_period}",
                # no idempotency_key — competing consumer creates ch_B
            )

            # complete_message() is only called after the Stripe call returns.
            # If the lock expired during the Stripe call, complete_message()
            # raises MessageLockLostError — the consumer knows the lock expired
            # only after the damage (duplicate charge) has already happened.
            receiver.complete_message(msg)
            print(f"Charged {customer_id}: {charge['id']}")

The failure is invisible in standard Azure Monitor metrics. ActiveMessages shows 0 while the first consumer holds the lock (the message is invisible to the queue metric). When the lock expires and the second consumer picks it up, ActiveMessages briefly shows 1 before falling back to 0. DeadLetterMessageCount is unchanged: the message was eventually completed by the second consumer (after creating ch_B). From Service Bus’s perspective, the message was delivered once and completed. From Stripe’s perspective, the customer was charged twice.

The AutoLockRenewer failure is compounded by Service Bus namespace throttling limits. A namespace in the Standard tier allows a limited number of management operations (including renew_message_lock) per second. A billing burst that processes many messages simultaneously can push all AutoLockRenewer background tasks to renew locks at the same moment, triggering ServerBusyError responses for some fraction of them. In a 10-consumer billing fleet processing 500 messages simultaneously, all 500 locks may be due for renewal within the same second — a thundering-herd pattern that the AutoLockRenewer’s jitter logic does not fully mitigate under quota pressure.

Failure mode 2: Dead-letter resubmission assigns a new MessageId — idempotency keys derived from message metadata bypass Stripe deduplication

When a Service Bus message fails processing enough times that its delivery count reaches the queue’s MaxDeliveryCount (default 10), Service Bus automatically moves the message to the dead-letter subqueue ($DeadLetterQueue). The message lands in the DLQ with dead_letter_reason="MaxDeliveryCountExceeded" and a dead_letter_error_description field containing the last processing error. The MaxDeliveryCount mechanism is Service Bus’s built-in protection against poison messages that crash consumers — a sensible safeguard for general workloads.

For billing pipelines, the interaction between DLQ and Stripe idempotency is dangerous. Operators who investigate dead-letter messages commonly use Service Bus Explorer, the Azure Portal’s Service Bus interface, or SDK tooling to resubmit messages: they read the dead-letter message, construct a new ServiceBusMessage from its body, and send it back to the main queue. Azure Service Bus assigns a fresh message_id to every newly constructed ServiceBusMessage (a UUID generated at construction time), regardless of what message_id the original dead-letter message carried.

A billing callback that builds its Stripe idempotency key from msg.message_id — a common approach because message_id is the Service Bus message’s own unique identifier — produces a different idempotency key for the resubmitted message than for the original message. Stripe receives a stripe.charges.create() call with a new idempotency key and has no way to know this is a retry of an earlier request. If the original charge succeeded server-side before the consumer exhausted its delivery count, the resubmission fires a new Stripe charge for the same customer.

from azure.servicebus import ServiceBusClient, ServiceBusMessage, ServiceBusReceiveMode
import stripe
import os
import json

connection_string = os.environ["AZURE_SERVICEBUS_CONNECTION_STRING"]
queue_name = os.environ["AZURE_SERVICEBUS_QUEUE_NAME"]

# --- unsafe billing callback (used in original main queue consumer) ---

with ServiceBusClient.from_connection_string(connection_string) as client:
    with client.get_queue_receiver(queue_name) as receiver:
        for msg in receiver:
            event = json.loads(str(msg))

            # msg.message_id is a UUID assigned by Service Bus when the message
            # was first enqueued. Using it as an idempotency key seems natural —
            # it's the message's own unique ID — but it only works if the SAME
            # message object is retried. A resubmitted message from the DLQ
            # is a NEW ServiceBusMessage with a NEW message_id.
            idempotency_key = f"charge-{msg.message_id}"  # WRONG

            charge = stripe.Charge.create(
                amount=event["amount_cents"],
                currency="usd",
                customer=event["stripe_customer_id"],
                description=f"Usage billing for {event['billing_period']}",
                idempotency_key=idempotency_key,
                api_key=os.environ["STRIPE_SECRET_KEY"],
            )
            receiver.complete_message(msg)


# --- operator tooling for DLQ resubmission (creates new MessageId) ---

def resubmit_dead_letter(connection_string: str, queue_name: str) -> None:
    with ServiceBusClient.from_connection_string(connection_string) as client:
        # Read from the dead-letter subqueue.
        dlq_path = f"{queue_name}/$DeadLetterQueue"
        with client.get_queue_receiver(dlq_path) as dlq_receiver:
            for dlq_msg in dlq_receiver.receive_messages(max_message_count=10):
                body = str(dlq_msg)  # the original message body

                # Construct a NEW ServiceBusMessage from the body.
                # Azure Service Bus assigns a new message_id (UUID) to this object.
                # The original dlq_msg.message_id is NOT preserved unless explicitly
                # set via ServiceBusMessage(body, message_id=dlq_msg.message_id).
                # Most operator tooling (Service Bus Explorer, Azure Portal) does NOT
                # preserve the original message_id on resubmission.
                new_msg = ServiceBusMessage(body)
                # new_msg.message_id is a brand-new UUID — different from dlq_msg.message_id.

                with client.get_queue_sender(queue_name) as sender:
                    sender.send_messages(new_msg)

                dlq_receiver.complete_message(dlq_msg)
                print(f"Resubmitted: original message_id={dlq_msg.message_id}, "
                      f"new message_id={new_msg.message_id}")
                # The billing callback receives new_msg with a new message_id.
                # idempotency_key = "charge-{new_msg.message_id}" is different
                # from the original "charge-{dlq_msg.message_id}".
                # Stripe treats this as a new billing request and creates ch_B
                # if ch_A already exists for the customer from the original attempt.

The DLQ resubmission failure mode is particularly insidious because it triggers during manual incident response: an engineer notices billing messages in the dead-letter queue, correctly identifies them as needing reprocessing, and resubmits them using standard tooling. The duplicate charges appear on customer statements days after the original incident — long after the incident was considered resolved. The root cause is not a bug in the consumer logic but in the assumption that message_id is a stable identity across Service Bus message lifecycle, which it is not for DLQ resubmissions.

A secondary path to the same failure: Service Bus auto-forwarding. If the dead-letter queue is configured to auto-forward to another queue (a pattern used for centralized dead-letter processing), the forwarded message retains its message_id during the forward operation. But if the centralized dead-letter processor constructs new messages for the target billing queue — rather than using ServiceBusMessage’s session_id-preserving forward semantics — the same message_id rotation occurs. The billing consumer cannot distinguish a forwarded message from a resubmitted one by inspecting message_id alone.

Failure mode 3: receiver.abandon_message() on OperationTimeoutError redelivers immediately and re-fires billing for charges that already succeeded server-side

The standard error-handling pattern for Service Bus consumers is to call receiver.abandon_message(msg) when processing fails with a retriable error. abandon_message() releases the lock token immediately — the message is no longer invisible to competing consumers and can be received by any available consumer in the next poll cycle. The delivery count increments by 1. When the delivery count reaches MaxDeliveryCount, Service Bus moves the message to the dead-letter subqueue.

For billing callbacks, abandon_message() on OperationTimeoutError is the wrong response. The azure-servicebus SDK raises OperationTimeoutError when the AMQP operation times out — which can happen when the stripe-python client’s HTTP request to Stripe’s API is blocked waiting for a response while the AMQP connection to Service Bus times out independently. But more commonly, OperationTimeoutError is raised by the Service Bus SDK itself on management operations (like renew_message_lock) while the Stripe call is in-flight and the lock renewal fails under throttling pressure. In both cases, the consumer catches an exception that looks like a retriable error and calls abandon_message() — not knowing whether the Stripe call that preceded the timeout completed successfully on Stripe’s servers.

Unlike SQS’s visibility timeout, which provides a configurable delay before a message reappears to consumers after a visibility reset, or Pub/Sub’s retry policy (minimum_backoff / maximum_backoff on nack()), Service Bus’s abandon_message() has no configurable backoff at the abandon layer. The message becomes available for redelivery immediately. In a fleet with multiple competing consumers, the abandoned message can be picked up within milliseconds by the next consumer’s poll. That consumer calls stripe.charges.create() without an idempotency key, Stripe processes it as a new request, and ch_B is created alongside ch_A that Stripe had already created before the timeout response was lost.

from azure.servicebus import ServiceBusClient, ServiceBusReceiveMode
from azure.servicebus.exceptions import OperationTimeoutError, ServiceBusError
import stripe
import os
import json

connection_string = os.environ["AZURE_SERVICEBUS_CONNECTION_STRING"]
queue_name = os.environ["AZURE_SERVICEBUS_QUEUE_NAME"]

with ServiceBusClient.from_connection_string(connection_string) as client:
    with client.get_queue_receiver(queue_name) as receiver:
        for msg in receiver:
            event = json.loads(str(msg))
            customer_id = event["customer_id"]
            billing_period = event["billing_period"]

            try:
                stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

                charge = stripe.Charge.create(
                    amount=event["amount_cents"],
                    currency="usd",
                    customer=event["stripe_customer_id"],
                    description=f"Usage billing for {billing_period}",
                    # no idempotency_key — each delivery attempt is a new Stripe request
                )
                receiver.complete_message(msg)
                print(f"Charged {customer_id}: {charge['id']}")

            except stripe.error.Timeout:
                # Stripe timed out. ch_A may already exist on Stripe's servers
                # (the charge was created before the response was lost).
                # abandon_message() releases the lock immediately — the message
                # is available to competing consumers within milliseconds.
                # No configurable backoff at the abandon layer in Service Bus.
                # The next consumer calls stripe.charges.create() again without
                # an idempotency key — ch_B is created if ch_A exists.
                receiver.abandon_message(msg)  # WRONG

            except OperationTimeoutError:
                # Service Bus SDK timed out (AMQP operation, lock renewal,
                # or management API call). The Stripe call may or may not have
                # completed. abandon_message() causes immediate redelivery.
                # If the Stripe call created ch_A before this exception was raised,
                # the redelivery creates ch_B.
                receiver.abandon_message(msg)  # WRONG

            except stripe.error.CardError:
                # Card declined — permanent failure. abandon_message() redelivers
                # indefinitely until MaxDeliveryCount is reached, then moves to DLQ.
                # This wastes delivery attempts on an unrecoverable error and
                # creates 10 failed Stripe charge attempts before the message
                # lands in the DLQ — each attempt incurring a Stripe API call
                # and a potential card re-decline event visible to the customer.
                receiver.abandon_message(msg)  # WRONG: use complete_message after writing decline

The abandon_message() pattern is particularly misleading for stripe.error.Timeout because the error name suggests the operation did not complete. Engineers unfamiliar with Stripe’s server-side processing model — where the charge is created on Stripe’s servers before the HTTP response is sent — treat Timeout as a safe retry signal. The idempotency key documentation clarifies this, but only for engineers who have read it. The default instinct is: timeout means failure, failure means retry, retry means abandon. That instinct creates billing duplicates in proportion to the rate of Stripe timeout events under load.

The fix: content-hash idempotency key + vault key + pre-flight check

All three failure modes share the same root cause: stripe.charges.create() is called more than once for the same (customer_id, billing_period) pair, and neither Service Bus’s lock-expiry mechanism nor Stripe’s API layer prevents the second call from creating a new charge when the idempotency key is absent or rotated. The fix has two independent layers that close all three modes without changing queue LockDuration, MaxDeliveryCount, or consumer deployment architecture.

Layer 1: content-hash idempotency key stable across lock expirations and DLQ resubmissions

The Stripe idempotency key must be derived from the billing event’s stable business fields — fields present in the message body that represent the same business intent regardless of which Service Bus message object carries them. The key must not include any Service Bus metadata that changes across deliveries or resubmissions: message_id (a UUID assigned by Service Bus at enqueue time; a resubmitted DLQ message gets a new UUID), lock_token (a UUID assigned per delivery attempt — each new delivery of the same message gets a different lock token), delivery_count (incremented on every failed delivery), enqueued_time_utc (set when the message first entered the queue; a resubmitted message has a new enqueue time), or sequence_number (a monotonically increasing integer assigned by Service Bus; a resubmitted message gets a new sequence number). Including any of these fields produces a different idempotency key on each resubmission, defeating Stripe’s 24-hour deduplication window.

import hashlib
import stripe
import os
import json
import psycopg2

def make_idempotency_key(customer_id: str, billing_period: str) -> str:
    # Derived from stable business identity: customer + billing period only.
    # Must NOT include: Service Bus message_id (UUID assigned at enqueue;
    # DLQ resubmission assigns new UUID), lock_token (per-delivery UUID),
    # delivery_count (increments on failure), enqueued_time_utc (set at
    # original enqueue; resubmitted message has new enqueue time), or
    # sequence_number (monotonically increasing; new number per resubmission).
    # All of these change across lock expirations, DLQ resubmissions, and
    # competing-consumer deliveries — making them unsafe as idempotency sources.
    raw = f"{customer_id}:{billing_period}:servicebus-billing"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]


def process_billing_event(conn, event: dict) -> dict:
    customer_id = event["customer_id"]
    stripe_customer_id = event["stripe_customer_id"]
    amount_cents = event["amount_cents"]
    billing_period = event["billing_period"]

    idempotency_key = make_idempotency_key(customer_id, billing_period)

    # Pre-flight check: was this customer already billed for this period?
    # Authoritative source: billing_records with a unique constraint on
    # (customer_id, billing_period). Closes three failure modes:
    # (1) lock expiry: if the first consumer wrote billing_records before its
    #     lock expired, the second consumer's pre-flight check skips the Stripe call.
    # (2) DLQ resubmission: if the original delivery wrote billing_records before
    #     exhausting MaxDeliveryCount, the resubmission's pre-flight check
    #     finds the existing row and skips the Stripe call regardless of
    #     whether the resubmitted message has a new message_id.
    # (3) abandon_message() on timeout: if billing_records was written before
    #     the timeout or OperationTimeoutError was raised, the next delivery's
    #     pre-flight check skips the Stripe call.
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT charge_id, status FROM billing_records
            WHERE customer_id = %s AND billing_period = %s
            """,
            (customer_id, billing_period),
        )
        existing = cur.fetchone()

    if existing:
        return {
            "skipped": True,
            "reason": "already_billed",
            "charge_id": existing[0],
        }

    vault_key = os.environ["KEYBRAKE_VAULT_KEY"]   # per-billing-period key with spend cap

    charge = stripe.Charge.create(
        amount=amount_cents,
        currency="usd",
        customer=stripe_customer_id,
        description=f"Usage billing for {billing_period}",
        idempotency_key=idempotency_key,
        api_key=vault_key,   # proxied through spend-cap layer
    )

    # Write to billing_records BEFORE calling complete_message().
    # Write-before-complete: if complete_message() fails after this write,
    # Service Bus redelivers the message. The pre-flight check above skips
    # the customer on the next delivery — no second Stripe call is made.
    # ON CONFLICT DO NOTHING closes the concurrent lock-expiry race:
    # if two competing consumers both passed the pre-flight check simultaneously
    # (lock expired just before one consumer's billing call returned),
    # the unique constraint ensures only one INSERT succeeds — the second is
    # silently ignored. The second consumer's complete_message() also succeeds
    # because Service Bus allows completing a message even if another consumer
    # completed it first (MessageLockLostError is raised instead — handled below).
    with conn.cursor() as cur:
        cur.execute(
            """
            INSERT INTO billing_records
                (customer_id, billing_period, charge_id, amount_cents, status, created_at)
            VALUES (%s, %s, %s, %s, 'charged', NOW())
            ON CONFLICT (customer_id, billing_period) DO NOTHING
            """,
            (customer_id, billing_period, charge["id"], amount_cents),
        )
    conn.commit()

    return {"charge_id": charge["id"], "status": "charged"}

Layer 2: vault key with per-billing-period spend cap

The vault key is a per-billing-period Stripe key issued through a spend-cap proxy. The vault key enforces a maximum spend for the billing run, set at 110% of the expected total. For the lock-expiry failure mode — where a competing consumer picks up an expired-lock message and calls stripe.charges.create() before the idempotency key has deduplicated — the vault key provides the hard cap when the idempotency key window has expired (a DLQ message retained for 14 days under Service Bus’s default dead-letter retention, then resubmitted), ensuring that even worst-case lock expiry churn cannot exceed expected_total × 1.10 in total Stripe charges for a single billing run.

import requests
import os
import datetime

def create_billing_vault_key(
    billing_period: str,
    expected_total_cents: int,
) -> str:
    resp = requests.post(
        "https://proxy.keybrake.com/vault/keys",
        headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
        json={
            "vendor": "stripe",
            "label": f"servicebus-billing-run-{billing_period}",
            "daily_usd_cap": round((expected_total_cents * 1.10) / 100, 2),
            "allowed_endpoints": ["/v1/charges", "/v1/payment_intents"],
            "expires_at": (
                datetime.datetime.utcnow() + datetime.timedelta(hours=6)
            ).isoformat() + "Z",
        },
    )
    resp.raise_for_status()
    return resp.json()["vault_key"]

# For a billing run with 100 customers at $50 each = $5,000 expected total:
# - Vault key cap: $5,500 (110% of expected total)
# - If lock expiry causes 2 competing consumers both to attempt billing for
#   10 customers (10 × $50 × 2 = $1,000 potential duplicate):
#   - Idempotency key deduplicates charges within 24h of original attempt
#   - Vault cap blocks charges past $5,500 (covers expired-idempotency edge)
#   - Per-billing-period key: issue one vault key per billing run, not per message

Safe Service Bus billing consumer with write-before-complete

from azure.servicebus import ServiceBusClient, ServiceBusReceiveMode
from azure.servicebus.exceptions import (
    MessageLockLostError,
    OperationTimeoutError,
    ServiceBusError,
)
from azure.servicebus import AutoLockRenewer
import stripe
import psycopg2
import os
import json
import logging

logger = logging.getLogger(__name__)

connection_string = os.environ["AZURE_SERVICEBUS_CONNECTION_STRING"]
queue_name = os.environ["AZURE_SERVICEBUS_QUEUE_NAME"]

with ServiceBusClient.from_connection_string(connection_string) as client:
    with client.get_queue_receiver(
        queue_name,
        receive_mode=ServiceBusReceiveMode.PEEK_LOCK,
    ) as receiver:
        with AutoLockRenewer() as renewer:
            for msg in receiver:
                # Register the message for automatic lock renewal.
                # AutoLockRenewer renews the lock before LockDuration expires.
                # max_lock_renewal_duration sets how long to keep renewing (seconds).
                # Set it longer than your worst-case Stripe call duration.
                renewer.register(
                    receiver,
                    msg,
                    max_lock_renewal_duration=300,  # 5 minutes
                )

                try:
                    conn = psycopg2.connect(os.environ["DATABASE_URL"])
                    event = json.loads(str(msg))

                    result = process_billing_event(conn, event)

                    if result.get("skipped"):
                        # Pre-flight check found existing billing record.
                        # complete_message() removes the message from the queue.
                        # Do NOT abandon: abandoning redelivers and increments
                        # delivery_count toward MaxDeliveryCount.
                        receiver.complete_message(msg)
                        logger.info(
                            "Skipped %s/%s: already_billed (%s)",
                            event["customer_id"],
                            event["billing_period"],
                            result.get("charge_id"),
                        )
                        continue

                    # billing_records write completed inside process_billing_event.
                    # complete_message() removes the message — no redelivery.
                    receiver.complete_message(msg)
                    logger.info(
                        "Charged %s/%s: %s",
                        event["customer_id"],
                        event["billing_period"],
                        result.get("charge_id"),
                    )

                except MessageLockLostError:
                    # The lock expired before complete_message() was called.
                    # A competing consumer may have already received and processed
                    # this message. Do NOT abandon (lock is already lost — Service
                    # Bus will reject the abandon call). Do NOT retry here.
                    # The next delivery by the competing consumer hits the pre-flight
                    # check (billing_records row exists from our write) and skips
                    # the Stripe call. Log for monitoring but take no action.
                    logger.warning(
                        "Lock lost for %s/%s — competing consumer will handle",
                        event.get("customer_id"),
                        event.get("billing_period"),
                    )

                except stripe.error.Timeout:
                    # Stripe timed out. ch_A may or may not exist server-side.
                    # Do NOT call abandon_message() — abandon triggers immediate
                    # redelivery with no backoff, creating ch_B if ch_A exists.
                    # Instead, let the lock expire naturally: AutoLockRenewer will
                    # stop renewing (its max_lock_renewal_duration protects us from
                    # indefinite lock holding), and Service Bus will redeliver with
                    # the queue's built-in retry backoff (if configured via
                    # ServiceBusRetryOptions on the client). On redelivery, the
                    # content-hash idempotency key returns ch_A from Stripe within
                    # 24h. The pre-flight check skips the customer if billing_records
                    # was written before the timeout fired.
                    logger.warning(
                        "Stripe timeout for %s — letting lock expire for redelivery",
                        event.get("customer_id"),
                    )
                    # Do not call complete_message() or abandon_message().
                    # AutoLockRenewer will stop renewing after max_lock_renewal_duration.

                except stripe.error.CardError as e:
                    # Card declined — permanent failure. Write the decline to
                    # billing_records, then complete the message to prevent
                    # repeated card-decline events as delivery_count climbs to
                    # MaxDeliveryCount and Stripe re-attempts the card each time.
                    try:
                        with conn.cursor() as cur:
                            cur.execute(
                                """
                                INSERT INTO billing_records
                                    (customer_id, billing_period, charge_id,
                                     amount_cents, status, error, created_at)
                                VALUES (%s, %s, NULL, %s, 'card_declined', %s, NOW())
                                ON CONFLICT (customer_id, billing_period) DO NOTHING
                                """,
                                (
                                    event["customer_id"],
                                    event["billing_period"],
                                    event["amount_cents"],
                                    str(e),
                                ),
                            )
                            conn.commit()
                    finally:
                        receiver.complete_message(msg)

                except OperationTimeoutError as e:
                    # Service Bus AMQP operation timed out (lock renewal,
                    # management call, or connection reset). The Stripe call
                    # may or may not have completed. Do NOT abandon_message() —
                    # same reasoning as stripe.error.Timeout above. Let the
                    # lock expire naturally and let redelivery handle it with
                    # the idempotency key and pre-flight check in place.
                    logger.warning(
                        "Service Bus OperationTimeoutError for %s: %s — "
                        "letting lock expire",
                        event.get("customer_id"),
                        e,
                    )

                except Exception as e:
                    logger.error(
                        "Billing failed for %s: %s",
                        event.get("customer_id"),
                        e,
                    )
                    # For unknown errors, let the lock expire for redelivery.
                    # The idempotency key and pre-flight check make the next
                    # delivery safe regardless of what the current delivery did.

Comparison: protection levels across all three failure modes

FAQ

Does increasing LockDuration to 5 minutes prevent Stripe double charges from lock expiry?

It reduces the probability but does not eliminate it. The maximum LockDuration in Azure Service Bus is 5 minutes (300 seconds). A stripe-python call with max_network_retries=2 and a 30-second per-attempt timeout has a worst-case duration of approximately 90 seconds — well within a 5-minute lock. But the failure mode is not just about call duration: it is about AutoLockRenewer’s ability to successfully call renew_message_lock() before the lock expires. A namespace-level throttling event can cause ServerBusyError on the renewal call at T=45 even with a 300-second lock, causing the lock to expire at T=300 rather than being renewed at T=270. During that 30-second window between the renewal failure and lock expiry, a competing consumer in a multi-instance deployment can pick up the message. A longer LockDuration widens the window for successful renewal but does not eliminate the renewal-call-failure risk. The content-hash idempotency key and pre-flight check close the remaining window regardless of LockDuration setting.

How do I safely resubmit dead-letter messages without creating duplicate Stripe charges?

Preserve the original message_id when constructing the resubmitted ServiceBusMessage, and add the pre-flight check to the billing callback. To preserve message_id: ServiceBusMessage(body, message_id=dlq_msg.message_id) explicitly sets the message ID on the new message object before sending. This ensures the resubmitted message carries the original message_id, making a message_id-based idempotency key stable across resubmission. However, this relies on all resubmission tooling consistently setting the original message_id — which Azure Portal and many third-party Service Bus Explorer tools do not do by default. The content-hash idempotency key (derived from business fields, not message_id) is more robust because it does not depend on tooling behavior. The pre-flight check on billing_records closes both cases: it prevents re-charging regardless of what message_id the resubmitted message carries, because it queries by (customer_id, billing_period) — fields present in the message body, not the envelope.

What happens if AutoLockRenewer fails and the lock expires mid-Stripe-call?

The first consumer’s Stripe call continues to completion — lock expiry does not cancel the HTTP request. When stripe.charges.create() returns ch_A, the first consumer writes to billing_records and then calls receiver.complete_message(msg). If the lock has already expired, complete_message() raises MessageLockLostError. At this point, a competing consumer may have already picked up the message and called stripe.charges.create() for the same customer. If the first consumer’s billing_records write completed before the competing consumer’s Stripe call, the competing consumer’s pre-flight check finds the existing row and skips its own Stripe call, calling complete_message() on its own lock token (which is still valid). If the competing consumer’s Stripe call was in-flight simultaneously with the first consumer’s, the content-hash idempotency key ensures Stripe returns ch_A to both callers, and the ON CONFLICT DO NOTHING constraint in the billing_records INSERT ensures only one row is written. The MessageLockLostError on the first consumer is expected and can be caught and logged without further action.

Should I use receiver.defer_message() instead of abandon_message() for Stripe timeouts?

defer_message() is preferable to abandon_message() for Stripe timeouts because it removes the message from the normal consumer receive flow without triggering immediate redelivery. A deferred message is only accessible via its sequence_number using receiver.receive_deferred_messages([sequence_number]) — you must explicitly retrieve and process it. This gives you control over retry timing. However, defer_message() requires the operator to track deferred sequence_number values (typically in a side store) and re-process them explicitly. The simplest pattern for Stripe timeouts remains: let the lock expire naturally (do not call abandon_message() or defer_message()), and rely on the content-hash idempotency key and pre-flight check to make the next automatic Service Bus redelivery safe. This requires no additional infrastructure and closes the failure mode for all redelivery paths including DLQ resubmission.

Does the Stripe idempotency key work across different Service Bus namespaces in a geo-redundant setup?

Yes. The content-hash idempotency key is derived entirely from business fields in the message body (customer_id and billing_period) — it has no dependency on Service Bus namespace, queue name, region, or message metadata. In a geo-redundant setup where the same usage event is replicated across a primary namespace in East US and a failover namespace in West US, both regions produce identical idempotency keys for the same billing event. If a failover event causes both namespaces to deliver the same message simultaneously, Stripe’s 24-hour deduplication window returns ch_A to the second caller with HTTP 200 (not an error) — the second consumer sees ch_A, writes the billing_records row (the ON CONFLICT DO NOTHING constraint handles any race with the first write), and completes its message. The vault key’s spend cap provides an additional layer in the rare case where both consumers’ Stripe calls land within the same millisecond before Stripe has flushed ch_A’s idempotency record to all API gateway nodes — a sub-millisecond race that the idempotency key alone cannot close but the vault cap limits to expected_total × 1.10.