APScheduler Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

APScheduler is the most widely used Python job scheduling library, embedded in thousands of Flask, FastAPI, and standalone agent services to drive billing crons and periodic automation. Three deployment-level failure modes surface specifically when APScheduler jobs call Stripe — and all three produce duplicate charges that look like successful completions in the scheduler logs.

This post covers three APScheduler-specific failure modes — multi-worker WSGI process duplication, MemoryJobStore state loss on restart, and coalesce=False catch-up fire explosion — each of which produces multiple stripe.charges.create() calls against already-billed customers, and the two-layer governance pattern that closes all three: content-hash idempotency keys plus per-run vault keys via a spend-cap proxy, without changing your job topology.

Failure mode 1: BackgroundScheduler embedded in a multi-worker WSGI app fires one billing execution per worker process

APScheduler's BackgroundScheduler runs a background thread inside the process that creates it. When a Flask or FastAPI application instantiates a BackgroundScheduler inside its startup hook and the application is served by Gunicorn or uWSGI with multiple worker processes, each worker process creates its own scheduler instance. There is no coordination between workers. All schedulers are backed by independent MemoryJobStore instances that share nothing. Each scheduler fires its jobs according to its own clock and its own view of when each job last ran.

When the billing cron triggers at 00:00 UTC, all N workers fire charge_all_customers() simultaneously. Each worker process reads stripe.api_key = os.environ["STRIPE_SECRET_KEY"] from its own module-level state — the same unrestricted live key — and calls stripe.charges.create() for every customer in the billing cohort. With 4 Gunicorn workers, every customer is charged 4 times. The Gunicorn access log shows 4 successful billing runs. APScheduler reports no errors. The Stripe dashboard shows 4 charge objects per customer created within a few milliseconds of each other, all with status: succeeded.

# app.py — UNSAFE: BackgroundScheduler inside a WSGI app with multiple workers
from flask import Flask
from apscheduler.schedulers.background import BackgroundScheduler
import os
import stripe

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]  # shared, unrestricted

app = Flask(__name__)

def charge_all_customers():
    customers = db.fetch_unpaid_customers(billing_period="2026-07")
    for c in customers:
        # With 4 Gunicorn workers: 4 × stripe.charges.create() per customer
        stripe.charges.create(
            amount=c["amount_cents"],
            currency="usd",
            customer=c["id"],
            description="Monthly subscription July 2026",
        )

scheduler = BackgroundScheduler()
scheduler.add_job(charge_all_customers, "cron", hour=0, minute=0)
scheduler.start()

# gunicorn app:app --workers 4
# → 4 BackgroundScheduler instances → 4 billing runs at 00:00 UTC → 4× charges

The fix has two layers. The first is architectural: run APScheduler in a dedicated scheduler process that does not fork. A standalone Python script with a BlockingScheduler, or a separate Gunicorn worker with --workers 1 designated for scheduling, runs exactly one scheduler instance regardless of how many application workers serve HTTP traffic. The second layer is a content-hash idempotency key inside the billing function — so that even if the architectural fix is incomplete (e.g., a blue/green deployment briefly runs two scheduler instances during a rollover), Stripe deduplicates the second charge rather than creating a new one.

# scheduler.py — SAFE: dedicated scheduler process (run as separate service/container)
import hashlib
import os
import requests
import stripe
from apscheduler.schedulers.blocking import BlockingScheduler

scheduler = BlockingScheduler()

@scheduler.scheduled_job("cron", hour=0, minute=0)
def charge_all_customers():
    billing_period = get_current_billing_period()
    customers = db.fetch_unpaid_customers(billing_period)
    if not customers:
        return

    total_cents = sum(c["amount_cents"] for c in customers)
    resp = requests.post(
        "https://proxy.keybrake.com/keys",
        headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
        json={
            "vendor": "stripe",
            "allowed_endpoints": ["POST /v1/charges"],
            "daily_usd_cap": round((total_cents / 100) * 1.10, 2),
            "expires_in_seconds": 7200,
            "label": f"apscheduler-billing-{billing_period}",
        },
        timeout=10,
    )
    resp.raise_for_status()
    vault_key = resp.json()["vault_key"]

    stripe_client = stripe.Stripe(
        vault_key,
        base_url="https://proxy.keybrake.com/stripe/v1",
    )

    for c in customers:
        idempotency_key = hashlib.sha256(
            f"{c['id']}:{c['amount_cents']}:{billing_period}:apscheduler-billing".encode()
        ).hexdigest()[:32]

        try:
            stripe_client.charges.create(
                amount=c["amount_cents"],
                currency="usd",
                customer=c["id"],
                description=f"Subscription {billing_period}",
                idempotency_key=idempotency_key,
            )
        except stripe.error.CardError:
            db.record_decline(c["id"], billing_period)
            continue  # permanent — do not raise, scheduler does not retry
        except stripe.error.InvalidRequestError:
            db.record_error(c["id"], billing_period, "invalid_request")
            continue

scheduler.start()

# Run as: python scheduler.py — exactly one process, no fork, no duplication

Failure mode 2: MemoryJobStore state loss on restart fires catch-up billing executions against already-charged customers

APScheduler's default job store is MemoryJobStore, which holds all job state — including each job's last_run_time and computed next_run_time — in the Python process's heap. When the process exits, this state is lost. On restart, APScheduler reconstructs job fire times from the trigger definition and the current wall clock. For a cron trigger set to fire at 00:00 UTC monthly, APScheduler asks: "what was the most recent scheduled time before now?" If it is currently 00:05 UTC on the first of the month, APScheduler computes that the job was due at 00:00 UTC and fires a catch-up execution immediately.

The critical failure window is any restart that occurs after a billing job has already succeeded but before the next scheduled interval. A process restart at 00:01 UTC — due to a deploy, OOM kill, or exception in unrelated application code — causes APScheduler to fire the billing cron again for the same period. The billing function calls stripe.charges.create() for every customer in the cohort. Without idempotency keys, each call creates a new charge object. The application logs show a normal billing run. The process restart log shows a normal startup. The Stripe dashboard shows two charges per customer created 1 minute apart.

# UNSAFE: MemoryJobStore is default — state lost on every process restart
from apscheduler.schedulers.blocking import BlockingScheduler
import os, stripe

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

scheduler = BlockingScheduler()

@scheduler.scheduled_job("cron", day=1, hour=0, minute=0)
def monthly_billing():
    customers = db.fetch_unpaid_customers("2026-07")
    for c in customers:
        # Process restart at 00:01 UTC → APScheduler fires this again
        # → second stripe.charges.create() per customer with no idempotency key
        stripe.charges.create(
            amount=c["amount_cents"],
            currency="usd",
            customer=c["id"],
        )

scheduler.start()

Two fixes work in combination. The first is a SQLAlchemyJobStore backed by a persistent database — SQLite, PostgreSQL, or MySQL — which persists each job's next_run_time across restarts. When the process restarts at 00:01 UTC, APScheduler reads the persisted next_run_time of "2026-08-01 00:00:00" and does not fire a catch-up execution. The second fix is the content-hash idempotency key inside the billing function, which closes the residual race: a persistent store is authoritative only after the first process has committed the updated next_run_time, which happens after the job fires. In the few seconds between job dispatch and next_run_time persistence, a restart can still trigger a second execution — the idempotency key makes that execution a safe no-op at Stripe rather than a duplicate charge.

# SAFE: SQLAlchemyJobStore persists next_run_time across restarts
import hashlib
import os
import stripe
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore

jobstores = {
    "default": SQLAlchemyJobStore(url=os.environ["DATABASE_URL"])
}

scheduler = BlockingScheduler(jobstores=jobstores)

@scheduler.scheduled_job(
    "cron",
    day=1,
    hour=0,
    minute=0,
    id="monthly_billing",          # stable ID required for persistent store lookup
    replace_existing=True,         # update trigger if code changes between deploys
    coalesce=True,                 # fire at most once on recovery, not once per miss
    misfire_grace_time=3600,       # ignore missed fires older than 1 hour
)
def monthly_billing():
    billing_period = get_current_billing_period()
    customers = db.fetch_unpaid_customers(billing_period)

    stripe_client = stripe.Stripe(
        os.environ["KEYBRAKE_VAULT_KEY"],
        base_url="https://proxy.keybrake.com/stripe/v1",
    )

    for c in customers:
        idempotency_key = hashlib.sha256(
            f"{c['id']}:{c['amount_cents']}:{billing_period}:apscheduler-billing".encode()
        ).hexdigest()[:32]

        try:
            stripe_client.charges.create(
                amount=c["amount_cents"],
                currency="usd",
                customer=c["id"],
                idempotency_key=idempotency_key,
            )
        except stripe.error.CardError:
            db.record_decline(c["id"], billing_period)
            continue
        except stripe.error.InvalidRequestError:
            continue

scheduler.start()

Failure mode 3: coalesce=False fires one billing execution per missed scheduled interval on recovery

APScheduler's coalesce job option controls how missed scheduled firings are handled when the scheduler has been unavailable. The default is coalesce=True, which collapses all missed executions into a single catch-up execution — if the scheduler was down for 3 hours during an hourly billing job, one catch-up execution fires on recovery. Setting coalesce=False reverses this: APScheduler fires one execution per missed interval. Three hours of scheduler downtime with an hourly billing cron fires 3 billing executions in rapid succession on recovery.

The billing failure is not immediately obvious because coalesce=False is often set deliberately — developers want "run everything that was missed, not just one catch-up." For a data pipeline that processes hourly log files, missing 3 hours means 3 genuinely distinct batches that all need to run. But for a monthly billing job, missing 3 hours of downtime means the billing period did not change — the same billing_period = "2026-07" value is used by all 3 catch-up executions. Each of the 3 concurrent billing functions calls stripe.charges.create() for the full customer cohort, producing up to 3 charges per customer within seconds.

# UNSAFE: coalesce=False on a billing job that fires multiple times for the same period
from apscheduler.schedulers.blocking import BlockingScheduler
import stripe, os

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

scheduler = BlockingScheduler()

@scheduler.scheduled_job(
    "interval",
    hours=1,
    coalesce=False,   # fires once per missed interval on recovery
    max_instances=3,  # allows 3 concurrent executions
)
def hourly_billing_sync():
    billing_period = "2026-07"   # same value for all 3 catch-up executions
    customers = db.fetch_unpaid_customers(billing_period)
    for c in customers:
        # 3 catch-up executions → each calls stripe.charges.create() independently
        stripe.charges.create(
            amount=c["amount_cents"],
            currency="usd",
            customer=c["id"],
        )

scheduler.start()

Three configuration changes close this failure mode. First, set coalesce=True on billing jobs — billing periods are idempotent by definition; one catch-up execution is always correct, and multiple catch-up executions for the same period produce duplicates. Second, set max_instances=1 to prevent concurrent executions of the same job — APScheduler's default is already 1, but it is easy to accidentally override during debugging. Third, add a content-hash idempotency key inside the billing function: if both coalesce=True and max_instances=1 somehow allow two concurrent executions (e.g., during a deployment that briefly runs two scheduler processes), Stripe deduplicates rather than creates a second charge. The idempotency key is the last line of defense when scheduler configuration is violated.

# SAFE: coalesce=True + max_instances=1 on billing jobs
import hashlib
import os
import requests
import stripe
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore

scheduler = BlockingScheduler(
    jobstores={"default": SQLAlchemyJobStore(url=os.environ["DATABASE_URL"])}
)

@scheduler.scheduled_job(
    "interval",
    hours=1,
    id="hourly_billing_sync",
    coalesce=True,          # collapse all missed fires into one catch-up execution
    max_instances=1,        # no concurrent executions of the same job
    misfire_grace_time=300, # ignore misses older than 5 minutes (not worth catching up)
    replace_existing=True,
)
def hourly_billing_sync():
    billing_period = get_current_billing_period()
    customers = db.fetch_unbilled_customers(billing_period)
    if not customers:
        return

    total_cents = sum(c["amount_cents"] for c in customers)
    resp = requests.post(
        "https://proxy.keybrake.com/keys",
        headers={"Authorization": f"Bearer {os.environ['KEYBRAKE_API_KEY']}"},
        json={
            "vendor": "stripe",
            "allowed_endpoints": ["POST /v1/charges"],
            "daily_usd_cap": round((total_cents / 100) * 1.10, 2),
            "expires_in_seconds": 7200,
            "label": f"apscheduler-billing-{billing_period}",
        },
        timeout=10,
    )
    resp.raise_for_status()
    vault_key = resp.json()["vault_key"]

    stripe_client = stripe.Stripe(
        vault_key,
        base_url="https://proxy.keybrake.com/stripe/v1",
    )

    for c in customers:
        idempotency_key = hashlib.sha256(
            f"{c['id']}:{c['amount_cents']}:{billing_period}:apscheduler-billing".encode()
        ).hexdigest()[:32]

        try:
            stripe_client.charges.create(
                amount=c["amount_cents"],
                currency="usd",
                customer=c["id"],
                idempotency_key=idempotency_key,
            )
        except stripe.error.CardError:
            db.record_decline(c["id"], billing_period)
            continue
        except stripe.error.InvalidRequestError:
            continue

scheduler.start()

Approach comparison

Gap analysis

1. SQLAlchemyJobStore concurrent polling race allows two scheduler processes to both execute the same job in the same polling window

When multiple processes share a SQLAlchemyJobStore, APScheduler uses a SELECT + UPDATE pattern to claim each job. The scheduler reads the job row, checks if next_run_time <= now, and if so computes the new next_run_time and writes it back. Under concurrent load — two scheduler processes polling within the same scheduler wakeup interval — both processes can read the same job row before either has committed the updated next_run_time. Both processes execute the job. APScheduler does not use a SELECT FOR UPDATE or advisory lock by default on all database backends; SQLite, in particular, has weaker locking semantics than PostgreSQL. The fix is to run a single dedicated scheduler process and not share the job store across multiple simultaneously active schedulers. The idempotency key closes any residual race during deployments where two scheduler processes overlap briefly.

2. misfire_grace_time=None fires every missed execution regardless of age

APScheduler's misfire_grace_time controls the maximum age of a missed firing that will be caught up. The default is None on most triggers, which means "fire regardless of how old the miss is." If your scheduler was unexpectedly offline for a week — perhaps because it was a Docker container that was not restarted after a node failure — returning it to service with misfire_grace_time=None fires all missed billing executions for the entire offline period. For a daily billing job, that is 7 billing runs in rapid succession. Set misfire_grace_time to a value smaller than your billing interval — for example, 3600 seconds for a daily job — so that misses older than 1 hour are discarded rather than caught up. Billing periods are addressed by the next scheduled run; there is no value in catching up a billing run from 3 days ago with stale customer data.

3. replace_existing=False silently keeps the old job definition after a scheduler code change

When a SQLAlchemyJobStore persists jobs across restarts and a developer changes the billing function's cron schedule from monthly to weekly, the updated job definition in code coexists with the existing persisted job. If replace_existing=False (the default when using add_job() with an explicit id), APScheduler silently keeps the persisted job with the old schedule and ignores the new trigger. The billing job continues firing monthly even though the code was changed to weekly. After you ship a billing schedule change, verify the persisted job's next_run_time in the database directly, not just in application logs. Always use replace_existing=True on jobs defined in startup code so the persisted record is updated on every deploy.

4. ThreadPoolExecutor with max_workers=N allows max_instances to be reached without WSGI worker explosion — concurrent billing threads within one process share module-level state

Even in a single-process scheduler, BackgroundScheduler uses a ThreadPoolExecutor by default. If max_instances is raised above 1 and a billing job is still running when the next scheduled firing arrives — because the current batch is unusually large — APScheduler dispatches a second thread. Both threads execute the billing function simultaneously in the same process. Both threads read stripe.api_key from the module-level variable, which is a single shared object — assigning a new value in one thread writes it for all threads. If the billing function ever mutates stripe.api_key mid-execution (e.g., to switch between test and live keys), the second thread reads the mid-mutation value. Use the per-client pattern — stripe.Stripe(vault_key, base_url=proxy) per job invocation — so each thread holds its own client object with no shared mutable state between billing executions.

Enforcement tests

# tests/test_apscheduler_billing.py
import hashlib
import pytest
from unittest.mock import patch, MagicMock, call

def make_idempotency_key(customer_id: str, amount_cents: int, billing_period: str) -> str:
    return hashlib.sha256(
        f"{customer_id}:{amount_cents}:{billing_period}:apscheduler-billing".encode()
    ).hexdigest()[:32]

class TestAPSchedulerBillingIdempotency:
    def test_idempotency_key_stable_across_restarts(self):
        """Same billing args on first run and catch-up run produce the same key."""
        key_run1 = make_idempotency_key("cus_ABC", 2999, "2026-07")
        key_run2 = make_idempotency_key("cus_ABC", 2999, "2026-07")
        assert key_run1 == key_run2
        assert len(key_run1) == 32

    def test_idempotency_key_unique_per_period(self):
        """Same customer in different billing periods produces different keys."""
        key_jul = make_idempotency_key("cus_ABC", 2999, "2026-07")
        key_aug = make_idempotency_key("cus_ABC", 2999, "2026-08")
        assert key_jul != key_aug

    def test_card_declined_suppresses_exception_no_raise(self):
        """CardError is caught, decline recorded, function continues — scheduler does not retry."""
        import stripe
        with patch("stripe.Stripe") as mock_stripe_cls:
            mock_client = MagicMock()
            mock_stripe_cls.return_value = mock_client
            mock_client.charges.create.side_effect = stripe.error.CardError(
                "Your card was declined.", None, "card_declined"
            )
            with patch("scheduler.db") as mock_db:
                mock_db.fetch_unbilled_customers.return_value = [
                    {"id": "cus_FAIL", "amount_cents": 2999}
                ]
                mock_db.record_decline = MagicMock()
                hourly_billing_sync()
                mock_db.record_decline.assert_called_once_with("cus_FAIL", pytest.approx(any))

    def test_coalesce_true_prevents_multi_execution(self):
        """BlockingScheduler with coalesce=True fires at most one catch-up job."""
        from apscheduler.schedulers.blocking import BlockingScheduler
        scheduler = BlockingScheduler()
        job = scheduler.add_job(
            lambda: None,
            "interval",
            hours=1,
            coalesce=True,
            max_instances=1,
        )
        assert job.coalesce is True
        assert job.max_instances == 1

    def test_vault_key_cap_includes_ten_percent_buffer(self):
        """Per-run vault key spend cap is total of all customer amounts × 1.10."""
        customers = [{"amount_cents": 4999} for _ in range(8)]
        total_cents = sum(c["amount_cents"] for c in customers)
        expected_cap = round((total_cents / 100) * 1.10, 2)
        assert expected_cap == pytest.approx(43.991)

FAQ

Can I keep BackgroundScheduler inside my Flask app if I run Gunicorn with a single worker?

Yes — with exactly one worker process there is no duplication. The risk appears the moment you scale beyond one worker, which is common when requests per second grows. A safer architectural pattern is to always run scheduling in a dedicated process regardless of worker count: a separate scheduler.py using BlockingScheduler, deployed as a separate container or Supervisor process. This guarantees one scheduler instance independent of how you scale your HTTP tier, and it allows you to scale HTTP workers freely without accidentally multiplying billing executions.

Does max_instances=1 protect against the multi-worker WSGI duplication?

No. max_instances=1 prevents a job from running more than once concurrently within a single scheduler instance. It has no awareness of other scheduler instances in other processes. Each worker process's BackgroundScheduler enforces its own max_instances limit independently — so each scheduler allows one instance of the job, and with 4 workers you get 4 concurrent job instances across 4 schedulers. max_instances is a concurrency guard, not a cross-process dedup mechanism.

How do I handle a legitimate billing catch-up after a planned maintenance window?

If you intentionally took the scheduler down during a billing window and need to run the missed billing job manually, trigger it via a dedicated management command rather than relying on APScheduler's catch-up mechanism. A python manage.py run_billing --period 2026-07 command calls the billing function directly with an explicit billing period argument. The content-hash idempotency key derived from the period ensures the run is a no-op for customers already charged. This is safer than relying on misfire_grace_time catch-up because you control exactly which period runs and can verify the results before the next scheduled execution fires.

What is the correct misfire_grace_time for a monthly billing job?

Set it to a value smaller than your billing interval but large enough to absorb normal scheduler start latency. For a monthly billing job, misfire_grace_time=3600 (1 hour) works well: if the scheduler starts within 1 hour of the scheduled fire time, the catch-up executes; if the scheduler was down for more than 1 hour, the miss is discarded and billing waits for the next month's scheduled run. Combined with a SQLAlchemyJobStore (so state survives restarts) and a content-hash idempotency key (so a catch-up against already-charged customers is safe), this configuration handles the common restart case without accumulating stale catch-up executions from extended outages.

Does the vault key pattern work with APScheduler's AsyncIOScheduler?

Yes. Replace the synchronous stripe.Stripe client with stripe.AsyncStripe and use await stripe_client.charges.create() inside the async job function. The vault key is a string argument — it passes through AsyncIOScheduler's job executor identically to BlockingScheduler. Issue the vault key before dispatching the billing coroutine, cap it at the total batch amount × 1.10, and set a TTL that covers the expected async batch duration. The proxy enforces the spend cap across all concurrent async calls sharing the key in the same way it enforces it across concurrent threads or processes.

How do I prevent a blue/green deployment from briefly running two scheduler processes simultaneously?

The idempotency key is your primary defense here. During a rolling deploy, the old scheduler process and the new scheduler process can both be running for up to the deployment duration (typically 30–120 seconds). If a billing job fires during that window, both schedulers may execute it. With a content-hash idempotency key, the second execution returns the original charge object from Stripe rather than creating a new one — the duplicate execution is safe. For additional protection, use a SQLAlchemyJobStore so the first process to write the updated next_run_time prevents the second process from seeing the job as due in subsequent polling cycles. The combination of an idempotency key (closes the per-call race) and a persistent job store (closes the scheduling-level race) is sufficient for zero-downtime deploys.