Quartz Scheduler and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Quartz Scheduler is the de-facto Java job scheduler for recurring billing pipelines — monthly invoice jobs, per-seat subscription charges, usage-aggregation crons. Its persistence layer, clustering support, and misfire-handling instructions create three billing failure modes that are invisible in local development and surface only in production under GC pressure, scheduler restarts, or multi-node deployments.

This post covers those three failure modes with Java Quartz 2.3 code, content-hash idempotency keys stable across misfire re-fires and cluster node assignments, per-billing-period vault keys via a spend-cap proxy, and pre-flight database checks — the two-layer governance pattern that closes all three without changing your trigger topology or replacing Quartz with a different scheduler.

Failure mode 1: getFireTime() and getScheduledFireTime() both change on misfire re-fire — any timestamp-based idempotency key produces a different hash and Stripe creates ch_B

Quartz provides two time values in the JobExecutionContext that developers reach for when building idempotency keys for recurring billing jobs: context.getFireTime() and context.getScheduledFireTime(). The names suggest stability — one is when the trigger actually fired, the other is when it was scheduled to fire. Neither is stable across misfire re-fires.

context.getFireTime() is the actual JVM wall-clock instant Quartz started the job thread. It is unique per execution by definition. A billing job fires at 2026-06-01T00:00:00Z, calls Charge.create() with key sha256("cust_123:2026-06-01T00:00:00Z"), receives ch_A, then crashes (OOM kill, SIGKILL from rolling deploy, unhandled exception after the Stripe call returns but before the job marks itself complete). Quartz detects the scheduler did not mark the trigger COMPLETE and applies its misfire instruction. The re-fire executes at 2026-06-01T04:17:00Z. context.getFireTime() returns 2026-06-01T04:17:00Z. The key is sha256("cust_123:2026-06-01T04:17:00Z") — completely different from the original. Stripe has no record of this key and creates ch_B.

context.getScheduledFireTime() appears to be the answer: it represents the trigger’s intended fire time, which should be stable for a given trigger instance. For CronTrigger with the default misfire instruction MISFIRE_INSTRUCTION_FIRE_ONCE_NOW, however, Quartz generates a new trigger instance for the re-fire and sets its scheduledFireTime to the current wall clock at the moment the misfire handler runs — not to the original missed fire time. The re-fire at 2026-06-01T04:17:00Z has getScheduledFireTime() returning 2026-06-01T04:17:00Z. Same result as getFireTime(); same new key; same ch_B.

The only CronTrigger misfire instruction that preserves the original fire time in getScheduledFireTime() is MISFIRE_INSTRUCTION_DO_NOTHING, which skips the missed fire entirely and waits for the next scheduled occurrence. A billing team that adopts MISFIRE_INSTRUCTION_DO_NOTHING to get a stable scheduled time is also skipping the re-fire altogether — missed billing runs are not retried, a different problem than duplicate charges but equally wrong for a recurring billing pipeline.

// UNSAFE: timestamp from JobExecutionContext in the idempotency key
public class BillingJob implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        List<Customer> customers = customerRepo.findDueForBilling();

        for (Customer c : customers) {
            // getFireTime(): wall-clock instant the job started — different on misfire re-fire
            long fireEpochMs = context.getFireTime().getTime();

            // sha256("cust_123:Q3-2026:1748736000000") on original fire
            // sha256("cust_123:Q3-2026:1748751420000") on misfire re-fire → ch_B
            String key = DigestUtils.sha256Hex(
                c.getId() + ":" + c.getBillingPeriod() + ":" + fireEpochMs
            ).substring(0, 32);

            Charge.create(ChargeCreateParams.builder()
                .setAmount(c.getAmountCents())
                .setCurrency("usd")
                .setCustomer(c.getStripeCustomerId())
                .putIdempotencyKey(key)  // different key on re-fire → ch_B
                .build());

            billingRepo.save(new BillingRecord(c.getId(), c.getBillingPeriod(), key));
        }
    }
}

The variant using getScheduledFireTime() is more insidious because it looks like the correct fix after reading that getFireTime() is unstable. A developer who switches to getScheduledFireTime() and deploys to production will see the idempotency work correctly in the normal path (no misfire) and in testing (local schedulers rarely misfire). The failure surfaces only when the scheduler restarts after a crash — exactly when the billing was already partially completed and the risk of duplicate charges is highest.

// ALSO UNSAFE: getScheduledFireTime() with MISFIRE_INSTRUCTION_FIRE_ONCE_NOW (CronTrigger default)
// Quartz sets scheduledFireTime to "now" when the misfire handler fires — not to the original missed time
public class BillingJob implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        List<Customer> customers = customerRepo.findDueForBilling();

        for (Customer c : customers) {
            // scheduledFireTime is set by Quartz misfire handler to current wall clock
            // for MISFIRE_INSTRUCTION_FIRE_ONCE_NOW — NOT the original cron-scheduled time
            Date scheduledTime = context.getScheduledFireTime();

            // On normal fire: sha256("cust_123:Q3-2026:2026-06-01T00:00:00Z") → ch_A
            // On misfire re-fire: sha256("cust_123:Q3-2026:2026-06-01T04:17:00Z") → ch_B
            String key = DigestUtils.sha256Hex(
                c.getId() + ":" + c.getBillingPeriod() + ":" + scheduledTime.toInstant().toString()
            ).substring(0, 32);

            Charge.create(ChargeCreateParams.builder()
                .setAmount(c.getAmountCents())
                .setCurrency("usd")
                .setCustomer(c.getStripeCustomerId())
                .putIdempotencyKey(key)
                .build());

            billingRepo.save(new BillingRecord(c.getId(), c.getBillingPeriod(), key));
        }
    }
}

The failure is compounded by the loop structure. The billing job iterates over all customers in sequence. Customer 1 through customer 47 are charged successfully (ch_A₁ through ch_A₄₇) before the crash. The misfire re-fire starts from the beginning of the list. Customers 1 through 47 receive a second call to Charge.create() with a different timestamp-derived key. Stripe’s 24-hour idempotency cache, already expired by the time the misfire fires, provides no protection. All 47 customers are double-charged.

The fix for failure mode 1

A content-hash idempotency key derived exclusively from stable business fields — customer ID and billing period — is identical on the original fire and on every misfire re-fire, regardless of which timestamp Quartz provides in the JobExecutionContext. The key must not include any value that changes between executions: getFireTime() (different wall clock on every fire), getScheduledFireTime() (set to current time by misfire handler under FIRE_ONCE_NOW), System.currentTimeMillis() or Instant.now() captured at job start, the Quartz trigger key name or group (stable, but unnecessary and fragile on trigger rename), or the job execution instance ID.

A pre-flight PostgreSQL check using ON CONFLICT DO NOTHING provides a second layer. When the misfire re-fire reaches a customer already charged before the crash, the pre-flight insert finds the row already present and returns zero rows affected. The job skips the Stripe call and moves to the next customer. Customers not yet charged (48 through N) are processed normally. The billing run completes correctly without any duplicate charges.

import org.apache.commons.codec.digest.DigestUtils;
import com.stripe.model.Charge;
import com.stripe.param.ChargeCreateParams;

public class SafeBillingJob implements Job {

    private final BillingRepository billingRepo;
    private final CustomerRepository customerRepo;

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        List<Customer> customers = customerRepo.findDueForBilling();

        for (Customer c : customers) {
            String key = makeIdempotencyKey(c.getId(), c.getBillingPeriod());

            // Pre-flight: claim the billing slot before calling Stripe.
            // ON CONFLICT DO NOTHING on (customer_id, billing_period) returns 0 rows affected
            // if this customer was already charged — on original fire OR on misfire re-fire.
            int rowsAffected = billingRepo.insertIfAbsent(
                c.getId(), c.getBillingPeriod(), key
            );
            if (rowsAffected == 0) {
                continue; // already charged — misfire re-fire caught by pre-flight
            }

            try {
                Charge charge = Charge.create(ChargeCreateParams.builder()
                    .setAmount(c.getAmountCents())
                    .setCurrency("usd")
                    .setCustomer(c.getStripeCustomerId())
                    .putIdempotencyKey(key) // Stripe-side cache within 24h window
                    .build());

                billingRepo.updateChargeId(key, charge.getId());
            } catch (Exception e) {
                // If Stripe call fails: pre-flight row stays with status='pending'.
                // On re-fire, rowsAffected=0, job skips Stripe. A reconciliation job
                // reissues Charge.create() with the same key for 'pending' rows older than 5m.
                throw new JobExecutionException(e);
            }
        }
    }

    static String makeIdempotencyKey(String customerId, String billingPeriod) {
        // Stable fields only: customerId + billingPeriod + namespace.
        // Must NOT include: getFireTime(), getScheduledFireTime(), System.currentTimeMillis(),
        // Instant.now(), trigger key name (fragile on rename), job execution ID (per-run).
        // Produces the same value on original fire AND on every misfire re-fire.
        return DigestUtils.sha256Hex(
            customerId + ":" + billingPeriod + ":quartz-billing"
        ).substring(0, 32);
    }
}

-- billing_records schema
CREATE TABLE billing_records (
    customer_id      TEXT NOT NULL,
    billing_period   TEXT NOT NULL,
    idempotency_key  TEXT NOT NULL,
    status           TEXT NOT NULL DEFAULT 'pending',
    charge_id        TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT billing_records_pk PRIMARY KEY (idempotency_key),
    CONSTRAINT billing_records_uq UNIQUE (customer_id, billing_period)
);

-- Pre-flight insert — returns 0 rows affected if already exists
INSERT INTO billing_records (customer_id, billing_period, idempotency_key)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id, billing_period) DO NOTHING;

Failure mode 2: Clustered Quartz GC pause causes DB lock timeout, two nodes both fire the same trigger concurrently — node-local value in idempotency key produces ch_B on the second node

Quartz clustering uses a database row lock — specifically a SELECT ... FOR UPDATE on the QRTZ_LOCKS table row named TRIGGER_ACCESS — to serialize trigger acquisition across all cluster nodes. Each Quartz node, before firing any trigger, acquires this lock, reads the QRTZ_TRIGGERS table, selects triggers due to fire, marks them ACQUIRED in the same transaction, releases the lock, and then starts the job threads. This serializes trigger selection across nodes: only one node can be in the trigger-acquisition transaction at a time.

The lock-based model has a known failure mode under JVM GC pressure. A billing Quartz node running a stop-the-world full GC pauses all application threads — including the scheduler’s acquire thread — for the duration of the GC pause. If the GC pause exceeds the database connection’s statement timeout or idle timeout (common in connection pools configured with short defaults: HikariCP’s default idleTimeout is 10 minutes, but a long GC pause can cause the DB to close the connection), the QRTZ_LOCKS row lock is released by the database when the connection is dropped. The trigger row reverts from ACQUIRED to WAITING (the transaction that set it to ACQUIRED is rolled back when the connection drops).

A second Quartz node, running its next scheduling loop while the first node is paused, acquires the lock, reads the trigger in WAITING state, marks it ACQUIRED, releases the lock, and starts the billing job. The first node’s GC pause ends. Its scheduler thread resumes. The billing job it started before the GC pause is still running in the job thread pool — Quartz does not kill in-progress job threads on connection failure. Both nodes now have an active execute() call for the same trigger fire time. Both iterate over the customer list. Both call Charge.create() for the same customers.

If either node includes a node-local value in the Stripe idempotency key — the server hostname, a JVM-local random seed, a ThreadLocal counter, a SchedulerInstanceId from context.getScheduler().getSchedulerInstanceId() — the two keys differ and Stripe creates ch_B on the second call. Even without a node-local value, two concurrent calls to Charge.create() with the same key within Stripe’s 24-hour window will race: Stripe allows concurrent identical idempotency key requests but the second one blocks until the first completes and then returns the cached response. If Node B’s call arrives while Node A’s is still in-flight, Stripe returns a 409 Conflict for the second request. Many Stripe client libraries surface this as a StripeException. If Node B’s billing job catches and silently ignores this exception, it leaves a pending row in the billing table without a charge_id — the reconciliation job then re-issues the charge, creating ch_B the next morning.

// UNSAFE: SchedulerInstanceId in the idempotency key — different per cluster node
public class BillingJob implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        List<Customer> customers = customerRepo.findDueForBilling();

        for (Customer c : customers) {
            String schedulerInstanceId;
            try {
                // SchedulerInstanceId is unique per Quartz node — e.g., "billing-node-1" vs "billing-node-2"
                // or an auto-generated UUID if instanceId=AUTO in quartz.properties
                schedulerInstanceId = context.getScheduler().getSchedulerInstanceId();
            } catch (SchedulerException e) {
                schedulerInstanceId = "unknown";
            }

            // Node A: sha256("cust_123:Q3-2026:billing-node-1") → ch_A
            // Node B: sha256("cust_123:Q3-2026:billing-node-2") → ch_B — Stripe creates second charge
            String key = DigestUtils.sha256Hex(
                c.getId() + ":" + c.getBillingPeriod() + ":" + schedulerInstanceId
            ).substring(0, 32);

            Charge.create(ChargeCreateParams.builder()
                .setAmount(c.getAmountCents())
                .setCurrency("usd")
                .setCustomer(c.getStripeCustomerId())
                .putIdempotencyKey(key)
                .build());
        }
    }
}

The instanceId=AUTO configuration in quartz.properties generates a UUID-based instance ID for each Quartz node at startup. Teams that configure instanceId=AUTO for easy horizontal scaling (no manual per-node configuration) automatically introduce node-local identifiers that differ between Node A and Node B. The billing job looks correct in a single-node setup — instanceId=AUTO always produces the same UUID within a single JVM lifetime — and only fails when a second node picks up the trigger during the GC-pause window.

# quartz.properties — instanceId=AUTO generates UUID per JVM startup
# org.quartz.scheduler.instanceId=AUTO   ← different UUID on each node
# This UUID is returned by context.getScheduler().getSchedulerInstanceId()
# Including it in the Stripe idempotency key makes the key node-specific → ch_B

# Safe configuration: fixed instanceId per node (or exclude instanceId from key entirely)
org.quartz.scheduler.instanceId=billing-node-1
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval=5000

The GC-pause trigger is real in production billing services. Java billing jobs commonly load large customer result sets into memory, perform in-memory aggregations, and call external APIs sequentially. A 10,000-customer billing job holding a List<Customer> in the heap, plus intermediate ChargeCreateParams objects and API response objects, creates GC pressure proportional to customer count. Full GC pauses of 10–30 seconds are not unusual on heap-constrained billing pods — easily exceeding the 5-second Quartz cluster check-in interval and triggering the node failure detection that causes the second node to re-acquire the trigger.

The fix for failure mode 2

The content-hash key from stable business fields resolves the node-local-value problem: sha256(customerId + ":" + billingPeriod + ":quartz-billing") produces the same 32-character string on Node A and on Node B. When both nodes call Charge.create() with the identical key, Stripe deduplicates within the 24-hour window: the second call receives the cached response from the first call.

The pre-flight PostgreSQL check provides a second defense. The UNIQUE (customer_id, billing_period) constraint on billing_records means only one of the two racing nodes can win the INSERT ... ON CONFLICT DO NOTHING. The winning node proceeds to call Stripe. The losing node (whose insert returns zero rows affected) skips the Stripe call entirely for that customer. PostgreSQL’s row-level locking ensures exactly one winner per customer per billing period — even under concurrent GC-pause-induced dual-node execution.

// Safe clustered billing job — same fix as mode 1, works for mode 2 as well
// The content-hash key and pre-flight check are node-agnostic:
// - sha256(customerId + ":" + billingPeriod + ":quartz-billing") is identical on every node
// - UNIQUE (customer_id, billing_period) in PostgreSQL serializes concurrent inserts across nodes
// - The losing node's INSERT returns 0 rows affected; it skips stripe.charges.create()
// - Stripe's 24h idempotency cache closes any remaining concurrent-call window

public class SafeClusteredBillingJob implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        // Do NOT include: context.getScheduler().getSchedulerInstanceId() (per-node),
        // InetAddress.getLocalHost().getHostName() (per-node),
        // Thread.currentThread().getId() (per-JVM per-thread),
        // UUID.randomUUID() (per-invocation),
        // System.currentTimeMillis() at job start (different on each node's run)
        List<Customer> customers = customerRepo.findDueForBilling();

        for (Customer c : customers) {
            String key = makeIdempotencyKey(c.getId(), c.getBillingPeriod());

            // PostgreSQL UNIQUE constraint on (customer_id, billing_period) ensures
            // exactly one winner between concurrent Node A and Node B inserts.
            // The loser gets rowsAffected=0 and skips Stripe — no ch_B.
            int rowsAffected = billingRepo.insertIfAbsent(
                c.getId(), c.getBillingPeriod(), key
            );
            if (rowsAffected == 0) {
                continue;
            }

            Charge charge = Charge.create(ChargeCreateParams.builder()
                .setAmount(c.getAmountCents())
                .setCurrency("usd")
                .setCustomer(c.getStripeCustomerId())
                .putIdempotencyKey(key)
                .build());

            billingRepo.updateChargeId(key, charge.getId());
        }
    }

    static String makeIdempotencyKey(String customerId, String billingPeriod) {
        return DigestUtils.sha256Hex(
            customerId + ":" + billingPeriod + ":quartz-billing"
        ).substring(0, 32);
    }
}

A Keybrake vault key scoped to the billing period adds a hard backstop above the pre-flight and Stripe-side idempotency layers. The vault policy caps total Stripe spend to expected_total × 1.10 for the billing run. If the GC pause and pre-flight race somehow both lose simultaneously — Node A wins the PostgreSQL insert for customer X, Node B wins for customer Y, and both proceed to Stripe before the other’s insert is visible (within a sub-millisecond transaction isolation window) — the vault cap stops the spend at ten percent above expected before the billing job completes. The vault log records the exact request that hit the cap, the timestamp, and the vault key, providing the forensic trail to identify the root cause without waiting for Stripe disputes.

Failure mode 3: SimpleTrigger misfire with MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_COUNT creates a new trigger instance — counter-based or fire-time-based idempotency keys produce ch_B on the re-fired trigger

Quartz’s SimpleTrigger fires a job a fixed number of times at a fixed interval: repeatCount=0 (fire once), repeatCount=N (fire N+1 times total), or repeatCount=REPEAT_INDEFINITELY. A common billing pattern uses SimpleTrigger with repeatCount=0 to fire a billing job exactly once at the start of each billing period — the trigger is created fresh each month by a management job or API call, fires once, and completes.

When a SimpleTrigger is found in WAITING state past its misfireThreshold (default 60 seconds) on scheduler restart, Quartz applies the trigger’s misfire instruction. The default misfire instruction for SimpleTrigger is MISFIRE_INSTRUCTION_SMART_POLICY, which resolves to one of three concrete instructions depending on the trigger’s configuration. For a fire-once trigger (repeatCount=0), SMART_POLICY resolves to MISFIRE_INSTRUCTION_FIRE_NOW: Quartz fires the trigger immediately, treating it as a new trigger instance with a new startTime set to the current wall clock.

For a repeating trigger (repeatCount=N, timesTriggered<repeatCount), SMART_POLICY resolves to MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_COUNT: Quartz calculates the remaining fires (original repeatCount minus timesTriggered), creates a new trigger with that remaining count, and fires immediately. Critically, the new trigger instance has a new startTime (current wall clock), a new getFireTime(), and a new getScheduledFireTime() — none of which match the original trigger instance’s times.

The billing failure arises from a developer pattern that stores a "billing run ID" or execution counter in the JobDataMap and includes it in the Stripe idempotency key. The intention is to distinguish "billing run for June 2026" from "billing run for July 2026" without including a timestamp (correctly recognizing that timestamps are fragile). The developer initializes billingRunId in the JobDataMap when the trigger is created (e.g., "billing-run-2026-06" or a sequential counter stored in a management table), reads it inside execute(), and builds the key from it. On the original fire, this works: sha256("cust_123:billing-run-2026-06") is stable across customers in the same run.

The problem: on misfire with MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_COUNT, Quartz creates a new trigger instance. The original trigger’s JobDataMap is copied into the new trigger (Quartz persists the JobDataMap alongside the trigger row). But if the developer’s initialization logic re-generates the billingRunId on scheduler start (from a DB counter that was incremented since the original trigger was created), or if the management job creates a new trigger with a newly generated ID, the JobDataMap value seen inside execute() differs from the original. Different billingRunId → different key → ch_B for customers already charged before the crash.

// UNSAFE: billingRunId from an auto-incrementing counter re-generated on misfire trigger creation
public class BillingManagementJob implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        // Management job creates a billing trigger each month.
        // Misfire handling under RESCHEDULE_NOW creates a new trigger — management job may
        // also be re-creating the trigger concurrently with the misfire handler.

        // Counter from a management table — increments each time this method runs
        long runCounter = billingRunRepo.nextRunId();  // e.g., 42 for June, but 43 if re-run

        JobDataMap data = new JobDataMap();
        data.put("billingRunId", "run-" + runCounter);  // "run-42" on original, "run-43" on re-run

        JobDetail job = JobBuilder.newJob(BillingJob.class)
            .usingJobData(data)
            .build();

        Trigger trigger = TriggerBuilder.newTrigger()
            .startNow()
            .withSchedule(SimpleScheduleBuilder.simpleSchedule())  // fire once
            .build();

        scheduler.scheduleJob(job, trigger);
    }
}

public class BillingJob implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        // On original fire: billingRunId = "run-42"
        // On misfire re-fire with a new trigger: billingRunId may be "run-43"
        String billingRunId = context.getMergedJobDataMap().getString("billingRunId");

        List<Customer> customers = customerRepo.findDueForBilling();

        for (Customer c : customers) {
            // sha256("cust_123:run-42") on original fire → ch_A
            // sha256("cust_123:run-43") on misfire re-fire → ch_B
            String key = DigestUtils.sha256Hex(
                c.getId() + ":" + billingRunId
            ).substring(0, 32);

            Charge.create(ChargeCreateParams.builder()
                .setAmount(c.getAmountCents())
                .setCurrency("usd")
                .setCustomer(c.getStripeCustomerId())
                .putIdempotencyKey(key)
                .build());

            billingRepo.save(new BillingRecord(c.getId(), billingRunId, key));
        }
    }
}

A second variant of this failure mode arises from SimpleTrigger with repeatCount > 0 in a billing pipeline that fires several times per billing period (e.g., daily partial billing during a month). MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_COUNT recalculates the remaining fires from the QRTZ_TRIGGERS.TIMES_TRIGGERED column. If a billing job for day 3 of 30 fails mid-execution (day 3 is marked as ACQUIRED but not COMPLETE), and the scheduler restarts and applies the misfire instruction with timesTriggered=2 (days 1 and 2 completed), the remaining count is 28. The new trigger fires immediately (day 3 re-fire) and then continues for days 4 through 30. Developers who derive the Stripe idempotency key from timesTriggered or the trigger’s fire sequence number (which restarts from 1 on the new trigger instance) get a different key for the day 3 re-fire than for the original day 3 fire — and Stripe creates ch_B for all customers processed during the original day 3 run before the crash.

// UNSAFE: using the trigger's fire sequence number (which resets on new trigger instance)
public class DailyBillingJob implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        // TIMES_TRIGGERED for the *current* trigger instance — resets to 0 on new trigger
        // from MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_COUNT
        int dayOfMonth = (int) context.getTrigger().getTimesTriggered() + 1;

        List<Customer> customers = customerRepo.findDueForDailyBilling(dayOfMonth);

        for (Customer c : customers) {
            // Day 3, original trigger: sha256("cust_123:2026-06:day-3") → ch_A
            // Day 3, new trigger from misfire: timesTriggered=0+1=1 → sha256("cust_123:2026-06:day-1") → ch_B
            // (or day-3 on new trigger but with new trigger's sequential counter)
            String key = DigestUtils.sha256Hex(
                c.getId() + ":2026-06:day-" + dayOfMonth
            ).substring(0, 32);

            Charge.create(ChargeCreateParams.builder()
                .setAmount(c.getAmountCents())
                .setCurrency("usd")
                .setCustomer(c.getStripeCustomerId())
                .putIdempotencyKey(key)
                .build());
        }
    }
}

The fix for failure mode 3

The content-hash key from stable business fields resolves both SimpleTrigger variants. For the monthly fire-once trigger: the key is sha256(customerId + ":" + billingPeriod + ":quartz-billing") where billingPeriod is the calendar billing period (e.g., "2026-06"), not a counter from a management table or any value from JobDataMap that could be re-generated. For the daily partial-billing trigger: the key is sha256(customerId + ":" + billingPeriod + ":" + calendarDate + ":quartz-billing") where calendarDate is the wall-clock date (e.g., "2026-06-03") computed from LocalDate.now() at job execution time. The calendar date is stable across misfire re-fires for the same day (a day-3 misfire that re-fires on day 3 uses the same 2026-06-03 date). It is not stable if the misfire fires the next day, but a day-3 misfire that fires on day 4 is correctly a day-4 billing run, not a day-3 retry.

The pre-flight check is especially important for the SimpleTrigger misfire case because Quartz provides no application-level signal that a job execution is a misfire re-fire versus a normal fire. The execute() method receives the same JobExecutionContext in both cases; only the trigger times differ. The pre-flight insert on (customer_id, billing_period) detects already-charged customers regardless of whether this is a normal fire, a misfire re-fire, or a concurrent cluster execution — the idempotency guarantee is in the database constraint, not in the Quartz trigger semantics.

// Safe: billing period from calendar date, not from JobDataMap counter or trigger sequence
// Works for fire-once SimpleTrigger AND repeating daily SimpleTrigger misfire re-fires

public class SafeSimpleBillingJob implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        // Billing period from calendar — stable regardless of trigger instance, fire sequence,
        // or RESCHEDULE_NOW_WITH_REMAINING_COUNT creating a new trigger.
        // For monthly billing:
        String billingPeriod = YearMonth.now().toString(); // e.g., "2026-06"

        // For daily partial billing: include the calendar date, not timesTriggered
        // String billingPeriod = LocalDate.now().toString(); // e.g., "2026-06-03"

        List<Customer> customers = customerRepo.findDueForBilling(billingPeriod);

        for (Customer c : customers) {
            String key = makeIdempotencyKey(c.getId(), billingPeriod);

            int rowsAffected = billingRepo.insertIfAbsent(
                c.getId(), billingPeriod, key
            );
            if (rowsAffected == 0) {
                continue; // already charged this period — misfire re-fire caught
            }

            Charge charge = Charge.create(ChargeCreateParams.builder()
                .setAmount(c.getAmountCents())
                .setCurrency("usd")
                .setCustomer(c.getStripeCustomerId())
                .putIdempotencyKey(key)
                .build());

            billingRepo.updateChargeId(key, charge.getId());
        }
    }

    static String makeIdempotencyKey(String customerId, String billingPeriod) {
        // Must NOT include: timesTriggered (resets on new trigger from RESCHEDULE_NOW),
        // JobDataMap counter (may be re-generated by management job),
        // getFireTime() or getScheduledFireTime() (both change on new trigger instance),
        // trigger.getKey().getName() (fragile on trigger rename/delete-recreate),
        // UUID from JobDataMap initialized at trigger creation (regenerated on new trigger).
        return DigestUtils.sha256Hex(
            customerId + ":" + billingPeriod + ":quartz-billing"
        ).substring(0, 32);
    }
}

The vault key policy for a SimpleTrigger misfire scenario should be scoped to the billing period, not the trigger instance. Set expires_at to the end of the billing period (not the trigger’s endTime, which may be null for fire-once triggers). Set the daily cap to expected_total × 1.10. If a billing management job accidentally creates two trigger instances for the same billing period (a common operational mistake when a scheduler is restarted with incomplete cleanup), the vault cap stops the spend after the first full run completes. The second trigger instance hits the cap on the first Stripe call attempt and the proxy returns a 402 — the billing job fails cleanly with a loggable error, rather than double-charging all customers silently.

The two-layer pattern summarized

Gap analysis: additional Quartz billing patterns to audit

1. @PersistJobDataAfterExecution and @DisallowConcurrentExecution

Quartz provides two job annotations that affect billing behavior. @DisallowConcurrentExecution prevents two instances of the same job from running simultaneously on the same scheduler (single-node only — it does not prevent a second cluster node from acquiring the trigger in the GC-pause failure mode above, because the lock check happens at trigger acquisition time, before the job is running). @PersistJobDataAfterExecution causes Quartz to write the job’s JobDataMap state back to the QRTZ_JOB_DETAILS table after each execution completes. Teams that use these annotations to track "which customers were charged in the last run" as a dedup mechanism inside the JobDataMap face two issues: the JobDataMap is not written back if the job crashes (the table row retains the pre-execution state), and the JobDataMap is not shared between cluster nodes (each node reads from the database at job execution start, but concurrent writes from two nodes would race). The pre-flight PostgreSQL table is the correct dedup store — not the Quartz JobDataMap.

2. StatefulJob (deprecated) and CronTrigger overlap prevention

The deprecated StatefulJob interface in Quartz 1.x combined @PersistJobDataAfterExecution and @DisallowConcurrentExecution semantics and was commonly used in billing pipelines. Teams migrating from Quartz 1.x to Quartz 2.x sometimes miss the transition from StatefulJob to the two annotations, losing the @DisallowConcurrentExecution protection. A CronTrigger that fires every hour for a billing job that takes 75 minutes will have two instances running simultaneously after the second trigger fires. If the second instance does not see the first instance’s in-progress charges (no pre-flight check), it charges all customers again. The content-hash key and pre-flight check are the correct protection; @DisallowConcurrentExecution is a useful defense-in-depth for single-node setups but must not be treated as the primary idempotency mechanism.

3. Quartz’s Trigger.TriggerState.ERROR and manual refire via scheduler.triggerJob()

When a Job.execute() throws a JobExecutionException with setRefireImmediately(true), Quartz immediately fires the job again without going through the normal misfire handling path. getFireTime() and getScheduledFireTime() get new values. Developers who use setRefireImmediately(true) on Stripe-related exceptions (e.g., on StripeException from a 429 rate-limit response) will re-fire with new timestamp values, producing a different key and ch_B for customers already charged before the rate limit was hit. Use setRefireImmediately(false) for billing jobs and implement retry logic using a stable key outside the job execution (a reconciliation job, or the pre-flight row’s status='pending' detection). Similarly, scheduler.triggerJob(jobKey) called manually by an operator to re-run a failed billing job starts a new job execution with current wall-clock getFireTime(); the pre-flight check is the only guard in this case, since a manually triggered job carries no trigger-level misfire context at all.

4. JobDataMap key derivation from trigger.getKey().getName()

Some teams use the Quartz trigger name (from context.getTrigger().getKey().getName()) as a component of the Stripe idempotency key, treating the trigger name as a stable "billing run identifier." Trigger names are stable for the lifetime of the trigger row in QRTZ_TRIGGERS. They change when the trigger is deleted and recreated (a common pattern in management APIs that "reset" a billing schedule) or when the trigger is replaced by the misfire handler creating a new trigger instance under MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_COUNT. The content-hash key from business fields requires no reference to Quartz trigger metadata and is immune to trigger name changes.

5. Database-stored Quartz vs. in-memory RAMJobStore

Quartz’s in-memory RAMJobStore (the default when no jobStore is configured) does not persist trigger state across JVM restarts. A billing pod using RAMJobStore that crashes during a billing run loses all trigger state when it restarts. The triggers are not in ACQUIRED state when the new JVM starts; Quartz starts fresh with an empty job store. There is no misfire handling because there are no persisted triggers. The billing run simply does not resume. Teams using RAMJobStore for billing crons rely on the cron expression re-firing at the next scheduled time, which may be a month later for a monthly billing job. Using JDBCJobStore with a relational database is required for misfire handling and clustering — but introduces all three failure modes described in this post. The pre-flight check pattern applies regardless of which JobStore is in use: with RAMJobStore, it catches crashes within a single JVM lifetime; with JDBCJobStore, it also catches misfire re-fires and clustered concurrent execution.

FAQ

If I set MISFIRE_INSTRUCTION_DO_NOTHING on my CronTrigger, does that prevent duplicate charges?

MISFIRE_INSTRUCTION_DO_NOTHING causes Quartz to skip all missed fire times and wait for the next scheduled occurrence. A monthly billing job that misses its 2026-06-01T00:00:00Z fire time will not re-fire until 2026-07-01T00:00:00Z. This prevents the misfire re-fire duplicate charge scenario from failure mode 1, but it also means missed billing runs are never retried — customers who should have been billed on June 1st are not billed until July 1st, or not at all if the management job creates a fresh trigger each month. For a billing pipeline, MISFIRE_INSTRUCTION_DO_NOTHING is the wrong instruction. The correct approach is to use a standard misfire instruction (or the default SMART_POLICY) to ensure the re-fire happens, and to use the content-hash key and pre-flight check to ensure the re-fire does not create duplicate charges.

Can I use Quartz’s SchedulerContext (global key-value store) to share state between cluster nodes and avoid duplicate charges?

No. The SchedulerContext (accessed via context.getScheduler().getContext()) is an in-memory map scoped to a single JVM. It is not shared across cluster nodes and not persisted to the database. State stored in SchedulerContext is lost on JVM restart, invisible to other cluster nodes, and not a reliable dedup mechanism. Use PostgreSQL (or another shared external database) for cross-node state sharing. The pre-flight table pattern — billing_records with a UNIQUE (customer_id, billing_period) constraint — is the correct cross-node, cross-restart dedup mechanism for Quartz billing jobs.

Does @DisallowConcurrentExecution prevent the GC-pause cluster race in failure mode 2?

No. @DisallowConcurrentExecution prevents two instances of the same job from running simultaneously on a single Quartz scheduler node. In a clustered setup, the check is: Quartz looks at the QRTZ_TRIGGERS.TRIGGER_STATE column — if the trigger is ACQUIRED (another node is running it), the current node skips it. But in the GC-pause failure mode, Node A’s database connection drops during the GC pause, rolling back the transaction that set the trigger to ACQUIRED. The trigger reverts to WAITING. Node B reads the trigger as WAITING (not ACQUIRED) and fires it. @DisallowConcurrentExecution does not prevent this because the trigger state in the database shows WAITING — the previous ACQUIRED state was rolled back. The PostgreSQL pre-flight constraint is the correct defense; it checks whether the billing record exists (was ever inserted), not whether the trigger is currently ACQUIRED.

How should I scope vault keys for a Quartz billing job that runs multiple times per billing period?

Issue one vault key per billing period per vendor, regardless of how many trigger fires are in that period. A daily billing job that fires 30 times in June 2026 should use one vault key scoped to expires_at: 2026-07-01T00:00:00Z and daily_usd_cap: daily_expected_total × 1.10. The vault proxy aggregates spend across all 30 trigger fires. If trigger fires 1 through 29 exhaust the monthly expected total (because a misfire fired the job twice on one day), the vault cap stops the 30th trigger fire ’s Stripe call before it charges any customers — signaling that the daily cap was exceeded and an investigation is needed. Without the vault cap, a misfire-doubled day goes undetected until Stripe’s monthly statement arrives.

My billing job calls setRefireImmediately(true) on Stripe timeout exceptions. What should I change?

Remove setRefireImmediately(true) from your Stripe exception handler. A Stripe SDK timeout (APIConnectionException with a socket timeout cause) does not mean the charge was not created — Stripe may have processed the charge server-side before the HTTP response was dropped by a network device. Re-firing immediately with setRefireImmediately(true) generates new Quartz fire times, defeating any timestamp-based idempotency key. The correct pattern: catch the timeout, log it, and throw new JobExecutionException(e, false) (do not refire immediately). The pre-flight row remains in status='pending' for this customer. A reconciliation job running every 5 minutes finds pending rows older than 5 minutes and re-issues Charge.create() with the same stable content-hash key. If the original charge was created server-side, Stripe returns the cached ch_A object (within 24h) or the pre-flight constraint returns zero rows affected (beyond 24h, indicating a billing_records row already exists from a prior successful run). Either way, no ch_B is created.

Related: Celery Beat and Stripe integration — APScheduler and Stripe integration — Kubernetes CronJob and Stripe integration — Apache Kafka and Stripe integration — Spring Cloud Stream and Stripe integration