BullMQ Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

BullMQ is the most widely used Node.js job queue for distributed background workloads, built on Redis and designed for high-throughput processing with configurable retry, concurrency, and scheduling. When billing tasks run inside BullMQ workers, three framework-specific failure modes emerge that have nothing to do with your Stripe integration itself — they come from how BullMQ retries jobs, how its worker concurrency model shares state, and how its repeat scheduling interacts with manually triggered runs.

This post covers three BullMQ-specific failure modes — job retry re-execution, worker concurrency key sharing, and repeat-plus-manual-trigger race conditions — each of which causes a second stripe.charges.create() call on an already-billed customer, and the two-layer governance pattern that closes all three: content-hash idempotency keys plus per-batch vault keys via a spend-cap proxy, without restructuring your queue topology.

Failure mode 1: Job retry re-executes the processor function from line 1 after stripe.charges.create() has already succeeded

BullMQ jobs support automatic retry via the attempts option. When a processor function throws an unhandled exception, BullMQ moves the job to a backoff state and re-executes the entire processor function from line 1 on the next attempt. There is no mid-function resume capability — each retry is a full re-invocation of the processor with the original job data.

The failure window is between stripe.charges.create() returning successfully and the processor completing without error. Consider a billing processor that charges a customer, then writes the charge ID to a PostgreSQL table. If the database connection times out after the Stripe call succeeds — or any subsequent await in the processor throws — BullMQ catches the exception, applies the configured backoff, and re-executes the processor. The Stripe call fires again with fresh arguments. Without an idempotency key, Stripe creates ch_B. Both charge objects are valid; the customer has been charged twice; and the BullMQ dashboard shows the job succeeded on attempt 2 with no indication of the duplicate charge.

// workers/billing.ts — UNSAFE: no idempotency key
import { Worker, Job } from 'bullmq';
import Stripe from 'stripe';
import { pool } from '../db';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!); // shared, unrestricted

const billingWorker = new Worker('billing', async (job: Job) => {
  const { customerId, amountCents, billingPeriod } = job.data;

  // If this succeeds but the DB write below throws → processor retried → ch_B
  const charge = await stripe.charges.create({
    amount: amountCents,
    currency: 'usd',
    customer: customerId,
    description: `Subscription ${billingPeriod}`,
  });

  // DB connection timeout here → BullMQ retries entire processor from line 1
  await pool.query(
    'INSERT INTO charges (charge_id, customer_id, billing_period) VALUES ($1, $2, $3)',
    [charge.id, customerId, billingPeriod]
  );
}, {
  connection,
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 2000 },
  },
});

The fix is a content-hash idempotency key derived from the job data fields that are stable across all retry attempts. BullMQ passes the same job.data object on every retry — customerId, amountCents, and billingPeriod are immutable for the lifetime of the job — so a key derived from these inputs is identical on every attempt. Stripe returns the original ch_A rather than creating ch_B. Permanent Stripe errors (card declined, invalid customer) should be converted to UnrecoverableError so BullMQ does not schedule further retry attempts for a charge that will never succeed.

// workers/billing.ts — SAFE: stable idempotency key + UnrecoverableError for permanent failures
import { Worker, Job, UnrecoverableError } from 'bullmq';
import Stripe from 'stripe';
import crypto from 'crypto';
import { pool } from '../db';

const billingWorker = new Worker('billing', async (job: Job) => {
  const { customerId, amountCents, billingPeriod, vaultKey } = job.data;

  const idempotencyKey = crypto
    .createHash('sha256')
    .update(`${customerId}:${amountCents}:${billingPeriod}:bullmq-billing`)
    .digest('hex')
    .slice(0, 32);

  // Use vault key from job data — issued once before batch dispatch, not process.env
  const stripe = new Stripe(vaultKey, {
    host: 'proxy.keybrake.com',
    port: 443,
    protocol: 'https',
    basePath: '/stripe',
  });

  let chargeId: string | null = null;
  let status: string;

  try {
    const charge = await stripe.charges.create(
      { amount: amountCents, currency: 'usd', customer: customerId,
        description: `Subscription ${billingPeriod}` },
      { idempotencyKey }
    );
    chargeId = charge.id;
    status = 'charged';
  } catch (err) {
    if (err instanceof Stripe.errors.StripeCardError ||
        err instanceof Stripe.errors.StripeInvalidRequestError) {
      // Permanent — record and skip all remaining retry attempts
      await pool.query(
        'INSERT INTO charges (customer_id, billing_period, status) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING',
        [customerId, billingPeriod, 'card_declined']
      );
      throw new UnrecoverableError(`Non-retriable Stripe error for ${customerId}: ${(err as Error).message}`);
    }
    throw err; // Transient (connection, timeout) → BullMQ retries; idempotency key absorbs duplicate
  }

  // If this DB write throws, retry re-calls stripe.charges.create() with the same idempotency
  // key — Stripe returns ch_A, not ch_B
  await pool.query(
    'INSERT INTO charges (charge_id, customer_id, billing_period, status) VALUES ($1, $2, $3, $4) ON CONFLICT DO NOTHING',
    [chargeId, customerId, billingPeriod, status]
  );
}, { connection });

Failure mode 2: Worker concurrency dispatches N simultaneous job executions that all share one unrestricted STRIPE_SECRET_KEY

BullMQ workers support a concurrency option that controls how many jobs execute simultaneously in the same worker process. A concurrency of 10 means 10 processor function invocations run as concurrent async tasks in the same Node.js event loop. All 10 read process.env.STRIPE_SECRET_KEY from the same process environment — there is no per-job key isolation at the worker level.

The blast radius of a data error scales with the concurrency setting. If the amountCents value pushed into each job has a unit bug — 299900 (dollars) instead of 2999 (cents) — all 10 concurrent billing jobs charge their respective customers the wrong amount before any single processor completes and surfaces an error. With a monthly billing batch of 500 customers dispatched as 500 individual jobs and concurrency: 10, 10 incorrect charges fire simultaneously in the first wave. Without a spend cap on the key, there is no circuit breaker to halt the remaining 490 jobs once the first 10 complete with errors.

// UNSAFE: all concurrent jobs share one unrestricted STRIPE_SECRET_KEY
import { Worker } from 'bullmq';
import Stripe from 'stripe';

// Module-level Stripe client — shared across all concurrency: 10 jobs
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);

const billingWorker = new Worker('billing', async (job) => {
  const { customerId, amountCents, billingPeriod } = job.data;

  // Each of the 10 concurrent jobs calls this with the same unrestricted key
  // Data error in amountCents hits 10 customers per wave with no circuit breaker
  const charge = await stripe.charges.create({
    amount: amountCents,          // unit bug: 299900 instead of 2999
    currency: 'usd',
    customer: customerId,
  });

  return { chargeId: charge.id };
}, {
  connection,
  concurrency: 10,              // 10 concurrent processors, all share STRIPE_SECRET_KEY
});

The fix is to issue a per-batch vault key before dispatching jobs to the queue, cap it at the total expected billing amount for the period, and pass it as a field in each job's data. The vault key is scoped to POST /v1/charges and carries a daily spend cap equal to the sum of all customer amounts multiplied by a 1.10 buffer. Even if concurrency: 10 fires simultaneously, the proxy's spend cap is enforced across all concurrent calls sharing that key — the circuit breaker is at the proxy layer, not the job layer.

// dispatch.ts — SAFE: vault key issued once before batch, passed in each job's data
import { Queue } from 'bullmq';
import crypto from 'crypto';

interface Customer { id: string; amountCents: number; }

async function dispatchBillingBatch(billingPeriod: string, customers: Customer[]) {
  const totalCents = customers.reduce((sum, c) => sum + c.amountCents, 0);
  const maxSpendUsd = parseFloat(((totalCents / 100) * 1.10).toFixed(2));

  // Issue one vault key covering the entire batch
  const resp = await fetch('https://proxy.keybrake.com/keys', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.KEYBRAKE_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      vendor: 'stripe',
      allowed_endpoints: ['POST /v1/charges'],
      daily_usd_cap: maxSpendUsd,
      expires_in_seconds: 7200,
      label: `bullmq-billing-${billingPeriod}`,
    }),
  });
  const { vault_key: vaultKey } = await resp.json() as { vault_key: string };

  const billingQueue = new Queue('billing', { connection });

  for (const { id: customerId, amountCents } of customers) {
    // Deterministic jobId: BullMQ ignores duplicate adds with the same jobId
    const jobId = crypto
      .createHash('sha256')
      .update(`${customerId}:${amountCents}:${billingPeriod}`)
      .digest('hex')
      .slice(0, 32);

    await billingQueue.add('charge', {
      customerId,
      amountCents,
      billingPeriod,
      vaultKey,           // per-batch vault key, not process.env.STRIPE_SECRET_KEY
    }, {
      jobId,              // dedup: second add() for same customer+period is ignored
      attempts: 3,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: false,   // keep completed jobs so jobId dedup persists
      removeOnFail: false,       // keep failed jobs for inspection
    });
  }
}

Failure mode 3: repeat cron job and a manual queue.add() call create two independent billing executions for the same period

BullMQ's repeat option schedules a job on a cron pattern — a natural fit for monthly billing. When the cron fires, BullMQ adds a new job using an internally generated repeatJobKey-derived ID. If an operator, monitoring script, or webhook handler also calls queue.add() for the same billing period without specifying an identical jobId, BullMQ creates a second independent job with a different auto-generated UUID. Both jobs are valid, both enter the waiting state, and both execute the billing processor independently — there is no built-in deduplication between the two invocation paths.

This failure mode is most dangerous in at-least-once delivery scenarios. A webhook from your subscription system fires at midnight, your handler calls queue.add('billing', { billingPeriod: '2026-07' }), and the BullMQ repeat cron fires at the same time. Both paths reach the same billing processor with the same billingPeriod but different job IDs. Both execute stripe.charges.create() for every customer in the period. Without an idempotency key, ch_B is created alongside ch_A for every customer in the cohort.

// scheduler.ts — UNSAFE: repeat cron + manual add() create two independent jobs
import { Queue } from 'bullmq';

const billingQueue = new Queue('billing', { connection });

// Cron: fire monthly billing at midnight on the 1st
await billingQueue.add('monthly-billing',
  { billingPeriod: '2026-07' },
  { repeat: { pattern: '0 0 1 * *' } }
  // BullMQ assigns an internal repeatJobKey-derived ID to this job
);

// Meanwhile: operator notices billing not yet visible in Stripe at 00:05 UTC
// (it's actually in-flight — just slow) and manually triggers another run
await billingQueue.add('monthly-billing',
  { billingPeriod: '2026-07' }
  // No jobId → BullMQ generates a UUID → new, independent job, not a dedup of the repeat job
);

// Result: two Worker executions run the billing processor for the same billingPeriod
// Each independently calls stripe.charges.create() for every customer in the cohort
// Without idempotency keys: N customers × 2 charges each

Two layers close this failure mode. The first is a deterministic jobId computed from the billing period for all non-repeat additions — when a job with that jobId already exists in any state (waiting, active, completed), BullMQ silently ignores the duplicate add() call. The second is the idempotency key in the processor itself, which handles the race where two jobs enter the active state before either has written its result. For repeat jobs, set removeOnComplete: false so completed job records persist in Redis and the jobId dedup remains effective across cron ticks for recent periods.

// scheduler.ts — SAFE: deterministic jobId deduplicates all add() paths
import { Queue } from 'bullmq';
import crypto from 'crypto';

const billingQueue = new Queue('billing', { connection });

function billingJobId(billingPeriod: string): string {
  return crypto
    .createHash('sha256')
    .update(`monthly-billing:${billingPeriod}`)
    .digest('hex')
    .slice(0, 32);
}

// Operator manual trigger — uses deterministic jobId
async function triggerBillingForPeriod(billingPeriod: string) {
  const job = await billingQueue.add('monthly-billing',
    { billingPeriod },
    {
      jobId: billingJobId(billingPeriod),
      attempts: 3,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: false,  // dedup persists across cron ticks
      removeOnFail: false,
    }
  );
  // BullMQ returns null if a job with this jobId already exists — not an error
  if (!job) {
    console.log(`Billing job for ${billingPeriod} already queued or completed — skipped`);
  }
  return job;
}

// For repeat jobs, use a jobId template that resolves to the same value
// on every tick — prevents the repeat job from bypassing dedup on re-fire
await billingQueue.add('monthly-billing',
  { billingPeriod: getCurrentBillingPeriod() },
  {
    jobId: billingJobId(getCurrentBillingPeriod()),
    repeat: { pattern: '0 0 1 * *' },
    removeOnComplete: false,
    removeOnFail: false,
  }
);

// Processor: idempotency key is the second layer of protection
// — handles the race where two jobs enter active before either writes a result
// (same idempotency key from identical inputs → Stripe returns ch_A on both calls)

Approach comparison

Gap analysis

1. removeOnComplete: { count: N } removes completed job records, breaking jobId-based deduplication for future billing periods

BullMQ's removeOnComplete: { count: 500 } option removes the oldest completed job records once the completed set exceeds 500 entries. Once a billing job's record is pruned from Redis, a future queue.add() call with the same jobId is treated as a brand-new job — the deduplication guarantee only applies while the completed record exists. For billing jobs that run once per customer per month and have overlapping billingPeriod values across months, this is particularly dangerous: after pruning, a duplicate webhook or re-trigger for a period that completed two months ago will execute the billing processor again. Use removeOnComplete: false for billing queues, or rely exclusively on the idempotency key in the processor rather than jobId dedup for long-window protection.

2. job.retry() via the BullMQ API or dashboard creates a fresh processor execution that bypasses UnrecoverableError classification

Failed jobs in BullMQ can be manually retried via job.retry(), the Bull Board dashboard, or the BullMQ REST API. A manual retry re-queues the job and re-executes the processor from line 1, regardless of whether the original failure was thrown as an UnrecoverableError. If an operator retries a billing job that failed because of a card decline — one you correctly classified as UnrecoverableError — the processor runs again and may call stripe.charges.create() a second time. The idempotency key protects against the Stripe duplicate, but the processor will again throw UnrecoverableError after recording the decline status. Add a pre-flight database check at the top of the processor: if a charge record already exists for (customerId, billingPeriod), return immediately without calling Stripe, regardless of how the processor was triggered.

3. BullMQ FlowProducer child job retries re-execute billing steps that share a parent job's vault key scope

BullMQ's FlowProducer allows parent-child job hierarchies where a parent job waits on the completion of child jobs. In a billing flow where child jobs handle individual customer charges and the parent aggregates results, each child job retry is an independent processor re-execution. If all child jobs are passed the same vault key (issued at the parent level), the per-batch spend cap is shared across all retries of all children. A scenario where 10 out of 500 child jobs fail and retry can push total Stripe spend to 510 × amountCents — breaching the 500 × 1.10 buffer if retries are not accounted for in the cap. Size the vault key cap to accommodate: total_customers × amountCents × 1.10 × max_attempts, or issue per-child vault keys that expire after a single successful charge.

4. BullMQ sandboxed processors in a separate worker file do not share module-level state — but they still share the same process.env

BullMQ supports running processors in sandboxed child processes by pointing the Worker constructor at a file path rather than an inline function. Sandboxed workers run in isolated Node.js processes, so module-level Stripe client instances are not shared across concurrent jobs (each child process has its own). However, all child processes still inherit process.env from the parent — including STRIPE_SECRET_KEY. If a sandboxed processor reads process.env.STRIPE_SECRET_KEY directly rather than using the vault key from job.data, the concurrency isolation benefit of sandboxing does not extend to key isolation. Always route through the vault key from job.data, whether running inline or sandboxed.

Enforcement tests

// test/billing.test.ts
import crypto from 'crypto';
import { describe, it, expect, vi, beforeEach } from 'vitest';

function makeIdempotencyKey(customerId: string, amountCents: number, billingPeriod: string): string {
  return crypto
    .createHash('sha256')
    .update(`${customerId}:${amountCents}:${billingPeriod}:bullmq-billing`)
    .digest('hex')
    .slice(0, 32);
}

function makeBillingJobId(customerId: string, amountCents: number, billingPeriod: string): string {
  return crypto
    .createHash('sha256')
    .update(`${customerId}:${amountCents}:${billingPeriod}`)
    .digest('hex')
    .slice(0, 32);
}

describe('BullMQ billing idempotency', () => {
  it('produces the same idempotency key across all processor retries', () => {
    const key1 = makeIdempotencyKey('cus_ABC', 2999, '2026-07');
    const key2 = makeIdempotencyKey('cus_ABC', 2999, '2026-07');
    expect(key1).toBe(key2);
    expect(key1).toHaveLength(32);
  });

  it('produces unique idempotency keys per customer', () => {
    const keyAbc = makeIdempotencyKey('cus_ABC', 2999, '2026-07');
    const keyDef = makeIdempotencyKey('cus_DEF', 2999, '2026-07');
    expect(keyAbc).not.toBe(keyDef);
  });

  it('throws UnrecoverableError on card decline and does not re-attempt Stripe', async () => {
    const { UnrecoverableError } = await import('bullmq');
    const { processBillingJob } = await import('../workers/billing');

    const mockStripe = { charges: { create: vi.fn().mockRejectedValueOnce(
      Object.assign(new Error('Card declined'), { type: 'StripeCardError' })
    )}};
    const mockDb = { query: vi.fn().mockResolvedValue({ rows: [] }) };

    await expect(
      processBillingJob(
        { data: { customerId: 'cus_DECLINED', amountCents: 2999, billingPeriod: '2026-07', vaultKey: 'vk_test' } } as any,
        mockStripe as any, mockDb as any
      )
    ).rejects.toBeInstanceOf(UnrecoverableError);

    expect(mockStripe.charges.create).toHaveBeenCalledTimes(1);
  });

  it('deterministic jobId is identical for same customer+period inputs', () => {
    const id1 = makeBillingJobId('cus_ABC', 2999, '2026-07');
    const id2 = makeBillingJobId('cus_ABC', 2999, '2026-07');
    expect(id1).toBe(id2);
    expect(id1).toHaveLength(32);
  });

  it('vault key cap includes 10% buffer over total batch amount', () => {
    const customers = [
      { amountCents: 2999 },
      { amountCents: 4999 },
      { amountCents: 9999 },
    ];
    const totalCents = customers.reduce((s, c) => s + c.amountCents, 0);
    const expectedCap = parseFloat(((totalCents / 100) * 1.10).toFixed(2));
    expect(expectedCap).toBeGreaterThan(totalCents / 100);
    expect(expectedCap).toBeLessThan((totalCents / 100) * 1.20);
  });
});

Frequently asked questions

Can I use job.id as the Stripe idempotency key?

No. BullMQ's job.id is a UUID generated at job creation time and is stable across all retry attempts of the same job — it will not change on retries, which makes it superficially attractive. However, it is not stable across the two independent job creation scenarios described in Failure mode 3: when a repeat job and a manual queue.add() call each create a new job, they produce different job.id values for the same billing intent. If both reach the processor without the jobId dedup intercepting them, two different idempotency keys are sent to Stripe and both charges succeed. Use a content-hash key derived from (customerId, amountCents, billingPeriod) — these values are identical regardless of which code path created the job.

Is attempts: 3 safe to configure on billing jobs if I add idempotency keys?

Yes — with stable idempotency keys, the Stripe call itself is safe to retry. When the processor re-executes on attempt 2 or 3 and calls stripe.charges.create() with the same key, Stripe returns the original charge object. The retry risk is in the surrounding code: database writes without ON CONFLICT DO NOTHING will raise a uniqueness constraint on the second execution. Make every operation in the processor idempotent: ON CONFLICT DO NOTHING for DB writes, UnrecoverableError for permanent Stripe errors, and a check that the vault key issued at batch dispatch time has not expired before calling Stripe on attempt 3.

How do I charge the same customer twice in one billing period legitimately?

Include a chargeReason or invoiceId in the idempotency key derivation: sha256(customerId:amountCents:billingPeriod:chargeReason:bullmq-billing).slice(0,32). A subscription renewal and a mid-month usage overage for the same customer in the same period produce different keys when chargeReason differs. Also scope the deterministic jobId to include chargeReason so BullMQ does not deduplicate the two legitimate jobs. Do not use a timestamp or random UUID — these produce an unstable key that defeats idempotency on retries.

What happens if the vault key expires before the batch finishes?

The Keybrake proxy returns a 401 Unauthorized for calls made with an expired vault key. The BullMQ processor will throw on the failed stripe.charges.create() call, and BullMQ will retry the job according to the attempts setting. The retry will also fail with a 401 because the same expired vault key is in job.data. Set expires_in_seconds conservatively: for a batch of 500 customers with concurrency: 10 and an average 300ms per Stripe call, the batch takes approximately 15 seconds — a 7200s TTL is more than sufficient. If batches are very large (thousands of customers), issue the vault key with an extended TTL or add a fallback in the dispatcher that re-issues the key when the batch is split across multiple queue additions over time.

Does Keybrake work with BullMQ's FlowProducer parent-child topology?

Yes. The recommended pattern is to issue the vault key in the parent job's processor before it produces its child jobs, and pass the vault key as a field in each child job's data via FlowProducer.add(). All child jobs share one vault key whose spend cap covers the full cohort. The proxy enforces the cap across all concurrent child processor calls. Parent-job retries will re-issue a new vault key (use the pre-flight database check to avoid re-charging customers whose child jobs already completed successfully before the parent failed).

Does Keybrake work when BullMQ workers run across multiple machines?

Yes. BullMQ is designed for distributed deployments where multiple worker processes across different machines all connect to the same Redis instance. Each worker makes HTTP calls directly to proxy.keybrake.com using the vault key from job.data. The proxy is a network service, not a local library, so it works regardless of how workers are distributed. The per-batch vault key spend cap is enforced at the proxy across all concurrent calls from all worker machines combined — if ten machines each process 50 of the 500 batch jobs, the proxy correctly aggregates all 500 calls and rejects any call that would exceed the cap. The label on the vault key (bullmq-billing-2026-07) identifies which batch issued which calls in the Keybrake audit log.