Hazelcast and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Hazelcast — the Java-native in-memory data grid deployed in financial services, telecom, and increasingly in AI agent pipelines that require distributed, low-latency state — introduces three Stripe billing failure modes that differ from those in traditional message queues. Hazelcast is not a broker: it has no central server and no queue-per-topic topology. Its billing-relevant primitives are IMap (distributed partitioned map), Reliable Topic (durable pub/sub backed by a cluster-wide RingBuffer), and the CP Subsystem (linearizable distributed primitives built on the Raft consensus protocol). Each of these primitives has a re-execution or re-delivery behavior that is invisible in normal operation but that surfaces precisely when billing is in flight at the moment a JVM crashes, a GC pause fires, or a maintenance window runs long.

The first failure mode targets IMap.executeOnKey(). When the Hazelcast member that owns the primary partition for a billing key fails while an EntryProcessor is executing, Hazelcast promotes the backup partition to primary and the Hazelcast client automatically retries the executeOnKey() call on the new primary. If the EntryProcessor.process() method generates a UUID.randomUUID() to build the Stripe idempotency key, the re-execution on the backup member produces a different UUID, and Stripe creates ch_B for a customer whose ch_A was created in the original execution before the primary failed. The second failure mode targets Reliable Topic subscribers that checkpoint their RingBuffer sequence position in IMap with a MaxIdle eviction policy. When a maintenance window runs longer than the MaxIdle TTL, the checkpoint entry is evicted. On restart, the subscriber reads null from IMap, falls back to OLDEST initial position, and replays the entire 7-day RingBuffer retention window — including billing events that were already processed and charged during previous runs. The third failure mode targets CP Subsystem FencedLock. When the lock holder’s JVM enters a GC pause long enough for the CP session heartbeat to miss its deadline, the CP Subsystem releases the lock and another member acquires it with a new fencing token. If the developer included the previous fencing token in the Stripe idempotency key (a reasonable attempt to “version” the billing run), the second member’s different token produces a different key and Stripe creates ch_B.

This post covers all three failure modes with Java code, content-hash idempotency keys that exclude all Hazelcast execution and session metadata (EntryProcessor execution UUID, RingBuffer sequence, CP fencing token, member UUID, CP session ID, partition migration timestamp), pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, and per-billing-period vault keys via a spend-cap proxy capped at expected_total × 1.10 — the three-layer governance pattern that closes all three failure modes without changing Hazelcast cluster topology, partition count, or CP Subsystem session configuration.

Failure mode 1: IMap.executeOnKey() re-executes the EntryProcessor on the backup partition after primary member failure — UUID.randomUUID() inside process() generates a new value on re-execution, Stripe creates ch_B

Hazelcast’s IMap.executeOnKey(Object key, EntryProcessor processor) sends the serialized EntryProcessor to the member that owns the primary partition for the given key. The process() method runs in-place on that member, with direct access to the partition’s local data. This design enables atomic read-modify-write operations without round-trip serialization of the entire map value. For billing, developers use it to atomically check and update a billing status entry while also calling Stripe: “if the entry says ‘not billed’, call Stripe and update entry to ‘billed’.”

The failure is triggered by primary member departure. If the member executing process() fails — OOM kill, hardware failure, JVM crash — after stripe.charges.create() returns but before the EntryProcessor returns and the result is acknowledged, Hazelcast promotes the backup partition to primary and the client-side Hazelcast SDK retries the operation. The retry sends the same serialized EntryProcessor object to the new primary and calls process() again. If the developer generated a UUID inside process() — intending to create a “unique run identifier” for the Stripe key — the re-execution generates a new, different UUID. The Stripe idempotency key is different from the one used in the original execution. Stripe has no record of a request with the new key and creates a fresh charge.

// Java Hazelcast — BillingEntryProcessor with UUID inside process() (unsafe)
import com.hazelcast.map.EntryProcessor;
import com.stripe.Stripe;
import com.stripe.exception.StripeException;
import com.stripe.model.Charge;
import com.stripe.param.ChargeCreateParams;
import com.stripe.net.RequestOptions;

import java.util.Map;
import java.util.UUID;

public class BillingEntryProcessor
        implements EntryProcessor<String, BillingEntry, String> {

    private final String billingPeriod;
    private final long amountCents;
    private final String stripeCustomerId;

    public BillingEntryProcessor(String billingPeriod, long amountCents, String stripeCustomerId) {
        this.billingPeriod = billingPeriod;
        this.amountCents = amountCents;
        this.stripeCustomerId = stripeCustomerId;
    }

    @Override
    public String process(Map.Entry<String, BillingEntry> entry) {
        BillingEntry current = entry.getValue();
        if (current != null && current.status.equals("BILLED")) {
            return current.chargeId; // already billed
        }

        // Unsafe: UUID generated inside process(). EntryProcessors are not idempotent
        // with respect to Hazelcast partition migration. If the primary member fails
        // after stripe.charges.create() returns ch_A but before this method returns,
        // Hazelcast re-executes process() on the backup. The re-execution generates
        // UUID.randomUUID() again — a new, different UUID. The Stripe key is different.
        // Stripe creates ch_B.
        //
        // Primary execution:  executionId = "a1b2c3d4-..." → sha256("a1b2c3d4-...:cust_123:Q3-2026") → ch_A
        // Re-exec on backup:  executionId = "f9e8d7c6-..." → sha256("f9e8d7c6-...:cust_123:Q3-2026") → ch_B
        String executionId = UUID.randomUUID().toString();
        String idempotencyKey = sha256(executionId + ":" + entry.getKey() + ":" + billingPeriod, 32);

        try {
            ChargeCreateParams params = ChargeCreateParams.builder()
                    .setAmount(amountCents)
                    .setCurrency("usd")
                    .setCustomer(stripeCustomerId)
                    .build();

            RequestOptions options = RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .build();

            Charge charge = Charge.create(params, options);

            // If the primary member fails here — after Charge.create() returns ch_A
            // but before process() returns — Hazelcast retries on the backup.
            // The backup's process() call generates a new UUID → ch_B.

            BillingEntry updated = new BillingEntry("BILLED", charge.getId());
            entry.setValue(updated);
            return charge.getId();

        } catch (StripeException e) {
            throw new RuntimeException("Stripe error in EntryProcessor", e);
        }
    }

    private static String sha256(String input, int len) {
        try {
            java.security.MessageDigest md = java.security.MessageDigest.getInstance("SHA-256");
            byte[] digest = md.digest(input.getBytes(java.nio.charset.StandardCharsets.UTF_8));
            StringBuilder sb = new StringBuilder();
            for (byte b : digest) sb.append(String.format("%02x", b));
            return sb.toString().substring(0, len);
        } catch (Exception e) { throw new RuntimeException(e); }
    }
}

The failure is invisible in standard Hazelcast monitoring. The executeOnKey() retry is transparent to the application: from the caller’s perspective, the method call returned normally (possibly with a brief additional latency during partition migration). The caller gets back a charge ID — but it is ch_B. Meanwhile, ch_A already exists in Stripe from the original execution. The customer has been charged twice. There is no exception, no error log, no dead letter queue. The IMap entry may now contain ch_B as the canonical charge ID while ch_A is a silent duplicate in Stripe.

A variant: the developer uses System.currentTimeMillis() or System.nanoTime() inside process() as a “processing timestamp” component of the idempotency key. These return different values on the re-execution (the primary failure and partition migration take time; even milliseconds are enough to produce a different timestamp-based key). A more subtle variant: the developer derives the key from the Hazelcast member’s UUID (Hazelcast.getCluster().getLocalMember().getUuid()). The primary and backup members have different UUIDs, so the re-execution produces a completely different key regardless of when it runs. Both variants have the same root cause: any value generated or read from the executing member’s local state inside process() is not stable across re-execution on a different member.

Failure mode 2: Reliable Topic subscriber sequence checkpoint stored in IMap with MaxIdle eviction — checkpoint evicted during a long maintenance window causes subscriber to replay the full RingBuffer retention window on restart, re-billing already-charged customers

Hazelcast Reliable Topic provides durable pub/sub backed by a cluster-wide Ringbuffer. Unlike traditional queues, each subscriber tracks its own position in the RingBuffer independently. When a subscriber restarts, it needs to know where to resume. A common pattern: store the last processed RingBuffer sequence number in an IMap entry keyed by the subscriber’s worker ID. On startup, read the stored sequence from IMap, configure the ReliableMessageListener to start from that sequence, and process only new events.

The failure is triggered by the interaction between IMap eviction policy and subscriber downtime. In production Hazelcast clusters, IMap memory management often includes a MaxIdle eviction policy: any entry not accessed for a configured duration is automatically evicted from all cluster members. This is a standard JVM heap pressure management technique. For a sequence checkpoint IMap, a typical configuration might set MaxIdle = 24 hours — if the subscriber is running and updating its checkpoint every few seconds, the entry is never at risk. But if the subscriber is taken offline for a maintenance window that runs longer than 24 hours — a planned outage that overruns, a Kubernetes deployment that stalls, or a circuit-breaker that keeps the subscriber paused — the IMap entry expires and is evicted from all cluster members.

When the subscriber restarts, it calls checkpointMap.get(workerId) and receives null (the entry was evicted). The developer’s error handler: “if null, start from the oldest available position to avoid missing any billing events.” InitialPosition.OLDEST starts the subscriber at the oldest sequence still present in the RingBuffer. If the RingBuffer was configured with 7-day retention and 100,000-event capacity, the subscriber will replay every billing event from the past week — including thousands of events that were already processed and charged during the subscriber’s previous run.

// Java Hazelcast — Reliable Topic subscriber with IMap MaxIdle checkpoint (unsafe)
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.topic.ITopic;
import com.hazelcast.topic.Message;
import com.hazelcast.topic.MessageListener;
import com.hazelcast.topic.ReliableMessageListener;
import com.hazelcast.map.IMap;
import com.stripe.model.Charge;
import com.stripe.param.ChargeCreateParams;
import com.stripe.net.RequestOptions;

import java.util.UUID;

public class BillingTopicSubscriber implements ReliableMessageListener<BillingEvent> {

    private final HazelcastInstance hz;
    private final IMap<String, Long> checkpointMap;
    private final String workerId;
    private long lastProcessedSeq = -1L;

    public BillingTopicSubscriber(HazelcastInstance hz, String workerId) {
        this.hz = hz;
        this.workerId = workerId;
        // checkpointMap config: MaxIdle=24h, BackupCount=2
        // Risk: entry evicted if subscriber is offline for > 24 hours.
        this.checkpointMap = hz.getMap("billing-seq-checkpoints");
    }

    @Override
    public long retrieveInitialSequence() {
        Long stored = checkpointMap.get(workerId);
        if (stored != null) {
            return stored + 1; // resume from after last processed
        }
        // Unsafe fallback: OLDEST replays the entire RingBuffer retention window.
        // If the 7-day RingBuffer has 80,000 billing events and this subscriber
        // processed 60,000 of them before the 26-hour maintenance window caused
        // the IMap entry to expire, this fallback re-processes all 80,000 —
        // creating 60,000 duplicate Stripe charges.
        return -1L; // Hazelcast: -1L means OLDEST
    }

    @Override
    public void storeSequence(long sequence) {
        // Update the checkpoint on every message (or every N messages for performance).
        // This is the right design — but it only helps if the IMap entry doesn't expire.
        checkpointMap.put(workerId, sequence);
        this.lastProcessedSeq = sequence;
    }

    @Override
    public boolean isLossTolerant() {
        // false: throw exception if RingBuffer has overwritten past this subscriber's seq.
        // true: fast-forward to current tail (lose events in the gap).
        // Neither option prevents the MaxIdle eviction failure — the fallback in
        // retrieveInitialSequence() fires before isLossTolerant() is even consulted.
        return false;
    }

    @Override
    public boolean isTerminal(Throwable failure) {
        return false; // retry on any error
    }

    @Override
    public void onMessage(Message<BillingEvent> message) {
        BillingEvent event = message.getMessageObject();

        // Unsafe: no pre-flight check. If this message was already processed and
        // Stripe was called successfully in a previous run, calling it again here
        // creates ch_B. The subscriber has no memory of what it processed before
        // the checkpoint was evicted.
        String idempotencyKey = sha256(event.customerId + ":" + event.billingPeriod + ":hz-topic", 32);
        try {
            Charge.create(
                ChargeCreateParams.builder()
                    .setAmount(event.amountCents)
                    .setCurrency("usd")
                    .setCustomer(event.stripeCustomerId)
                    .build(),
                RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .build()
            );
            // storeSequence() is called by Hazelcast after onMessage() returns normally.
        } catch (Exception e) {
            throw new RuntimeException(e); // trigger retry via isTerminal()
        }
    }

    private static String sha256(String input, int len) { /* ... */ return ""; }
}

The severity scales with the RingBuffer retention window and the billing event volume. A Reliable Topic configured with a 7-day TTL and 100,000-event capacity will accumulate up to 100,000 billing events. If 70,000 of those events represent customers already charged (processed during the subscriber’s previous run), the fallback to OLDEST schedules 70,000 duplicate Stripe charges to be processed serially as the subscriber works through the RingBuffer replay. The failures are silent: stripe.charges.create() returns a charge ID for each one; only the customer’s Stripe dashboard (and their bank statement) reveals that they were charged twice. Stripe’s 24-hour idempotency window prevents duplicates for events processed within the last 24 hours — but a 7-day retention window means events from days 2–7 are fully exposed. Content-hash idempotency keys extend this window to Stripe’s maximum (24 hours from first request), and the pre-flight PostgreSQL check extends it indefinitely.

A variant: developer sets MaxIdle on the checkpoint IMap to a sensible-seeming value of 72 hours for a service that “should never be down more than 3 days.” An extended incident (cascade failure, a multi-region network partition that takes 4 days to fully remediate) exceeds the TTL. A variant using TTL (time-to-live from insertion, not from last access) rather than MaxIdle creates a different eviction pattern: the entry expires 72 hours after the subscriber last wrote its checkpoint, which can happen during a busy period when the subscriber is actively running. The subscriber is not offline; it just wrote a checkpoint at time T and the entry expires at T+72h. If no new billing events arrive for 73 hours (a weekend pause in a B2B SaaS billing system), the entry expires silently while the subscriber is running normally. On the next billing event, the subscriber calls storeSequence(), which re-creates the entry — but retrieveInitialSequence() is only called at startup, so the running subscriber is unaffected. The subscriber that restarts after the entry has been re-populated with the most recent sequence is also unaffected. Only a restart that occurs in the window between the entry expiring and the subscriber processing the next event — a relatively narrow window — triggers the fallback. This makes the TTL-based variant harder to catch in staging environments where billing events arrive continuously.

Failure mode 3: CP Subsystem FencedLock session expiry during JVM GC pause allows two members to hold the lock simultaneously — fence token in Stripe idempotency key produces different keys and Stripe creates ch_B

Hazelcast’s CP Subsystem provides FencedLock, a linearizable distributed lock built on Raft. Unlike the legacy ILock (deprecated in Hazelcast 5), FencedLock uses a monotonically increasing fencing token to detect cases where the lock holder’s session expires and another member acquires the lock before the original holder realizes it. The canonical use of the fencing token is to pass it to a database or storage operation as a guard against stale writes from a session-expired holder. Developers who read about fencing tokens and apply them to Stripe billing — using the fence token as a component of the Stripe idempotency key to “version the billing attempt” — create a failure mode that is the mirror image of the protection they intended.

The failure sequence: Worker A acquires the lock, receives fencing token fence=1, stores it locally. Worker A computes the Stripe idempotency key as sha256(customerId + ":" + billingPeriod + ":" + fence) = sha256("cust_123:Q3-2026:1"). Worker A calls stripe.charges.create() with this key and gets back ch_A. At this moment, Worker A’s JVM enters a long GC pause (stop-the-world full collection, typical in pre-G1GC JVMs under heap pressure). The GC pause exceeds hazelcast.cp.session.heartbeat.interval.seconds (default: 5 seconds) times hazelcast.cp.session.miss.count (default: 3) = 15 seconds. The CP Subsystem’s Raft state machine marks Worker A’s CP session as expired and releases the lock. Worker B acquires the lock, receives fencing token fence=2. Worker B computes the Stripe idempotency key as sha256("cust_123:Q3-2026:2") — a different hash. Worker B calls stripe.charges.create() with this key. At this point, ch_A already exists in Stripe from Worker A’s call. Stripe does not know about Worker B’s key; it creates ch_B. Worker A’s GC pause ends. Worker A assumes it still holds the lock (it called lock.lock() and never called lock.unlock()) and believes its billing run completed when stripe.charges.create() returned ch_A before the GC pause. The customer is now charged twice.

// Java Hazelcast — FencedLock with fence token in idempotency key (unsafe)
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.cp.CPSubsystem;
import com.hazelcast.cp.lock.FencedLock;
import com.stripe.model.Charge;
import com.stripe.param.ChargeCreateParams;
import com.stripe.net.RequestOptions;

import java.util.concurrent.TimeUnit;

public class BillingWorker {

    private final HazelcastInstance hz;

    public String billCustomer(String customerId, String billingPeriod,
                               long amountCents, String stripeCustomerId) throws Exception {

        CPSubsystem cp = hz.getCPSubsystem();
        FencedLock lock = cp.getLock("billing-lock-" + billingPeriod);

        // Acquire the lock and get the fencing token.
        // FencedLock.lock() blocks until the lock is acquired.
        // If this member previously held the lock (cp session still valid),
        // it returns the same fencing token. If the session expired and another
        // member acquired and released the lock first, it returns a higher token.
        long fence = lock.lock();

        try {
            // Unsafe: fence token included in Stripe idempotency key.
            // Rationale: "I want each lock acquisition to produce a distinct billing key
            // so that if Worker A held the lock and partially billed, Worker B's key
            // (with a higher fence) will be treated as a new billing attempt."
            //
            // The actual result: if Worker A called Stripe with fence=1 (creating ch_A)
            // then lost its CP session during GC, and Worker B acquired the lock with fence=2,
            // Worker B's key (fence=2) produces a completely different hash. Stripe has never
            // seen this key and creates ch_B. The customer is charged twice.
            String idempotencyKey = sha256(customerId + ":" + billingPeriod + ":" + fence, 32);

            // Worker A calls Stripe with fence=1 key → ch_A
            // GC pause starts here. Worker A's CP session expires during the pause.
            // Worker B acquires lock with fence=2, calls Stripe with fence=2 key → ch_B.
            // Worker A's GC pause ends. It has ch_A already. Two charges exist.

            Charge charge = Charge.create(
                ChargeCreateParams.builder()
                    .setAmount(amountCents)
                    .setCurrency("usd")
                    .setCustomer(stripeCustomerId)
                    .build(),
                RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .build()
            );

            return charge.getId();

        } finally {
            // If this member's CP session expired during the try block,
            // unlock() will throw IllegalMonitorStateException
            // (because this member no longer holds the lock).
            // The finally block's exception masks the Stripe call's result.
            try { lock.unlock(); } catch (IllegalMonitorStateException e) {
                // Session expired during execution — lock was already released by CP Subsystem.
                // The Stripe call may or may not have succeeded. The charge ID is lost.
            }
        }
    }

    private static String sha256(String input, int len) { /* ... */ return ""; }
}

The failure is specific to the interaction between the fence token’s semantics and Stripe’s idempotency model. The fence token is designed to be used with storage systems that support compare-and-swap on the fence value (e.g., “only write if the stored fence value is less than my current fence”). Stripe does not support fence-based conditional writes. Stripe’s idempotency is a 24-hour cache keyed by the idempotency key string: the same key returns the same result; a different key always attempts a new charge. The fence token causes keys to differ across CP session boundaries, defeating Stripe’s deduplication exactly when it is most needed.

A variant: developer uses the legacy ILock API (available in Hazelcast 4.x, deprecated in 5.x). ILock does not use the CP Subsystem and does not have a monotonically increasing fencing token. Instead, it uses a lease mechanism tied to the holding member. When the holding member fails, the lease expires and another member can acquire the lock. The developer using ILock who includes ILock.getRemainingLeaseTime() or the cluster member’s UUID in the Stripe key faces the same failure mode: lease renewal time changes, member UUID changes on the new lock holder, and Stripe creates ch_B. A further variant: developer uses IAtomicLong from the CP Subsystem as a billing epoch counter and includes the current epoch value in the Stripe key. If the CP session expires and the IAtomicLong value has been incremented by another worker before the original resumes, the original worker reads a stale in-memory epoch value (from before the GC pause) while the new worker uses the incremented value. Same key-divergence failure, different CP primitive.

What doesn’t prevent duplicate Stripe charges in Hazelcast

The fix: content-hash idempotency key + pre-flight database check + vault key spend cap

Three changes close all three failure modes without modifying Hazelcast cluster topology, partition count, CP Subsystem session configuration, or Reliable Topic RingBuffer capacity.

1. Content-hash idempotency key derived exclusively from stable business fields

The Stripe idempotency key must be derived from fields that are identical on every possible execution path: original EntryProcessor execution, re-execution on backup, Reliable Topic replay from OLDEST position, and second FencedLock holder after session expiry. The billing event’s business fields — customer ID, billing period, and a service-level namespace — satisfy this requirement. Hazelcast execution and session metadata does not: EntryProcessor execution UUID, RingBuffer sequence number, fencing token, Hazelcast member UUID, CP session ID, partition owner address, lock lease expiry timestamp, and any value generated inside process() or inside the lock body all change across re-execution, replay, or lock re-acquisition.

// Java Hazelcast — Safe BillingEntryProcessor with content-hash key and pre-flight check
import com.hazelcast.map.EntryProcessor;
import org.springframework.jdbc.core.JdbcTemplate;
import com.stripe.model.Charge;
import com.stripe.param.ChargeCreateParams;
import com.stripe.net.RequestOptions;

import java.security.MessageDigest;

public class SafeBillingEntryProcessor
        implements EntryProcessor<String, BillingEntry, String> {

    private final String billingPeriod;
    private final long amountCents;
    private final String stripeCustomerId;
    // JdbcTemplate must be serializable or injected as transient + re-initialized.
    // In practice, use a Hazelcast ExecutorService pattern with a singleton datasource
    // on the executing member, or use a pre-flight check in the IMap entry itself.
    private final transient JdbcTemplate db;

    public SafeBillingEntryProcessor(String billingPeriod, long amountCents,
                                     String stripeCustomerId, JdbcTemplate db) {
        this.billingPeriod = billingPeriod;
        this.amountCents = amountCents;
        this.stripeCustomerId = stripeCustomerId;
        this.db = db;
    }

    @Override
    public String process(Map.Entry<String, BillingEntry> entry) {
        String customerId = entry.getKey();
        BillingEntry current = entry.getValue();

        // Check IMap entry first — fast path before touching external systems.
        if (current != null && current.status.equals("BILLED")) {
            return current.chargeId;
        }

        // Safe: content-hash key — same on every execution of this EntryProcessor,
        // whether running on the original primary, a re-execution on backup, or a
        // FencedLock-protected retry with a different fencing token.
        //
        // Excluded (all Hazelcast execution metadata):
        //   - UUID.randomUUID() inside process() — changes on every EntryProcessor invocation
        //   - System.currentTimeMillis() — changes between primary execution and backup re-exec
        //   - Hazelcast.getCluster().getLocalMember().getUuid() — different per cluster member
        //   - fence token from FencedLock.lock() — changes on every new lock acquisition
        //   - CP session ID — changes on CP session expiry and re-acquisition
        //   - RingBuffer sequence number — different on replay vs. original delivery
        String idempotencyKey = contentHash(customerId, billingPeriod, "hz-billing");

        // Pre-flight check in external PostgreSQL — survives Hazelcast member restarts,
        // IMap eviction, CP session expiry, and EntryProcessor re-execution on backup.
        int rows = db.update(
            "INSERT INTO billing_records (customer_id, billing_period, idempotency_key) " +
            "VALUES (?, ?, ?) ON CONFLICT (customer_id, billing_period) DO NOTHING",
            customerId, billingPeriod, idempotencyKey
        );

        if (rows == 0) {
            // Pre-flight row already exists: a prior EntryProcessor execution (on the
            // primary before it failed) already called Stripe. Skip Stripe, mark entry.
            BillingEntry lookupEntry = lookupBillingRecord(customerId, billingPeriod);
            if (lookupEntry != null) {
                entry.setValue(lookupEntry);
                return lookupEntry.chargeId;
            }
        }

        try {
            Charge charge = Charge.create(
                ChargeCreateParams.builder()
                    .setAmount(amountCents)
                    .setCurrency("usd")
                    .setCustomer(stripeCustomerId)
                    .build(),
                RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .build()
            );

            // Update external record with Stripe charge ID.
            db.update(
                "UPDATE billing_records SET charge_id = ?, status = 'BILLED' " +
                "WHERE customer_id = ? AND billing_period = ?",
                charge.getId(), customerId, billingPeriod
            );

            // Update IMap entry.
            entry.setValue(new BillingEntry("BILLED", charge.getId()));
            return charge.getId();

        } catch (Exception e) {
            throw new RuntimeException("Stripe error", e);
        }
    }

    private static String contentHash(String customerId, String billingPeriod, String ns) {
        try {
            String raw = customerId + ":" + billingPeriod + ":" + ns;
            MessageDigest md = MessageDigest.getInstance("SHA-256");
            byte[] digest = md.digest(raw.getBytes(java.nio.charset.StandardCharsets.UTF_8));
            StringBuilder sb = new StringBuilder();
            for (byte b : digest) sb.append(String.format("%02x", b));
            return sb.toString().substring(0, 32);
        } catch (Exception e) { throw new RuntimeException(e); }
    }

    private BillingEntry lookupBillingRecord(String customerId, String billingPeriod) {
        try {
            return db.queryForObject(
                "SELECT charge_id, status FROM billing_records WHERE customer_id = ? AND billing_period = ?",
                (rs, n) -> new BillingEntry(rs.getString("status"), rs.getString("charge_id")),
                customerId, billingPeriod
            );
        } catch (Exception e) { return null; }
    }
}

2. Pre-flight database check with ON CONFLICT DO NOTHING

The content-hash idempotency key closes FM1 and FM3 within Stripe’s 24-hour idempotency window: the same key on EntryProcessor re-execution returns ch_A silently, and the same key from the second FencedLock holder (if using the content-hash approach, not the fence token) also returns ch_A. FM2 is closed within 24 hours for events recently processed before the checkpoint eviction.

The pre-flight PostgreSQL check closes all three failure modes unconditionally. The billing worker executes INSERT INTO billing_records (customer_id, billing_period, idempotency_key) VALUES (?, ?, ?) ON CONFLICT (customer_id, billing_period) DO NOTHING before calling stripe.charges.create(). If the row already exists (rowsAffected == 0), the billing event was already processed and the Stripe call is skipped. PostgreSQL’s UNIQUE (customer_id, billing_period) constraint is the durable deduplication layer that survives Hazelcast member failures, IMap evictions, CP session expirations, and EntryProcessor re-executions. The IMap entry is a fast-path cache; the PostgreSQL row is the authoritative deduplication record.

Write-before-call ordering is critical: the INSERT must be committed to PostgreSQL before stripe.charges.create() is called. If the billing process is killed between the PostgreSQL commit and the Stripe call, the customer is not charged (correct: the pre-flight row prevents a later re-execution from charging either). If the process is killed between the Stripe call returning and the UPDATE billing_records SET charge_id = ? call, ch_A exists in Stripe and the pre-flight row exists in PostgreSQL. The next execution finds the pre-flight row (rowsAffected == 0) and skips Stripe. The charge_id column in billing_records can be backfilled by a reconciliation job that queries Stripe for the charge using the stable idempotency key.

3. Per-billing-period vault key capped at expected_total × 1.10

A Keybrake vault key issued per billing period with a spend cap of expected_total × 1.10 provides a hard backstop when both the content-hash key and the pre-flight check are transiently unable to prevent a duplicate charge. This occurs when the pre-flight PostgreSQL database is unreachable (connection pool exhaustion, rolling maintenance, network partition between the Hazelcast member and the database) at the moment of an EntryProcessor re-execution or a FencedLock session expiry. Without the vault cap, the Stripe call proceeds with the content-hash key (which returns ch_A if within 24 hours, or creates ch_B if outside 24 hours). With the vault cap, the Keybrake proxy compares the total charges for this billing period against the cap and rejects any charge that would exceed it. The cap absorbs the second charge regardless of the idempotency key value, providing defense in depth at the Stripe API boundary when both upstream layers (content-hash key and pre-flight check) fail simultaneously.

Gap analysis: Hazelcast-specific scenarios not covered by simpler approaches

Hazelcast split-brain partition healing — both brain halves processed the same billing events independently and merge

During a network partition that splits the Hazelcast cluster into two groups, if both groups have a quorum of CP members, both can continue operating independently. Both may process the same billing events concurrently. When the partition heals and the cluster merges, Hazelcast’s merge policy (e.g., LatestUpdateMergePolicy) resolves IMap conflicts by keeping the most recently updated entry. The IMap entry will reflect one billing attempt, but two Stripe charges may exist. The content-hash key prevents duplication within 24 hours (Stripe returns ch_A from the cache for both attempts). The pre-flight PostgreSQL check prevents duplication regardless of time, assuming PostgreSQL was not itself split. In a true split-brain where PostgreSQL is also partitioned, the pre-flight insert may succeed in both partitions independently if they write to different PostgreSQL replicas with eventual consistency. The vault key spend cap catches the second charge regardless.

Hazelcast near-cache stale reads — billing status read from near-cache shows “not billed” after the primary IMap entry was updated to “billed” by another member

IMap near-cache configuration caches IMap values on the client or on each member locally to reduce network round-trips. Near-cache entries are invalidated when the primary IMap entry is updated, but invalidation is asynchronous. In high-throughput billing scenarios with multiple members running EntryProcessors concurrently, a member that reads the IMap entry’s billing status from its near-cache may see a stale “not billed” value for up to the near-cache TTL (configurable, typically 1–10 seconds) after another member updated it to “billed.” If the billing status check in the EntryProcessor is based on a near-cache read rather than a direct IMap read, both members may conclude the customer has not been billed and both call Stripe. The content-hash key prevents ch_B within 24 hours; the pre-flight PostgreSQL check prevents it unconditionally. The safe pattern: EntryProcessor status checks always read from the partition-local entry (which is always current), never from near-cache. IMap.executeOnKey() runs the process() method on the primary partition owner with direct access to the entry, bypassing near-cache entirely. Near-cache risks appear when a developer reads the IMap entry before calling executeOnKey() to decide whether to execute at all.

Hazelcast jet pipeline re-processing on job restart — billing events in a tumbling window re-enter the pipeline after the job is restarted without snapshot state

Hazelcast Jet (now part of the Platform) processes streaming events with exactly-once or at-least-once delivery. A Jet pipeline that processes billing events reads from a streaming source (IMap journal, Kafka, or Kinesis) and may call Stripe from a mapUsingService() stage with an external Stripe client. If the Jet job is configured for at-least-once (the default) and the job is restarted after a failure, events that were in a processing window at the time of failure are re-processed from the last committed snapshot. If the snapshot was not taken immediately before the failure, events whose Stripe calls completed successfully (after the snapshot but before the failure) will be re-processed in the next job run. A content-hash idempotency key closes this case within 24 hours. The pre-flight PostgreSQL check closes it unconditionally. The safe pattern for Jet billing pipelines: always use a content-hash key in the Stripe call, and always use a pre-flight idempotency check in the external database, treating Jet’s at-least-once guarantee as the lower bound, not the upper bound.

Frequently asked questions

Can I use the Hazelcast member UUID as the Stripe idempotency key component to “scope billing to this cluster member”?

No. The Hazelcast member UUID (HazelcastInstance.getCluster().getLocalMember().getUuid()) is assigned to a member when it joins the cluster and does not change while the member is alive. But when the member fails and restarts, it receives a new UUID. A billing event re-processed after a member restart will use a different member UUID, producing a different Stripe idempotency key and creating a new charge. The member UUID is also useless for EntryProcessor re-execution: the EntryProcessor is re-executed on a different member (the backup that was promoted to primary), which has a different UUID by definition. The only safe idempotency key components for Stripe are fields derived from the billing event’s business data (customer ID, billing period, amount tier) that are identical on every possible re-execution path.

Does FencedLock prevent FM1 (EntryProcessor re-execution)?

No. FencedLock is a distributed lock primitive, not a distributed execution primitive. IMap.executeOnKey() handles its own concurrency through Hazelcast’s partition locking model: only one EntryProcessor can run on a given key at a time on the primary partition owner. The re-execution on backup after primary failure is not a concurrency issue in the traditional sense — it is a retry triggered by the Hazelcast client when the RPC to the primary fails. Wrapping the executeOnKey() call with a FencedLock on the calling member does not prevent the Hazelcast client from retrying the executeOnKey() on the backup after the primary fails. The FencedLock only serializes which calling thread/member initiates the executeOnKey() call; it does not prevent Hazelcast’s internal retry of the RPC.

If I configure the CP Subsystem session-heartbeat-interval-seconds to 1 second, does that prevent FM3?

A shorter heartbeat interval reduces the window for FM3 because the CP session expiry time = session-heartbeat-interval-seconds × session-miss-count. With 1s × 3 = 3 seconds, a GC pause longer than 3 seconds (instead of 15 seconds with the default 5s × 3) triggers session expiry. This reduces the required GC pause duration but does not eliminate the failure. Stop-the-world GC pauses of 3–10 seconds are common in pre-G1GC JVMs under heap pressure; even G1GC can produce pauses of 1–3 seconds under certain conditions. The correct fix is to not include the fencing token in the Stripe idempotency key, making the key independent of whether the session expired or not. Tuning the heartbeat interval is a mitigation, not a fix; it trades reduced FM3 frequency for increased CP Subsystem heartbeat traffic (more frequent Raft messages between CP members).

What if Stripe returns a network timeout from inside the EntryProcessor? Do I need to handle this differently?

Yes. A StripeException with a network timeout inside process() may mean that Stripe received the request and processed it (creating ch_A) but the response was lost in transit, or it may mean Stripe never received the request. In either case, the correct behavior is to let the EntryProcessor throw an exception (by not catching the StripeException and wrapping it in a RuntimeException instead). This causes executeOnKey() on the caller side to throw as well. The caller retries the executeOnKey() call. The EntryProcessor re-executes on the same primary (which is still alive — only a Stripe network error, not a Hazelcast failure). The re-execution uses the same content-hash idempotency key. If Stripe received and processed the original request, it returns ch_A with an HTTP 200 and the idempotency replay header. If Stripe never received the request, it creates ch_A now. In both cases, one charge. The pre-flight PostgreSQL check also catches the timeout-then-retry case: the pre-flight row was written before the first Stripe call, so the retry finds the row and skips Stripe if rowsAffected == 0. The only dangerous timeout variant is a StripeException.getShouldRetry() == false error (e.g., invalid card) where the developer calls a different Stripe API to create a new charge with a new key — in that case, the pre-flight row must be cleared or updated before the second Stripe API call.