Spring Data Cassandra and Stripe Integration: How WriteTimeoutException on the Billing Guard Table Is Treated as a Write Failure Causing @Retryable to Regenerate the Idempotency Key After ch_A Commits, an IF NOT EXISTS Lightweight Transaction WriteTimeoutException Leaves the [applied] Flag Unknown Causing a Re-Call to Stripe with a New Key, and a ConsistencyLevel.ONE Guard Read After a LOCAL_QUORUM Write Routes to a Replica With Replication Lag Returning Not Found After ch_A Commits

This post is part of the ongoing series on Stripe idempotency failure modes in JVM frameworks. Earlier entries covered Spring Data JPA (JDBC transaction boundary and Stripe commit ordering), Spring Cloud Circuit Breaker (Resilience4j TimeLimiter post-commit fallback), Spring Cloud OpenFeign (RequestInterceptor on every retry, retryOnAllOperations re-sends POST), Spring Data Elasticsearch (@Retryable UUID reset after VersionConflictEngineException), Spring Cloud Config (@RefreshScope bean destruction clears in-flight key cache), Spring Cloud Sleuth (@NewSpan fresh spanId per @Retryable invocation), Spring Data Neo4j (@Transactional rollback clears COMPLETED guard), and Spring Cloud Bus (busrefresh cluster-wide broadcast, RemoteApplicationEvent.getId() random at construction, spring.application.instance-id random per pod start). This post moves to Spring Data Cassandra. Cassandra’s distributed write model introduces a set of behaviors that are fundamentally different from RDBMS transaction semantics: writes can time out without failing, lightweight transactions use a multi-phase PAXOS protocol whose acknowledgment is separable from the write’s actual application, and tunable consistency levels create read-write combinations that do not guarantee that a just-written record is visible to a subsequent read. Each of these properties independently causes Stripe double-charges when billing idempotency guard logic is built without understanding Cassandra’s write ambiguity model.

The three failure modes are: a billing service that writes an idempotency guard to a Cassandra table and catches WriteTimeoutException as a write failure — annotated with @Retryable — where Cassandra’s WriteTimeoutException does not mean the write was not applied but only that the coordinator did not receive enough acknowledgments within the timeout window, causing the retry to call UUID.randomUUID() again, producing key_B, and committing ch_B alongside the already-committed ch_A; a billing service that uses a lightweight transaction (LWT) INSERT INTO charge_guards ... IF NOT EXISTS as its idempotency guard, where a WriteTimeoutException on the LWT means the PAXOS propose-and-commit phase may have completed before the final acknowledgment was lost, so the [applied] column value is unknown, and a developer who interprets the timeout as “guard was not written” proceeds to call Stripe with a new idempotency key and commits ch_B; and a billing service that writes the guard at ConsistencyLevel.LOCAL_QUORUM but reads it at ConsistencyLevel.ONE for performance, where the guard was committed to replicas A and B but the ONE read routes to replica C which has not yet applied the write due to replication lag, returns an empty result set, and the developer concludes no guard exists and generates key_B and commits ch_B.

Stack for examples: Spring Boot 3.x, Spring Data Cassandra 4.x (spring-boot-starter-data-cassandra), DataStax Java Driver 4.x, Spring Retry (spring-retry), Stripe Java SDK v24+, Testcontainers Cassandra 3.x (org.testcontainers:cassandra), WireMock 3.x, JUnit 5.

Background: what Cassandra’s write model does and why it interacts with billing idempotency

Cassandra is a distributed database that prioritizes availability and partition tolerance over strict consistency. Its write model is fundamentally different from that of a relational database, and every one of those differences has consequences for idempotency guard design in a billing system.

The first critical property is write ambiguity on timeout. When a Cassandra coordinator sends a write to the configured number of replica nodes and does not receive enough acknowledgments within the write_request_timeout_in_ms period, it throws a WriteTimeoutException. This exception does not mean the write was not applied. It means the coordinator gave up waiting. Some replicas may have applied the write and acknowledged it before the timeout; some may have applied it without the acknowledgment reaching the coordinator; some may not have received the write at all yet. The write’s state across the cluster is indeterminate at the moment the exception is thrown. A subsequent read may or may not find the written row, depending on which replica handles the read and how much time has passed since the write.

A developer who catches WriteTimeoutException and treats it as “the write failed, I can retry” is working from an incorrect mental model. The correct response to a Cassandra WriteTimeoutException is to perform a read-back to determine whether the write was applied before deciding to retry with a new value.

The second critical property is lightweight transaction PAXOS acknowledgment loss. Cassandra supports a limited form of conditional writes called lightweight transactions (LWTs), implemented via a four-phase PAXOS protocol: prepare, promise, propose, and commit. An LWT INSERT ... IF NOT EXISTS attempts to write a row only if no row with the same primary key already exists. The result set contains an [applied] boolean column: true if the row was inserted, false if a row already existed. LWTs are significantly more expensive than regular writes (multiple round trips) and more susceptible to timeout.

The pathological case: the PAXOS commit phase completes and the row is written, but before the result set containing [applied] = true reaches the client, the connection is interrupted or the client-side timeout fires. The client receives a WriteTimeoutException (or a NoHostAvailableException if the coordinator itself failed). The [applied] value is never delivered to the client. A developer who interprets this as “the guard was not written” may proceed to call Stripe with a new idempotency key — creating a double charge — when the guard was in fact durably written and the original Stripe charge may have also committed.

The third critical property is tunable consistency level mismatch. Cassandra allows each read and write operation to specify its own consistency level: ONE, QUORUM, LOCAL_QUORUM, ALL, and others. A write at LOCAL_QUORUM with replication factor 3 in a single datacenter writes to 2 out of 3 replicas before acknowledging to the client. A subsequent read at ONE reads from any single replica. If the read lands on the one replica that did not receive the write, it returns no row — even though the write was committed and is durably visible to a LOCAL_QUORUM read. The read-write consistency rule for guaranteed visibility is: write_CL + read_CL > replication_factor. LOCAL_QUORUM (2) + LOCAL_QUORUM (2) > 3: safe. LOCAL_QUORUM (2) + ONE (1) = 3: not safe (requires >3). A billing system that writes guards at LOCAL_QUORUM and reads them at ONE has a race condition window where the guard is not visible to the read.

As in every post in this series: Stripe deduplicates charges by the string value of the Idempotency-Key request header, scoped to the Stripe account. The key is opaque to Stripe — it does not inspect amount, customer, or currency. Two calls with different key strings are two distinct charge requests. A key that changes between the original call and any retry will always produce a second charge if the first call committed.

Mode 1: WriteTimeoutException on billing guard table INSERT after ch_A commits — developer catches exception as write failure — @Retryable regenerates UUID.randomUUID() — key_B — ch_B

The developer builds a billing service backed by Cassandra. The billing flow is: generate an idempotency key from UUID.randomUUID(), call Stripe, then write the key and charge ID to a Cassandra guard table for deduplication on future retries. The method is annotated with @Retryable to retry on CassandraException (the Spring Data Cassandra base exception for driver exceptions, which includes WriteTimeoutException). The developer’s intent: “If Cassandra is temporarily unavailable, retry the whole operation. A WriteTimeoutException means the write didn’t succeed; no guard was written; retrying is safe.”

This model has one fatal flaw: WriteTimeoutException does not mean the write was not applied. It means the coordinator gave up waiting for acknowledgments. If the guard write times out after a successful guard write at some replicas, the retry will regenerate UUID.randomUUID() (because the billing method re-executes from the top) and call Stripe with a new key. Stripe committed ch_A on the first attempt. The retry commits ch_B.

-- Cassandra DDL for the billing guard table
CREATE TABLE IF NOT EXISTS billing.charge_guards (
    subscription_id TEXT,
    billing_period  TEXT,
    idempotency_key TEXT,
    charge_id       TEXT,
    status          TEXT,
    created_at      TIMESTAMP,
    PRIMARY KEY ((subscription_id, billing_period))
);

// BillingService.java — @Retryable on CassandraException; UUID regenerated on retry
@Service
public class BillingService {

    @Autowired
    private CassandraTemplate cassandraTemplate;

    @Autowired
    private StripeClient stripeClient;

    // Developer intent: "If Cassandra is temporarily slow or unavailable,
    // @Retryable handles the transient error transparently."
    //
    // Flaw: @Retryable re-executes the method from the top on retry.
    // UUID.randomUUID() is called again — key_B on retry.
    // WriteTimeoutException does NOT mean the guard write failed.
    @Retryable(
        retryFor = { CassandraException.class },
        maxAttempts = 3,
        backoff = @Backoff(delay = 500, multiplier = 2)
    )
    public String chargeSubscription(String subscriptionId,
                                     String billingPeriod,
                                     int amountCents) {

        // WRONG: UUID.randomUUID() called at method entry.
        // If this method is retried by @Retryable, a new UUID is generated each time.
        // Stripe is called with key_A on attempt 1; if that commits ch_A and then
        // the guard write times out, the retry calls Stripe with key_B — ch_B committed.
        String idempotencyKey = UUID.randomUUID().toString();

        ChargeResponse charge = stripeClient.createCharge(
            findCustomerId(subscriptionId),
            amountCents,
            idempotencyKey
        );
        // ch_A committed in Stripe.

        // Write the guard to Cassandra.
        // If this throws WriteTimeoutException, the write may or may not have applied.
        // @Retryable catches CassandraException (supertype of WriteTimeoutException)
        // and re-invokes the method from the top — generating key_B.
        ChargeGuard guard = new ChargeGuard(
            subscriptionId, billingPeriod, idempotencyKey, charge.getId(), "COMPLETED"
        );
        cassandraTemplate.insert(guard);  // throws WriteTimeoutException → caught by @Retryable

        return charge.getId();
    }
}

The failure sequence:

1. At 10:00:00 UTC, the billing job processor calls chargeSubscription("sub_001", "2026-10", 4900). The method generates idempotencyKey = "ik_4a9b2c8d" (key_A).
2. stripeClient.createCharge(customerId, 4900, "ik_4a9b2c8d") is called. Stripe processes the charge and durably commits ch_A = "ch_1CassDef...". Stripe returns HTTP 200. charge.getId() holds "ch_A".
3. cassandraTemplate.insert(guard) is called. The Cassandra coordinator sends the INSERT to replicas A, B, and C. Replicas A and B acknowledge. Replica C is under GC pause and does not acknowledge within the timeout window. The coordinator throws WriteTimeoutException: received only 2 responses, required 3.
4. Spring Retry’s @Retryable intercepts the WriteTimeoutException (which is a subclass of CassandraException). Retry count = 1. After the 500ms back-off delay, it re-invokes chargeSubscription("sub_001", "2026-10", 4900).
5. The method executes from the top. idempotencyKey = UUID.randomUUID() = "ik_7e3f1a0b" (key_B). No guard read is performed — the method goes straight to stripeClient.createCharge().
6. stripeClient.createCharge(customerId, 4900, "ik_7e3f1a0b") is called. Stripe looks up "ik_7e3f1a0b". It has no record for this key (it knows only about "ik_4a9b2c8d" → ch_A). Stripe processes a new charge and commits ch_B = "ch_1CassXyz...". The customer is billed $49.00 twice.
7. Replica C comes back online. It receives the hinted handoff for the original guard write (subscriptionId="sub_001", billingPeriod="2026-10", idempotencyKey="ik_4a9b2c8d", chargeId="ch_A"). The guard is now visible on all three replicas. The retry’s guard write for key_B and ch_B also completes. The guard table now has two records for ("sub_001", "2026-10") — but since the primary key is ((subscription_id, billing_period)), the second write overwrites the first. The guard table shows ch_B. ch_A is an orphaned charge in Stripe with no corresponding guard record.

The developer’s mental model — “WriteTimeoutException means the write failed; retrying the whole method is safe” — collapses because Cassandra’s write semantics are fundamentally different from RDBMS transaction semantics. In an RDBMS with serializable isolation and synchronous replication, a write exception reliably means the write was not committed. In Cassandra, a WriteTimeoutException is an acknowledgment timeout, not a write failure. The correct interpretation is: “I don’t know whether the write was applied. I must read the guard before deciding whether to retry with the same key or treat the write as successful.”

Why WriteTimeoutException does not mean the write was not applied

Cassandra’s write path in the standard LOCAL_QUORUM case (replication factor 3): the coordinator node receives the write request from the client and forwards it to all three replica nodes simultaneously. Each replica writes to its commit log and memtable and sends an acknowledgment to the coordinator. The coordinator waits for a quorum (2 out of 3) of acknowledgments. If it receives 2 acknowledgments within the timeout, it acknowledges the write to the client. If it receives only 1 (or 0) within the timeout, it throws WriteTimeoutException.

At the moment the coordinator throws the exception: replica A may have already acknowledged; replica B may have written to its commit log but the acknowledgment was lost in transit; replica C may not have received the write yet. Replica A’s copy of the data is durable. Replica B’s copy is durable (commit log is flushed before acknowledgment in Cassandra’s default configuration). The coordinator has not lost the writes — it has lost track of the acknowledgments. The write exists on at least one replica in most timeout scenarios; it is just not confirmed to the client at quorum.

The fix has two parts: first, change the @Retryable to not retry on CassandraException in the outer method, and instead handle WriteTimeoutException from the guard write specifically by reading back the guard before deciding to proceed. Second, derive the idempotency key from stable inputs rather than UUID.randomUUID(), so that even if the method does need to retry, the retry uses the same key.

// Fixed BillingService.java — stable key derivation; read-before-retry on WriteTimeoutException
@Service
public class BillingService {

    @Autowired
    private CassandraTemplate cassandraTemplate;

    @Autowired
    private StripeClient stripeClient;

    public String chargeSubscription(String subscriptionId,
                                     String billingPeriod,
                                     int amountCents) {

        // Key derived from stable business inputs.
        // Same key on every invocation for the same (subscriptionId, billingPeriod, amountCents).
        // No UUID.randomUUID() — key cannot change between attempts.
        String idempotencyKey = toStableKey(subscriptionId, billingPeriod, amountCents);

        // Pre-flight guard check: if a completed guard already exists, return early.
        ChargeGuard existing = cassandraTemplate.selectOne(
            Query.query(
                Criteria.where("subscription_id").is(subscriptionId)
                    .and("billing_period").is(billingPeriod)
            ),
            ChargeGuard.class
        );
        if (existing != null && "COMPLETED".equals(existing.getStatus())) {
            return existing.getChargeId();
        }

        ChargeResponse charge = stripeClient.createCharge(
            findCustomerId(subscriptionId),
            amountCents,
            idempotencyKey
        );

        ChargeGuard guard = new ChargeGuard(
            subscriptionId, billingPeriod, idempotencyKey, charge.getId(), "COMPLETED"
        );

        try {
            cassandraTemplate.insert(guard);
        } catch (WriteTimeoutException e) {
            // WriteTimeoutException: write may or may not have been applied.
            // Do NOT retry with a new key. Read back to verify.
            ChargeGuard written = cassandraTemplate.selectOne(
                Query.query(
                    Criteria.where("subscription_id").is(subscriptionId)
                        .and("billing_period").is(billingPeriod)
                ),
                ChargeGuard.class
            );
            if (written == null) {
                // Write genuinely did not apply. Retry the insert (same guard, same key).
                cassandraTemplate.insert(guard);
            }
            // If written != null, the guard was applied. Both paths return ch_A.
        }

        return charge.getId();
    }

    static String toStableKey(String subscriptionId, String billingPeriod, int amountCents) {
        try {
            MessageDigest sha = MessageDigest.getInstance("SHA-256");
            byte[] hash = sha.digest(
                (subscriptionId + ":" + billingPeriod + ":" + amountCents)
                    .getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash).substring(0, 36);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

The stable key (toStableKey) ensures that even if the method is called multiple times for the same (subscriptionId, billingPeriod, amountCents) — by any retry mechanism, whether @Retryable, a message queue redelivery, or a cron job re-run — all calls produce the same idempotencyKey and Stripe’s deduplication prevents ch_B. The read-back on WriteTimeoutException prevents a guard re-insert from overwriting a guard that was already written, ensuring the guard table remains consistent even after a coordinator timeout.

Test for Mode 1

@SpringBootTest
@AutoConfigureWireMock(port = 0)
class WriteTimeoutRetryDoubleChargeTest {

    @Autowired
    private BillingService billingService;

    @Autowired
    private CassandraTemplate cassandraTemplate;

    @Test
    void writeTimeoutException_with_uuid_key_causes_double_charge() {
        // Stripe: accept any key, return a new charge ID each time
        stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody("{\"id\":\"ch_test_{{randomValue}}\",\"status\":\"succeeded\"}")));

        // Simulate WriteTimeoutException on guard insert (first call),
        // success on retry attempt (second Cassandra write).
        // With buggy code: both Stripe calls use different UUID keys — two charges created.
        // Verify Stripe was called twice:
        billingService.chargeSubscription("sub_001", "2026-10", 4900);
        verify(2, postRequestedFor(urlEqualTo("/v1/charges")));
    }

    @Test
    void writeTimeoutException_with_stable_key_does_not_cause_double_charge() {
        String expectedKey = BillingService.toStableKey("sub_001", "2026-10", 4900);

        stubFor(post(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", equalTo(expectedKey))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody("{\"id\":\"ch_A\",\"status\":\"succeeded\"}")));

        billingService.chargeSubscription("sub_001", "2026-10", 4900);

        // Stable key: even if Cassandra write times out and method is retried,
        // all Stripe calls use the same key — Stripe returns ch_A on all calls.
        verify(postRequestedFor(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", equalTo(expectedKey)));

        // Guard exists in Cassandra with the stable key
        ChargeGuard guard = cassandraTemplate.selectOne(
            Query.query(Criteria.where("subscription_id").is("sub_001")
                .and("billing_period").is("2026-10")),
            ChargeGuard.class
        );
        assertThat(guard).isNotNull();
        assertThat(guard.getIdempotencyKey()).isEqualTo(expectedKey);
        assertThat(guard.getChargeId()).isEqualTo("ch_A");
    }
}

Mode 2: IF NOT EXISTS lightweight transaction WriteTimeoutException — PAXOS commit phase may have completed before acknowledgment was lost — developer interprets timeout as guard not written — calls Stripe with new idempotency key — ch_B

The developer refactors the billing service to use a Cassandra lightweight transaction (LWT) as the idempotency guard. The LWT INSERT INTO charge_guards ... IF NOT EXISTS offers conditional write semantics: the guard is written only if no row with the same (subscription_id, billing_period) primary key exists. The result set contains an [applied] boolean: true if the guard was written, false if it already existed. The developer’s intent is to avoid a separate read-before-write round trip: the LWT either succeeds (guard written, safe to proceed) or fails (guard already exists, charge already committed, return the existing charge ID).

The developer handles WriteTimeoutException from the LWT by assuming the guard was not written and falls back to calling Stripe with a freshly generated idempotency key. This reasoning appears sound: if the LWT failed, no guard exists, so the original idempotency key was never committed to the guard, and generating a new one is safe. The flaw is that LWT’s PAXOS protocol is more complex than a regular write, and WriteTimeoutException on an LWT carries a different meaning than on a regular write.

-- Cassandra DDL for the LWT guard table
CREATE TABLE IF NOT EXISTS billing.charge_guards_lwt (
    subscription_id TEXT,
    billing_period  TEXT,
    idempotency_key TEXT,
    charge_id       TEXT,
    status          TEXT,
    PRIMARY KEY ((subscription_id, billing_period))
);

// BillingService.java — LWT guard with IF NOT EXISTS; WriteTimeoutException fallback
@Service
public class BillingService {

    @Autowired
    private CassandraOperations cassandraOperations;

    @Autowired
    private StripeClient stripeClient;

    public String chargeSubscription(String subscriptionId,
                                     String billingPeriod,
                                     int amountCents) {

        String idempotencyKey = toStableKey(subscriptionId, billingPeriod, amountCents);

        // Attempt to insert the guard with IF NOT EXISTS.
        // If [applied] = true: guard written, safe to call Stripe.
        // If [applied] = false: guard already exists — fetch existing charge ID, return.
        // If WriteTimeoutException: ???
        InsertOptions options = InsertOptions.builder()
            .withIfNotExists()
            .build();

        WriteResult writeResult;
        try {
            writeResult = cassandraOperations.insert(
                new ChargeGuardLwt(subscriptionId, billingPeriod, idempotencyKey, null, "PENDING"),
                options
            );
        } catch (WriteTimeoutException e) {
            // Developer intent: "WriteTimeoutException means the LWT failed.
            // No guard was written. I can proceed with a new key."
            //
            // Flaw: WriteTimeoutException on an LWT means the PAXOS commit phase
            // MAY have completed. The guard MAY have been written.
            // The [applied] value was never delivered to the client.
            // Proceeding to call Stripe with idempotencyKey is unsafe —
            // the guard was written with idempotencyKey, and Stripe may already
            // have committed ch_A on a previous attempt.
            //
            // Calling Stripe again with a NEW idempotencyKey (even if derived stably)
            // is safe ONLY if the original Stripe call had not yet been made.
            // If the original Stripe call DID commit ch_A before this LWT was attempted,
            // and then the LWT timed out, calling Stripe again after the timeout
            // (regardless of key) risks a double charge if the billing flow
            // is not carefully separated from the LWT guard.
            //
            // The specific failure mode below: the entire method is retried with a
            // UUID.randomUUID()-based key (the developer hasn't adopted stable keys),
            // so key_B is used and ch_B is committed.
            log.error("LWT WriteTimeoutException — assuming guard not written", e);
            idempotencyKey = UUID.randomUUID().toString();  // key_B on LWT timeout
            // fall through to Stripe call with key_B
            writeResult = null;  // treat as if guard was written fresh
        }

        if (writeResult != null && !writeResult.wasApplied()) {
            // Guard already exists — return the existing charge ID.
            ChargeGuardLwt existing = cassandraOperations.selectOne(
                Query.query(Criteria.where("subscription_id").is(subscriptionId)
                    .and("billing_period").is(billingPeriod)),
                ChargeGuardLwt.class
            );
            return existing != null ? existing.getChargeId() : null;
        }

        // Call Stripe.  With LWT timeout fallback: may use key_B here.
        ChargeResponse charge = stripeClient.createCharge(
            findCustomerId(subscriptionId),
            amountCents,
            idempotencyKey
        );

        // Update the guard with the actual charge ID and status.
        cassandraOperations.update(
            new ChargeGuardLwt(subscriptionId, billingPeriod, idempotencyKey, charge.getId(), "COMPLETED")
        );

        return charge.getId();
    }
}

The failure sequence when the billing service is called twice for the same subscription (first call times out from the caller’s perspective; caller retries):

1. First attempt: idempotencyKey = "stable_key_A" (derived stably from inputs). LWT INSERT ... IF NOT EXISTS executes. PAXOS prepare phase: coordinator sends prepare to replicas A, B, C; all reply with promise. PAXOS propose phase: coordinator sends propose to replicas A, B, C; replicas A and B accept; replica C is under load. PAXOS commit phase: coordinator sends commit to replicas A and B (quorum achieved); replicas A and B apply the write. Acknowledgment to coordinator confirmed from A and B. But before the coordinator can send the final response to the client, the coordinator’s connection to the client application server is interrupted. The client receives WriteTimeoutException.
2. The guard row (subscription_id="sub_001", billing_period="2026-10", idempotency_key="stable_key_A", charge_id=null, status="PENDING") is now durably written to replicas A and B. Replica C has it via hint delivery shortly after.
3. The developer’s catch (WriteTimeoutException) block executes. It generates a new idempotencyKey = UUID.randomUUID() = "ik_uuid_B" (key_B). It calls stripeClient.createCharge(customerId, 4900, "ik_uuid_B").
4. Meanwhile, the original first attempt’s Stripe call (which ran in a separate thread or was submitted before the LWT) actually committed ch_A with key "stable_key_A". Now the retry calls Stripe with "ik_uuid_B". Stripe commits ch_B. Double charge.

Note: this failure requires that the Stripe call happened before the LWT guard attempt, or is retried after the LWT timeout in a code path that doesn’t check the guard. If the LWT guard is designed to run before the Stripe call (which is the correct design — write the guard first, then call Stripe), the failure mode shifts: the LWT guard times out, but Stripe has not been called yet. In this case the developer’s fallback to a new key would be safe for the Stripe call but would leave an orphaned PENDING guard row with stable_key_A in the guard table. Any subsequent check of the guard table for stable_key_A would find a PENDING row and might correctly refuse to proceed — leaving the billing job stuck. The correct resolution in all cases is to read back the LWT guard before deciding anything.

The PAXOS timeout distinction: what WriteTimeoutException.getWriteType() tells you

Cassandra’s DataStax Java Driver provides a WriteTimeoutException with a getWriteType() method that returns the type of write that timed out. For LWT operations, the write type is WriteType.CAS (compare-and-swap). For regular writes, it is WriteType.SIMPLE or WriteType.BATCH. The DataStax documentation states explicitly: “For CAS write timeouts, there is no way to know whether the PAXOS round completed or not. If the write type is CAS, you should read the value to find out.”

// Correct handling of WriteTimeoutException on an LWT
try {
    writeResult = cassandraOperations.insert(
        new ChargeGuardLwt(subscriptionId, billingPeriod, idempotencyKey, null, "PENDING"),
        InsertOptions.builder().withIfNotExists().build()
    );
} catch (WriteTimeoutException e) {
    if (e.getWriteType() == WriteType.CAS) {
        // LWT timeout: PAXOS round may or may not have completed.
        // Read back to determine the actual state.
        ChargeGuardLwt current = cassandraOperations.selectOne(
            Query.query(Criteria.where("subscription_id").is(subscriptionId)
                .and("billing_period").is(billingPeriod)),
            ChargeGuardLwt.class
        );

        if (current != null && idempotencyKey.equals(current.getIdempotencyKey())) {
            // Guard was written with our key. Proceed as if LWT succeeded.
            writeResult = WriteResult.APPLIED;  // or equivalent sentinel
        } else if (current != null && !idempotencyKey.equals(current.getIdempotencyKey())) {
            // Guard was written with a different key by a concurrent attempt.
            // Treat as [applied] = false: return existing charge.
            return fetchExistingCharge(subscriptionId, billingPeriod);
        } else {
            // current == null: guard genuinely not written. Safe to retry LWT.
            writeResult = cassandraOperations.insert(
                new ChargeGuardLwt(subscriptionId, billingPeriod, idempotencyKey, null, "PENDING"),
                InsertOptions.builder().withIfNotExists().build()
            );
        }
    } else {
        throw e;  // non-LWT write timeout: propagate
    }
}

The read-back determines three possible states: the guard exists with idempotencyKey (the LWT committed; proceed as applied), the guard exists with a different key (a concurrent attempt won the PAXOS race; treat as [applied] = false), or the guard does not exist (the LWT genuinely was not committed; safe to retry with the same key). None of these states result in calling Stripe with a new key derived from UUID.randomUUID().

Why LWT guard design matters: guard-before-Stripe vs. guard-after-Stripe

The order of operations matters for LWT guard correctness:

• Guard-before-Stripe (correct): Write PENDING guard with LWT → if applied, call Stripe → update guard to COMPLETED with charge_id. If the LWT is not applied, a COMPLETED guard with an existing charge_id is returned to the caller. If the LWT times out: read back, determine guard state, retry only the LWT (not the Stripe call). Stripe is never called twice because the Stripe call is gated on a successful LWT application.
• Guard-after-Stripe (incorrect): Call Stripe → write guard with LWT. If Stripe commits ch_A and then the LWT times out, there is a window where ch_A exists in Stripe but no guard exists in Cassandra. A retry can commit ch_B if it generates a new key before reading the guard. This is the pattern in the buggy example above.

The guard-before-Stripe pattern ensures that the Stripe call is inside the guard’s protection: the guard record claims the billing period for the subscription before the charge is attempted. A concurrent retry that arrives during the PAXOS timeout window finds the guard (once it propagates) and short-circuits without calling Stripe. The guard-after-Stripe pattern is vulnerable to the exact window between Stripe commit and guard write, which is the window where every double-charge failure described in this series occurs.

Test for Mode 2

@SpringBootTest
@AutoConfigureWireMock(port = 0)
class LwtWriteTimeoutDoubleChargeTest {

    @Autowired
    private BillingService billingService;

    @Autowired
    private CassandraOperations cassandraOperations;

    @Test
    void lwt_timeout_with_uuid_key_causes_double_charge_when_stripe_called_before_guard() {
        // Stripe: accept any key, return new charge ID per call
        stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody("{\"id\":\"ch_{{randomValue}}\",\"status\":\"succeeded\"}")));

        // Simulate: LWT times out, developer re-generates UUID, calls Stripe again
        // Expected (buggy): two Stripe calls with different keys
        billingService.chargeSubscription("sub_002", "2026-10", 9900);
        verify(2, postRequestedFor(urlEqualTo("/v1/charges")));
    }

    @Test
    void lwt_timeout_readback_prevents_double_charge() {
        String stableKey = BillingService.toStableKey("sub_002", "2026-10", 9900);

        // Stripe: accept the stable key, return ch_A
        stubFor(post(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", equalTo(stableKey))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody("{\"id\":\"ch_A\",\"status\":\"succeeded\"}")));

        billingService.chargeSubscription("sub_002", "2026-10", 9900);

        // With correct LWT timeout handling + stable key:
        // - LWT applied (guard exists with stableKey)
        // - Stripe called once with stableKey
        // - Guard updated to COMPLETED with ch_A
        verify(1, postRequestedFor(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", equalTo(stableKey)));

        ChargeGuardLwt guard = cassandraOperations.selectOne(
            Query.query(Criteria.where("subscription_id").is("sub_002")
                .and("billing_period").is("2026-10")),
            ChargeGuardLwt.class
        );
        assertThat(guard).isNotNull();
        assertThat(guard.getStatus()).isEqualTo("COMPLETED");
        assertThat(guard.getChargeId()).isEqualTo("ch_A");
    }
}

Mode 3: ConsistencyLevel.ONE guard read after LOCAL_QUORUM guard write routes to a replica with replication lag — guard returns not found — developer generates new UUID — key_B — ch_B

The developer adopts stable key derivation and guard-before-Stripe ordering. The billing flow now correctly writes the guard before calling Stripe. However, for performance, the developer reads the guard at ConsistencyLevel.ONE (reads from a single replica) while writing at ConsistencyLevel.LOCAL_QUORUM (writes to a quorum of local datacenter replicas). The reasoning: “LOCAL_QUORUM reads are expensive and add latency. For a pre-flight guard check, ONE is sufficient — if the guard exists anywhere, ONE will find it quickly.”

This reasoning is incorrect. Cassandra’s consistency model guarantees that a read reflects a write only if the sum of the write’s consistency level (number of replicas that acknowledged) and the read’s consistency level (number of replicas consulted) exceeds the replication factor. For replication factor 3: LOCAL_QUORUM (2) + LOCAL_QUORUM (2) = 4 > 3 → guaranteed. LOCAL_QUORUM (2) + ONE (1) = 3 = 3 → not guaranteed (“greater than,” not “greater than or equal to”). The write may have gone to replicas A and B. The read may route to replica C, which has not yet received the write. The read returns no row. The developer concludes the guard is absent and calls Stripe.

// BillingService.java — correct guard-before-Stripe logic; WRONG consistency levels
@Service
public class BillingService {

    // Spring Data Cassandra configuration:
    // READ consistency: ONE (configured for performance in CassandraConfig)
    // WRITE consistency: LOCAL_QUORUM (configured for durability in CassandraConfig)
    //
    // This mismatch creates a read-write visibility gap.

    @Autowired
    private CassandraOperations cassandraOperations;

    @Autowired
    private StripeClient stripeClient;

    public String chargeSubscription(String subscriptionId,
                                     String billingPeriod,
                                     int amountCents) {

        String idempotencyKey = toStableKey(subscriptionId, billingPeriod, amountCents);

        // Guard read at ConsistencyLevel.ONE (configured globally in CassandraConfig).
        // WRONG: if the guard was written at LOCAL_QUORUM to replicas A and B,
        // and this read routes to replica C (which hasn't received the write yet),
        // the read returns null — even though the guard exists on A and B.
        ChargeGuard existing = cassandraOperations.selectOne(
            Query.query(
                Criteria.where("subscription_id").is(subscriptionId)
                    .and("billing_period").is(billingPeriod)
            ),
            ChargeGuard.class
        );

        if (existing != null && "COMPLETED".equals(existing.getStatus())) {
            // Guard found — charge already committed. Return existing charge ID.
            return existing.getChargeId();
        }

        // Guard not found (may be a false negative due to consistency level mismatch).
        // Developer intent: "Guard absent = safe to proceed."
        // Reality: guard may exist on A and B but C hasn't received it yet.

        // Write guard (PENDING) before calling Stripe.
        // This write also goes at LOCAL_QUORUM — goes to A and B (or C and A).
        cassandraOperations.insert(
            new ChargeGuard(subscriptionId, billingPeriod, idempotencyKey, null, "PENDING")
        );

        ChargeResponse charge = stripeClient.createCharge(
            findCustomerId(subscriptionId),
            amountCents,
            idempotencyKey
        );

        cassandraOperations.update(
            new ChargeGuard(subscriptionId, billingPeriod, idempotencyKey, charge.getId(), "COMPLETED")
        );

        return charge.getId();
    }
}

// CassandraConfig.java — global consistency levels: ONE for reads, LOCAL_QUORUM for writes
@Configuration
public class CassandraConfig extends AbstractCassandraConfiguration {

    @Override
    protected String getKeyspaceName() { return "billing"; }

    @Bean
    @Override
    public CassandraClusterFactoryBean cluster() {
        CassandraClusterFactoryBean factory = super.cluster();
        factory.setContactPoints("cassandra-host");
        return factory;
    }

    @Bean
    @Override
    public CassandraSessionFactoryBean session() throws Exception {
        CassandraSessionFactoryBean factory = super.session();
        // WRONG: read consistency ONE does not guarantee seeing a LOCAL_QUORUM write
        // immediately (with RF=3). Use LOCAL_QUORUM for both to guarantee visibility.
        factory.setDefaultReadOptions(
            QueryOptions.builder().consistencyLevel(ConsistencyLevel.ONE).build()
        );
        factory.setDefaultWriteOptions(
            QueryOptions.builder().consistencyLevel(ConsistencyLevel.LOCAL_QUORUM).build()
        );
        return factory;
    }
}

The failure sequence:

1. At 11:00:00 UTC, the billing cron job calls chargeSubscription("sub_003", "2026-10", 9900). idempotencyKey = "stable_k_sub003". Guard read at ONE → routes to replica C (load balancer round-robin). Replica C has no guard for ("sub_003", "2026-10"). Returns null. Developer concludes guard is absent.
2. Developer writes PENDING guard to Cassandra at LOCAL_QUORUM. Write goes to replicas A and B (quorum). Guard exists on A and B; replica C receives it via asynchronous gossip/hinted handoff.
3. Developer calls stripeClient.createCharge(customerId, 9900, "stable_k_sub003"). Stripe commits ch_A and returns HTTP 200. Developer updates guard to COMPLETED on A and B.
4. At 11:00:45 UTC, a network partition temporarily isolates replicas A and B from the application. The guard update (COMPLETED, charge_id = "ch_A") is committed on A and B but the application loses the connection before receiving the acknowledgment. The application server’s connection pool reconnects and now routes exclusively to replica C for the next few seconds.
5. At 11:00:48 UTC, the billing cron job’s retry logic (the caller side, not @Retryable) detects no response and retries. Calls chargeSubscription("sub_003", "2026-10", 9900) again.
6. Guard read at ONE → routes to replica C (still the only reachable replica from the application’s perspective). Replica C has the PENDING guard (received from hinted handoff at 11:00:03) but the COMPLETED update (written at 11:00:35 to A and B) has not yet been gossip-replicated to C. Replica C returns the PENDING guard row. existing.getStatus() = "PENDING" — not "COMPLETED". The guard check does not short-circuit.
7. Developer calls stripeClient.createCharge(customerId, 9900, "stable_k_sub003"). Stripe looks up "stable_k_sub003" and finds the already-committed ch_A. It returns ch_A without creating a new charge. No double charge in this scenario because the stable key is used.

Note that in step 7, the stable key saves the billing system: Stripe deduplicates and returns ch_A. The consistency level mismatch did not cause a double charge in this trace because the idempotency key was stable. The double charge failure from Mode 3 occurs specifically when the guard read returns null (not a stale PENDING record) and the developer then generates a new UUID.randomUUID() key. Let me show that specific trace:

1. First call: guard written at LOCAL_QUORUM to replicas A and B. Guard read returns null because read routes to replica C (replication lag — C received the write 150ms later). Developer concludes guard absent. Calls Stripe with idempotencyKey = UUID.randomUUID() = "ik_uuid_A". Stripe commits ch_A. Guard updated to COMPLETED on A and B with charge_id = "ch_A".
2. Caller retries 2 seconds later (timeout from caller’s perspective — step 1 response was lost). Guard read routes to replica C again (still within the replication lag window — the COMPLETED update is on A and B but the initial PENDING write hasn’t even reached C yet). Returns null. Developer generates idempotencyKey = UUID.randomUUID() = "ik_uuid_B". Calls Stripe. Stripe has no record of "ik_uuid_B". Commits ch_B. Double charge.

This trace uses UUID.randomUUID(). With stable key derivation, both calls use "stable_k_sub003" and Stripe deduplicates even if the guard read returns null. This is why the double-charge failure in Mode 3 is ultimately still rooted in non-stable key derivation — the consistency level mismatch is a contributing factor that makes the guard read unreliable, but the final defense against double charges is the stable key at the Stripe layer.

The read-write consistency rule for Cassandra guard tables

For a guard table to be reliable across all transient replication lag scenarios, the read and write consistency levels must satisfy: write_consistency_nodes + read_consistency_nodes > replication_factor. With replication factor 3:

For billing guard tables, the recommended combination is LOCAL_QUORUM for both reads and writes. This guarantees that a guard written at LOCAL_QUORUM is visible to any subsequent read at LOCAL_QUORUM, eliminating the replication lag window that Mode 3 exploits. The latency cost of LOCAL_QUORUM reads compared to ONE reads is typically 5–15ms in a well-tuned Cassandra cluster — acceptable for a billing operation that is already making a synchronous Stripe API call.

// Fixed CassandraConfig.java — LOCAL_QUORUM for both reads and writes on billing tables
@Configuration
public class CassandraConfig extends AbstractCassandraConfiguration {

    @Override
    protected String getKeyspaceName() { return "billing"; }

    @Bean
    public ConsistencyLevel billingReadConsistency() {
        // LOCAL_QUORUM ensures a just-written guard is visible to the next read.
        // Required because write CL is LOCAL_QUORUM and RF=3.
        return ConsistencyLevel.LOCAL_QUORUM;
    }

    @Bean
    public ConsistencyLevel billingWriteConsistency() {
        return ConsistencyLevel.LOCAL_QUORUM;
    }
}

// BillingRepository using explicit consistency levels per query (preferred over global config):
@Repository
public class ChargeGuardRepository {

    @Autowired
    private CassandraOperations cassandraOperations;

    public Optional<ChargeGuard> findBySubscriptionAndPeriod(String subscriptionId,
                                                               String billingPeriod) {
        // Explicit LOCAL_QUORUM read — not inherited from global config.
        // Guarantees visibility of any LOCAL_QUORUM-written guard.
        return Optional.ofNullable(cassandraOperations.selectOne(
            Query.query(
                    Criteria.where("subscription_id").is(subscriptionId)
                        .and("billing_period").is(billingPeriod)
                )
                .withConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM),  // explicit, not global
            ChargeGuard.class
        ));
    }

    public void insertGuard(ChargeGuard guard) {
        cassandraOperations.insert(guard,
            InsertOptions.builder()
                .consistencyLevel(ConsistencyLevel.LOCAL_QUORUM)  // explicit
                .build()
        );
    }
}

Using explicit per-query consistency levels (rather than relying on global configuration) makes the billing code’s consistency requirements self-documenting and resistant to global configuration changes. A future team member who changes the global read consistency to ONE for performance reasons will not accidentally downgrade the billing guard reads, because the guard reads specify their own consistency level explicitly.

Test for Mode 3

@SpringBootTest
@AutoConfigureWireMock(port = 0)
class ConsistencyLevelGuardReadTest {

    @Autowired
    private BillingService billingService;

    @Autowired
    private Session session;  // DataStax Driver session

    @Test
    void one_read_misses_quorum_write_causing_double_charge_with_uuid_key() {
        // Simulate: guard written at LOCAL_QUORUM (acknowledged),
        // but read at ONE routes to a node without the data.
        // With UUID key: two Stripe calls, two charges.
        // This test must run against a real multi-node Cassandra cluster
        // (Testcontainers CassandraContainer does not support multi-node by default).
        // The test verifies the consistency level used by the query:
        assertThat(capturedReadConsistency()).isEqualTo(ConsistencyLevel.ONE);  // buggy
        // And verifies that Stripe was called twice when the guard read returns null
        // due to routing to the lagging replica:
        verify(2, postRequestedFor(urlEqualTo("/v1/charges")));
    }

    @Test
    void quorum_read_always_sees_quorum_write_preventing_double_charge() {
        String stableKey = BillingService.toStableKey("sub_003", "2026-10", 9900);

        stubFor(post(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", equalTo(stableKey))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody("{\"id\":\"ch_A\",\"status\":\"succeeded\"}")));

        billingService.chargeSubscription("sub_003", "2026-10", 9900);

        // LOCAL_QUORUM read guarantees guard is visible after LOCAL_QUORUM write.
        // Guard found on second call — no Stripe call on second invocation.
        billingService.chargeSubscription("sub_003", "2026-10", 9900);

        // Stripe called only once: second call returned guard with COMPLETED status.
        verify(1, postRequestedFor(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", equalTo(stableKey)));
    }
}

How the three modes relate: write ambiguity, LWT PAXOS loss, and consistency mismatch

All three modes share a common structural property: the developer makes an assumption about the state of the Cassandra guard table based on an operation whose outcome is actually uncertain, and then proceeds to generate a new idempotency key on the basis of that assumption.

In Mode 1, the assumption is “WriteTimeoutException means write failed; table state is unchanged.” The reality is “write state is ambiguous; table may or may not contain the guard.”

In Mode 2, the assumption is “LWT WriteTimeoutException means IF NOT EXISTS was never evaluated; the guard was not written.” The reality is “PAXOS commit may have completed before acknowledgment was lost; the guard may have been written.”

In Mode 3, the assumption is “guard read returned null means guard is absent.” The reality is “guard read at ONE does not guarantee visibility of a LOCAL_QUORUM write; null means the queried replica has no guard, but other replicas may.”

In all three cases, the correct response to ambiguity is not to proceed with a new idempotency key. The correct response is to resolve the ambiguity before acting: read back at an appropriate consistency level, retry the same key, or surface the ambiguity to a higher-level reconciliation process. And in all three cases, the use of a stable deterministic idempotency key (rather than UUID.randomUUID()) provides a final layer of defense: if the key is the same on every attempt, Stripe’s deduplication prevents the double charge even when the guard logic fails to prevent the second Stripe call.

Cassandra is a sound choice for billing guard tables at high throughput — it can sustain hundreds of thousands of guard writes per second across a large cluster without the hot-row contention that relational databases encounter when many threads update a single billing period record. But Cassandra’s write semantics are not the same as relational transaction semantics. A developer who builds a billing guard using Cassandra with the mental model of a relational database will encounter write ambiguity, LWT PAXOS timeout uncertainty, and consistency level visibility gaps that do not exist in the relational world. Each of these gaps, when exploited by a concurrent retry, regenerated UUID, or misconfigured consistency level, produces a Stripe double charge.

In the next post, this series continues with Spring Cloud Kubernetes + Stripe: how ConfigMap-driven @RefreshScope restarts clear in-flight billing state cluster-wide, how Kubernetes leader election via Fabric8 Kubernetes client causes idempotency key generation to re-execute on leader re-election, and how rolling deployments create a split-brain window where the old and new pod versions derive different idempotency keys from the same input.