Eclipse Vert.x and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Eclipse Vert.x is a reactive JVM toolkit used in high-throughput billing backends — clustered verticles processing EventBus billing triggers, periodic timers driving monthly charge loops, and reactive Web Client calling Stripe’s REST API directly. Its event-driven, non-blocking model and clustered EventBus create three billing failure modes that are invisible in single-node development and surface only in production under network blips, clustered deployments, or reactive chain retries.

This post covers all three failure modes with Java Vert.x 4.x code, content-hash idempotency keys stable across EventBus retries and Web Client re-subscriptions, a cluster-wide lock pattern using Vert.x SharedData to serialize setPeriodic() billing execution across nodes, pre-flight PostgreSQL checks — and per-billing-period vault keys via a spend-cap proxy as a hard backstop when the pre-flight is bypassed by concurrent cluster execution within Stripe’s transactional window.

Failure mode 1: eventBus.send() reply timeout triggers retry — consumer already called Stripe and created ch_A, new message carries a different idempotency key and Stripe creates ch_B

Vert.x’s clustered EventBus delivers messages across the network using a Hazelcast or Infinispan cluster manager. The send-with-reply pattern — eventBus.send(address, body, replyHandler) — routes the billing message to one consumer, waits for the consumer to call message.reply(), and invokes replyHandler with the response. If no reply arrives within the delivery timeout configured in DeliveryOptions (Vert.x 4 default: 30 seconds), the replyHandler is called with a ReplyException carrying ReplyFailure.TIMEOUT.

The failure mode is straightforward once you know Vert.x’s delivery semantics. A billing verticle receives a charge trigger on the EventBus. The consumer’s handler calls stripe.charges.create() via the Stripe Java SDK — a synchronous HTTP call that takes between 300ms and 8 seconds depending on Stripe’s latency. Stripe creates ch_A and returns the charge ID. The consumer is about to call message.reply(chargeId). At this moment the Vert.x EventBus cluster connection drops — a network blip, a brief Hazelcast member heartbeat timeout, or a rolling deploy that cycles the consumer node. The reply never reaches the sender.

The sender’s replyHandler fires with ReplyException(ReplyFailure.TIMEOUT, "Timed out waiting for reply"). The developer, reasonably interpreting a timeout as a failed delivery, calls eventBus.send() again. The retry message is delivered to the (now-recovered) consumer or to a different consumer instance on another node. The consumer receives a fresh Message object and calls stripe.charges.create() again.

Whether this creates ch_B depends entirely on the idempotency key. If the key was generated at the eventBus.send() call site — a common pattern for correlating the send with the expected reply — the retry call generates a completely new value:

// UNSAFE: idempotency key generated at eventBus.send() call time
public class BillingOrchestrator extends AbstractVerticle {

    private void chargeCutsomer(Customer c) {
        // UUID generated here — different value on every eventBus.send() call including retries
        String correlationId = UUID.randomUUID().toString();

        DeliveryOptions opts = new DeliveryOptions()
            .setSendTimeout(10_000L) // 10 second reply timeout
            .addHeader("correlation_id", correlationId)
            .addHeader("billing_period", c.getBillingPeriod());

        // correlationId embedded in the message — consumer uses it to build the Stripe key:
        // sha256("cust_123:Q3-2026:f47ac10b-58cc-4372-a567-0e02b2c3d479") on first send
        // sha256("cust_123:Q3-2026:550e8400-e29b-41d4-a716-446655440000") on retry send → ch_B
        vertx.eventBus().send("billing.charge", JsonObject.mapFrom(c), opts, reply -> {
            if (reply.failed()) {
                // ReplyException TIMEOUT — consumer may have already created ch_A
                // but we interpret "no reply" as "charge failed" and retry
                if (reply.cause() instanceof ReplyException re
                        && re.failureType() == ReplyFailure.TIMEOUT) {
                    chargeCutsomer(c); // recursive retry — new UUID on each call
                }
            }
        });
    }
}

// UNSAFE: timestamp captured at send time, used in consumer to build the Stripe key
// Different millisecond on every eventBus.send() call
DeliveryOptions opts = new DeliveryOptions()
    .addHeader("send_ts", String.valueOf(System.currentTimeMillis()));

// Consumer side:
vertx.eventBus().consumer("billing.charge", msg -> {
    String customerId  = msg.body().getString("customerId");
    String billingPeriod = msg.headers().get("billing_period");
    String sendTs      = msg.headers().get("send_ts"); // different on retry

    String key = DigestUtils.sha256Hex(
        customerId + ":" + billingPeriod + ":" + sendTs
    ).substring(0, 32);
    // → different key on retry, ch_B created
});

The failure is invisible in tests. In unit tests, the EventBus is typically local (no network, no cluster manager). Local EventBus delivery is synchronous under test schedulers and never times out. In integration tests, the consumer is on the same JVM as the sender; Hazelcast cluster partitions do not occur within a single JVM. The TIMEOUT reply failure — and the retry path it triggers — is exercised only in multi-node staging environments with intentional network partitioning, or in production during rolling deploys when cluster membership changes.

The failure is also insidious because ReplyException(ReplyFailure.TIMEOUT) looks like a delivery failure, not a processing failure. The consumer log shows a completed charge. The sender log shows a timeout and retry. Both sides appear to have done the right thing. The duplicate charge appears in Stripe’s dashboard, not in either service’s error log.

The fix for failure mode 1

The idempotency key must be derived from stable business fields and computed at the consumer side from the message body content — not from any value generated at eventBus.send() call time. The key must produce the same value on the first delivery and on any retry delivery, regardless of when the eventBus.send() call was made or what connection the message traveled over:

import org.apache.commons.codec.digest.DigestUtils;
import io.vertx.core.eventbus.Message;

// Consumer-side billing handler — key derived from message BODY, not from send-time metadata
vertx.eventBus().<JsonObject>consumer("billing.charge", msg -> {
    String customerId    = msg.body().getString("customerId");
    String billingPeriod = msg.body().getString("billingPeriod");
    long   amountCents   = msg.body().getLong("amountCents");
    String stripeCustomer = msg.body().getString("stripeCustomerId");

    // Stable key — same on first delivery and on every retry delivery.
    // Must NOT include: msg.headers().get("correlation_id") (new UUID per send call),
    // msg.headers().get("send_ts") (new timestamp per send call),
    // UUID.randomUUID() here (new per message handler invocation),
    // System.currentTimeMillis() here (different ms per invocation),
    // vertx.getOrCreateContext().deploymentID() (unique per verticle deployment),
    // the Vert.x node ID (UUID per cluster member — different on every cluster node).
    String key = DigestUtils.sha256Hex(
        customerId + ":" + billingPeriod + ":vertx-billing"
    ).substring(0, 32);

    // Pre-flight: claim the billing slot before calling Stripe.
    // ON CONFLICT DO NOTHING returns 0 rows affected if already claimed by the first delivery.
    billingRepo.insertIfAbsent(customerId, billingPeriod, key)
        .onSuccess(rowsAffected -> {
            if (rowsAffected == 0) {
                // Already billed — reply success so sender stops retrying
                msg.reply(new JsonObject().put("status", "already_billed"));
                return;
            }
            // Charge Stripe
            ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(amountCents)
                .setCurrency("usd")
                .setCustomer(stripeCustomer)
                .putIdempotencyKey(key)
                .build();
            try {
                Charge charge = Charge.create(params);
                billingRepo.updateChargeId(key, charge.getId());
                msg.reply(new JsonObject().put("chargeId", charge.getId()));
            } catch (StripeException e) {
                msg.fail(500, e.getMessage());
            }
        })
        .onFailure(err -> msg.fail(500, err.getMessage()));
});

-- billing_records schema
CREATE TABLE billing_records (
    customer_id      TEXT NOT NULL,
    billing_period   TEXT NOT NULL,
    idempotency_key  TEXT NOT NULL,
    status           TEXT NOT NULL DEFAULT 'pending',
    charge_id        TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT billing_records_pk PRIMARY KEY (idempotency_key),
    CONSTRAINT billing_records_uq UNIQUE (customer_id, billing_period)
);

-- Pre-flight insert
INSERT INTO billing_records (customer_id, billing_period, idempotency_key)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id, billing_period) DO NOTHING;

The pre-flight check closes the failure regardless of which delivery attempt reaches the consumer first. If the first delivery claimed the slot and called Stripe (creating ch_A), the retry delivery finds the slot already claimed, returns zero rows affected, and sends a success reply without calling Stripe again. The sender receives the success reply and stops retrying.

Failure mode 2: vertx.setPeriodic() is not cluster-aware — fires independently on every Vert.x cluster node, billing loop runs N times simultaneously in an N-node cluster

This is the most operationally dangerous of the three failure modes because it scales with cluster size. An engineer adds one billing verticle, deploys it horizontally for throughput, and creates N simultaneous billing runs for every single customer — one per cluster node.

vertx.setTimer() and vertx.setPeriodic() are per-instance, per-event-loop local timers. The Vert.x documentation states this clearly, but the implication for clustered deployments is easy to miss if you’ve worked with clustered job schedulers like Quartz (which serialize trigger acquisition across nodes via a database row lock) or Spring’s @Scheduled with ShedLock (which uses a distributed lock table to ensure only one node fires the scheduled method).

In a Vert.x cluster with three nodes, each running a billing verticle that calls vertx.setPeriodic(MONTHLY_BILLING_INTERVAL_MS, handler), three independent timers fire at (approximately) the same wall-clock time — within NTP drift of each other, typically a few milliseconds. Each node runs the complete billing loop for the full customer list. Each calls stripe.charges.create() for every customer:

// UNSAFE: setPeriodic() in a clustered Vert.x deployment
// Fires on EVERY node in the cluster — not cluster-coordinated
public class BillingVerticle extends AbstractVerticle {

    @Override
    public void start() {
        long intervalMs = Duration.ofDays(30).toMillis();

        // This timer runs on THIS node's event loop.
        // In a 3-node cluster: 3 timers, 3 concurrent billing runs, 3× Stripe charges per customer.
        vertx.setPeriodic(intervalMs, timerId -> {
            runBillingCycle();
        });
    }

    private void runBillingCycle() {
        // All 3 nodes reach this code simultaneously
        customerRepo.findDueForBilling()
            .onSuccess(customers -> {
                for (Customer c : customers) {
                    // Instant.now() at timer callback time — different nanoseconds on each node's JVM
                    // but close enough that if this were the only source of difference, Stripe's
                    // 24-hour window would catch same-key concurrent calls within the window.
                    // The real problem: Vert.x node ID is a UUID generated per cluster member.
                    String nodeId = vertx.getOrCreateContext().owner().toString();

                    // sha256("cust_123:Q3-2026:node-A-uuid") on node A
                    // sha256("cust_123:Q3-2026:node-B-uuid") on node B → ch_B
                    // sha256("cust_123:Q3-2026:node-C-uuid") on node C → ch_C
                    String key = DigestUtils.sha256Hex(
                        c.getId() + ":" + c.getBillingPeriod() + ":" + nodeId
                    ).substring(0, 32);

                    chargeSynchronously(c, key); // 3 different keys, 3 Stripe charges per customer
                }
            });
    }
}

Even with a content-hash key (no node-local value), three concurrent calls to stripe.charges.create() with the same key within Stripe’s 24-hour idempotency window cause the second and third calls to block at Stripe until the first completes, then return the cached response. This looks safe — and it is, within the 24-hour window. The problem surfaces during routine operations that extend the window: a scheduled maintenance that keeps two nodes down for 25 hours, a rolling deploy that takes longer than expected, or simply a configuration drift where one node’s NTP clock is ahead and fires the timer one billing-period-interval late. With a content-hash key and a 30-day billing interval, Node C fires 30 days and 26 hours after Node A’s first charge — Stripe’s idempotency cache has expired and ch_B is created.

The correct pattern is to restrict billing timer execution to a single elected node using Vert.x SharedData’s cluster-wide lock. Every node competes for the lock at timer callback time. Only the node that acquires the lock runs the billing cycle. Other nodes acquire the lock, find the billing period already recorded in a shared distributed map, release the lock, and return without charging:

public class BillingVerticle extends AbstractVerticle {

    private static final String BILLING_LOCK     = "billing.monthly.lock";
    private static final String BILLING_STATE_MAP = "billing.state";

    @Override
    public void start() {
        long intervalMs = 60_000L; // Check every minute; only one node runs the billing cycle per period

        vertx.setPeriodic(intervalMs, timerId -> {
            String billingPeriod = currentBillingPeriod(); // e.g. "2026-08"

            // Compete for cluster-wide lock — only one node wins per lock acquisition window.
            // getLockWithTimeout: timeout shorter than the billing period to avoid starvation.
            vertx.sharedData().getLockWithTimeout(BILLING_LOCK, 5_000L)
                .onSuccess(lock -> {
                    // Check if this billing period has already been processed
                    vertx.sharedData().<String, String>getClusterWideMap(BILLING_STATE_MAP)
                        .onSuccess(map -> {
                            map.get(billingPeriod)
                                .onSuccess(existingRun -> {
                                    if (existingRun != null) {
                                        // Already processed by another node — release lock and return
                                        lock.release();
                                        return;
                                    }
                                    // Claim this billing period before running the cycle
                                    String runId = "billing-" + billingPeriod;
                                    map.put(billingPeriod, runId)
                                        .onSuccess(v -> {
                                            lock.release(); // Release before long-running billing cycle
                                            runBillingCycle(billingPeriod);
                                        })
                                        .onFailure(err -> lock.release());
                                })
                                .onFailure(err -> lock.release());
                        })
                        .onFailure(err -> lock.release());
                })
                .onFailure(err -> {
                    // Lock acquisition timed out — another node already holds it, billing in progress
                    // Do nothing; the lock holder will run this period's billing cycle
                });
        });
    }

    private void runBillingCycle(String billingPeriod) {
        customerRepo.findDueForBilling(billingPeriod)
            .onSuccess(customers -> {
                for (Customer c : customers) {
                    // Stable content-hash key — same on every node, same on every timer tick
                    String key = DigestUtils.sha256Hex(
                        c.getId() + ":" + billingPeriod + ":vertx-billing"
                    ).substring(0, 32);

                    // Pre-flight check before calling Stripe
                    billingRepo.insertIfAbsent(c.getId(), billingPeriod, key)
                        .onSuccess(rowsAffected -> {
                            if (rowsAffected > 0) {
                                chargeCustomer(c, key);
                            }
                        });
                }
            });
    }

    private String currentBillingPeriod() {
        return java.time.YearMonth.now().toString(); // "2026-08"
    }
}

The SharedData lock serializes billing period entry. A node that acquires the lock, writes the billing period to the cluster-wide map, and releases the lock before running the cycle prevents all other nodes from entering the billing loop for that period. The pre-flight database check provides a second layer: if two nodes somehow acquire the lock in the same millisecond window (e.g., due to a Hazelcast member join event that causes a lock expiry), the UNIQUE (customer_id, billing_period) constraint in PostgreSQL ensures only one INSERT ... ON CONFLICT DO NOTHING wins per customer and billing period.

Failure mode 3: Vert.x Web Client rxSend().retry() re-subscribes the RxJava chain — currentTimeMillis() or UUID.randomUUID() inside Single.defer() produces a different value on each retry attempt

Vert.x’s reactive extensions (io.vertx.rxjava3) wrap the async Web Client in RxJava 3 observables. A billing service calling Stripe’s REST API directly (without the Stripe Java SDK — common in Vert.x codebases that want non-blocking HTTP) constructs the request in a reactive chain and uses .retry(N) or .retryWhen() for resilience. The retry semantics interact destructively with any value captured inside the reactive chain that changes between subscriptions.

The key property of RxJava’s cold observables: Single.defer() defers the creation of the upstream sequence until subscription time. Each subscription (including each retry subscription triggered by .retry()) evaluates the defer() factory function from scratch. Any value captured inside defer() is recomputed on every retry. System.currentTimeMillis() returns a different millisecond. UUID.randomUUID() returns a new UUID. The Stripe idempotency key derived from these values is different on retry 1 and retry 2.

import io.reactivex.rxjava3.core.Single;
import io.vertx.rxjava3.ext.web.client.WebClient;
import io.vertx.rxjava3.ext.web.client.HttpResponse;
import io.vertx.rxjava3.core.buffer.Buffer;

// UNSAFE: currentTimeMillis() inside Single.defer() — different on every retry subscription
public Single<String> chargeCustomer(WebClient client, Customer c) {
    return Single.defer(() -> {
        // Re-evaluated on every retry subscription
        long requestTs = System.currentTimeMillis();

        // sha256("cust_123:Q3-2026:1748736004321") on attempt 1
        // sha256("cust_123:Q3-2026:1748736009456") on retry 2 → ch_B
        String idempotencyKey = DigestUtils.sha256Hex(
            c.getId() + ":" + c.getBillingPeriod() + ":" + requestTs
        ).substring(0, 32);

        JsonObject body = new JsonObject()
            .put("amount", c.getAmountCents())
            .put("currency", "usd")
            .put("customer", c.getStripeCustomerId());

        return client.post(443, "api.stripe.com", "/v1/charges")
            .ssl(true)
            .putHeader("Authorization", "Bearer " + vaultKeyFor(c.getBillingPeriod()))
            .putHeader("Idempotency-Key", idempotencyKey) // different on retry
            .rxSendJsonObject(body);
    })
    .retry(3)  // retries re-subscribe to defer() — new requestTs, new key each time
    .map(resp -> resp.bodyAsJsonObject().getString("id"));
}

The failure mode is: attempt 1 sends the charge request to Stripe. Stripe receives it, processes it, creates ch_A, and begins returning the HTTP response. A network interruption cuts the connection before the full HTTP response arrives at the Vert.x Web Client. The RxJava observable emits a ConnectException or SocketTimeoutException. .retry(3) catches the exception and re-subscribes to the defer() block. System.currentTimeMillis() returns a new millisecond value. The retry sends the charge request to Stripe with a different idempotency key. Stripe has no record of this new key; it creates ch_B.

Stripe’s 24-hour idempotency cache only helps if the same key is used on both attempts. A different key on retry is indistinguishable to Stripe from a new, intentional charge for the same customer in the same billing period. There is no server-side protection; only the client-side key discipline prevents the duplicate.

The same failure occurs with UUID.randomUUID() inside defer(), with Instant.now() formatted as an ISO string, with a monotonic counter read from an AtomicLong that increments on each request creation (the counter increments on every new subscription), and with HttpRequest.hashCode() or System.identityHashCode(requestOptions) if the HttpRequest object is created inside the defer() factory.

The fix for failure mode 3

Compute the idempotency key outside the reactive chain, before subscribing. The key is derived from stable business fields and passed into defer() as a captured-final variable. Because it is captured before the first subscription, it is identical on every retry subscription:

import io.reactivex.rxjava3.core.Single;
import io.vertx.rxjava3.ext.web.client.WebClient;

public Single<String> chargeCustomer(WebClient client, Customer c) {
    // Compute the key BEFORE entering the reactive chain.
    // This value is captured once and reused on every retry subscription.
    // Must NOT include: System.currentTimeMillis() (re-evaluated per subscription),
    // UUID.randomUUID() (new UUID per subscription), Instant.now() (different timestamp per retry),
    // HttpRequest.hashCode() of a request object created inside defer() (new object per subscription),
    // vertx.getOrCreateContext().deploymentID() (unique per verticle deployment),
    // the Vert.x node ID (UUID per cluster member — differs across retries if load-balanced).
    final String idempotencyKey = DigestUtils.sha256Hex(
        c.getId() + ":" + c.getBillingPeriod() + ":vertx-billing"
    ).substring(0, 32);

    return Single.defer(() -> {
        // idempotencyKey is a captured-final from the outer scope.
        // Same value on attempt 1, retry 2, and retry 3.
        JsonObject body = new JsonObject()
            .put("amount", c.getAmountCents())
            .put("currency", "usd")
            .put("customer", c.getStripeCustomerId());

        return client.post(443, "api.stripe.com", "/v1/charges")
            .ssl(true)
            .putHeader("Authorization", "Bearer " + vaultKeyFor(c.getBillingPeriod()))
            .putHeader("Idempotency-Key", idempotencyKey) // stable across all retries
            .rxSendJsonObject(body);
    })
    .retry(3)
    .map(resp -> resp.bodyAsJsonObject().getString("id"));
}

// Pre-flight check before calling chargeCustomer() in the billing loop
public Single<Void> billingLoop(List<Customer> customers) {
    return Observable.fromIterable(customers)
        .concatMapSingle(c -> {
            String key = DigestUtils.sha256Hex(
                c.getId() + ":" + c.getBillingPeriod() + ":vertx-billing"
            ).substring(0, 32);

            // Pre-flight: returns 0 rows if already billed, 1 if newly claimed
            return billingRepo.rxInsertIfAbsent(c.getId(), c.getBillingPeriod(), key)
                .flatMap(rowsAffected -> {
                    if (rowsAffected == 0) {
                        return Single.just("already_billed");
                    }
                    return chargeCustomer(webClient, c);
                });
        })
        .ignoreElements()
        .toSingle(() -> null);
}

The pre-flight insert uses ON CONFLICT (customer_id, billing_period) DO NOTHING. The first delivery or first retry attempt claims the row. Every subsequent retry finds the row and returns zero rows affected. The billing loop skips the Stripe call for customers whose pre-flight row is already present. The reconciliation job handles the edge case where a customer has a pending pre-flight row but no charge_id — meaning the insert committed but the Stripe call was lost before returning — by re-issuing stripe.charges.create() with the same content-hash key.

Vault key governance: the per-billing-period spend cap

All three failure modes are addressed at the application layer by content-hash idempotency keys and pre-flight database checks. A vault key per billing period provides a hard backstop when the application-layer fix fails to prevent all duplicate paths — concurrent cluster execution within PostgreSQL’s millisecond-level isolation window, a Stripe 24-hour cache expiry on a delayed retry, or a pre-flight row written but Stripe call not yet issued when a second node arrives.

A vault key is a scoped proxy key issued to the billing system for a specific vendor (Stripe) and a specific billing period (e.g., August 2026). The proxy enforces a spend cap of expected_monthly_revenue × 1.10: 10% above the expected billing total. Any Stripe charge that would push the period total above the cap is rejected at the proxy layer before reaching Stripe, with an audit log entry recording the attempt.

// Issue a vault key for this billing period before starting the charge loop
// Cap = expected total × 1.10 — hard backstop against double-charge scenarios
// that slip through the application-layer idempotency check

POST https://proxy.keybrake.com/keys
{
  "vendor": "stripe",
  "billing_period": "2026-08",
  "daily_usd_cap": null,
  "period_usd_cap": 54890,  // expectedTotal × 1.10 = 49900 × 1.10
  "allowed_endpoints": ["/v1/charges", "/v1/payment_intents"],
  "expires_at": "2026-09-01T00:00:00Z"
}

// Response
{
  "vault_key": "vk_live_xxxxxxxxxx",
  "cap_usd": 54890,
  "vendor": "stripe"
}

// Use vault_key as the Authorization bearer in all Stripe calls for this period
// If a duplicate charge would exceed the cap, the proxy rejects it:
// HTTP 429 Too Many Requests — "spend cap exceeded for billing_period 2026-08"

The spend cap catches the failure when it bypasses the pre-flight check. Two cluster nodes race on the pre-flight insert within the same PostgreSQL transaction isolation window (sub-millisecond): both see zero rows, both INSERT, one wins the UNIQUE constraint, the other gets a constraint violation. The constraint violation is caught and the second node skips Stripe — but if the second node’s error handling is wrong (e.g., treats constraint violation as a retryable error and retries the entire billing customer loop from the top), it may re-attempt stripe.charges.create(). The vault key cap stops the second charge at the proxy layer.

Gap analysis

WorkerVerticle billing and the concurrent handler problem

Vert.x’s standard event loop verticles process messages from a single event loop thread — no concurrent handler invocations within one consumer. WorkerVerticle changes this: worker verticles are executed on Vert.x’s internal worker thread pool, and multiple messages can be dispatched concurrently to the same worker verticle from the EventBus. A billing verticle running as a WorkerVerticle with setInstances(4) creates four concurrent worker threads, each potentially processing a billing message for the same customer in the same billing period simultaneously. Four concurrent calls to stripe.charges.create() with the same key are safe within Stripe’s 24-hour window (concurrent requests with the same key are serialized by Stripe). But the pre-flight PostgreSQL UNIQUE constraint ensures only one worker claims the billing slot — the three that lose the race return zero rows affected and skip Stripe without a Stripe call at all.

Reactive chain composition: Single.zip() retries both subscriptions

A billing implementation that processes two customers in parallel using Single.zip(chargeCustomer(c1), chargeCustomer(c2)).retry(1) has a critical property: .retry(1) on a zip re-subscribes to both upstream singles when either one fails. If chargeCustomer(c1) succeeds (creating ch_A₁) and chargeCustomer(c2) fails with a SocketTimeoutException, the retry re-executes both chargeCustomer(c1) and chargeCustomer(c2). With a content-hash key and a pre-flight check, the retry of chargeCustomer(c1) hits the pre-flight row and exits without calling Stripe — safe. Without a pre-flight check, retry-of-c1 with the same key within 24 hours returns Stripe’s cached ch_A₁ response — also safe. The failure is that .retry(1) on zip is unintuitive: it looks like a retry of the failed operation, but it retries the entire composed sequence. Use chargeCustomer(c1).retry(1) and chargeCustomer(c2).retry(1) independently before composing with zip, so retries are scoped to individual customer charges.

EventBus at-most-once delivery and lost billing triggers

Vert.x’s clustered EventBus does NOT guarantee message delivery. Unlike Kafka, Pulsar, or a JMS broker, the clustered EventBus has no persistence layer. Messages sent during a network partition are lost — they are not redelivered after the partition heals. A billing trigger sent via eventBus.send() during a Hazelcast cluster split will silently disappear. The consumer never receives it. The pre-flight check cannot save a trigger it never sees.

For guaranteed billing trigger delivery, use a persistent scheduler (Quartz with JDBCJobStore, jobrunr with a database backend) or a persistent message queue (Kafka, RabbitMQ) as the billing trigger source. Use the EventBus for notifications and status updates — operations where at-most-once semantics are acceptable — not for guaranteed billing trigger delivery.

io.vertx.rxjava3.ext.web.client.WebClient retry vs. io.vertx.ext.web.client.WebClient callback

The RxJava3 Web Client (io.vertx.rxjava3.ext.web.client.WebClient) wraps the async API in cold observables with .rxSend() / .rxSendJsonObject(). Cold observables re-execute the HTTP request on each subscription, including each retry subscription. The plain async Web Client (io.vertx.ext.web.client.WebClient) callback-style API — client.post(...).sendJsonObject(body, handler) — executes the request once when sendJsonObject is called. Retries in the callback style are explicit imperative code: the handler is called, checks the error, and explicitly calls sendJsonObject again if retrying. This makes it straightforward to compute the idempotency key once before the first call and pass the same value to all retry calls, because the retry site is explicit code rather than an implicit re-subscription. If you are building a new billing service in Vert.x and have a choice between the RxJava3 wrapper and the callback API, the callback API’s explicit retry structure makes idempotency key discipline easier to enforce in code review.

Vert.x OpenAPI Router billing endpoints and HTTP client retries

A billing endpoint served by Vert.x’s OpenAPI Router handler and called by an HTTP client with automatic retries (Resilience4j, Feign with retry interceptor, OkHttp with connection timeout retry) experiences all the same failure modes as the EventBus retry scenario. The HTTP client sends a charge request. Vert.x handles it, routes to the billing handler, the handler calls stripe.charges.create() (ch_A), and then the HTTP response is lost on the network. The HTTP client times out, retries the request to Vert.x. The Vert.x handler receives a second request. If the billing handler generates the idempotency key from any request-time value — request.headers().get("X-Request-ID") if the client generates a new request ID on retry, Instant.now() captured at handler entry time, or RoutingContext.request().absoluteURI()` hash including a timestamp query parameter — the second request generates a different key and Stripe creates ch_B. The same content-hash + pre-flight solution applies: derive the key from the request body’s stable business fields (customer ID and billing period), not from any per-request metadata.

Summary

All three failure modes share the same root cause: idempotency key material generated at execution time (per send, per timer tick, per reactive subscription) rather than from stable business identity. The same two-layer fix closes all three: a content-hash key derived from customerId + billingPeriod + service-namespace that produces the same value on every execution, and a pre-flight INSERT ... ON CONFLICT DO NOTHING into a table with a UNIQUE (customer_id, billing_period) constraint that closes the gap for any retry or concurrent execution that bypasses Stripe’s 24-hour idempotency cache — including the failure modes that slip through within the window, like two cluster nodes racing on the same pre-flight insert within a single transaction isolation boundary.

The vault key spend cap is the operational backstop: a scoped proxy key with a per-billing-period cap set at expected_total × 1.10 stops any duplicate charge that bypasses both the idempotency key and the pre-flight check at the proxy layer, before the second request reaches Stripe’s servers. It does not replace the application-layer fix — it provides a hard financial limit when the application-layer fix has an edge case the pre-flight check cannot catch within its transaction isolation window.