Netty Pipeline Handlers and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Netty’s ChannelPipeline processes inbound messages through a chain of ChannelInboundHandlerAdapter instances, each passing a decoded message to the next via ctx.fireChannelRead(). Three billing failure modes are specific to this pipeline architecture: an exceptionCaught() handler that retries by re-firing the billing message through the pipeline (triggering a new UUID.randomUUID() in the billing handler); a billing system that triggers charges in channelActive() but loses its billing-state guard when the channel closes and reconnects; and a periodic billing scheduler registered via channel.eventLoop().scheduleAtFixedRate() that runs independently on every channel in the pipeline with no cross-channel or cross-replica coordination.

This post covers all three failure modes with Java Netty 4.1.x code, content-hash idempotency keys stable across exceptionCaught() retries and channelActive() reconnects, PostgreSQL advisory locks and Kubernetes leader election for scheduleAtFixedRate() serialization across channels and replicas, pre-flight ON CONFLICT DO NOTHING checks — and per-billing-period vault keys via a spend-cap proxy as a hard backstop. For related failure modes arising from the Vert.x event bus and periodic timer in similar reactive networking stacks, see the Eclipse Vert.x and Stripe Integration post.

Failure mode 1: exceptionCaught() retries by re-firing the BillingRequest through the pipeline — channelRead0() in the billing handler calls UUID.randomUUID() on every invocation — the retry produces a different UUID — Stripe creates ch_B for a customer whose charge (ch_A) completed before the downstream exception was thrown

The canonical pattern for resilient billing in a Netty pipeline is to attach an exceptionCaught() handler near the tail of the pipeline. When a downstream handler throws — an audit-log database write failure, a JSON serialization error in the response-encoding handler, a downstream HTTP call timeout for a notification service — the exception propagates up the inbound handler chain until exceptionCaught() catches it. The retry pattern that looks safest but is not is to re-fire the original billing request from the beginning of the pipeline:

// Netty 4.1.x — pipeline retry from exceptionCaught()
// Registered last in the pipeline; receives exceptions from all upstream handlers.
@ChannelHandler.Sharable
public class BillingRetryHandler extends ChannelInboundHandlerAdapter {

    private static final AttributeKey<BillingRequest> PENDING_REQ =
        AttributeKey.valueOf("pendingBillingRequest");
    private static final AttributeKey<Integer> ATTEMPT_COUNT =
        AttributeKey.valueOf("billingAttemptCount");

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
        int attempts = ctx.channel().attr(ATTEMPT_COUNT).get() == null
            ? 0 : ctx.channel().attr(ATTEMPT_COUNT).get();

        if (attempts < 3) {
            ctx.channel().attr(ATTEMPT_COUNT).set(attempts + 1);
            BillingRequest req = ctx.channel().attr(PENDING_REQ).get();

            if (req != null) {
                // Re-fire through the pipeline head — channelRead0() will be called again
                // in BillingHandler. This looks like a clean retry but is not safe
                // if BillingHandler regenerates its idempotency key on each invocation.
                ctx.pipeline().fireChannelRead(req);
            }
        } else {
            ctx.fireExceptionCaught(cause);
        }
    }
}

// BillingHandler — first handler in the pipeline after the decoder.
public class BillingHandler extends SimpleChannelInboundHandler<BillingRequest> {

    private final StripeClient stripeClient;
    private final BillingRepo billingRepo;

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, BillingRequest req) {
        ctx.channel().attr(BillingRetryHandler.PENDING_REQ).set(req);

        // UNSAFE: UUID.randomUUID() is generated fresh on every channelRead0() call.
        // Original invocation: key = "3f7a9b2c-1d4e-4f6a-8b5c-9d0e1f2a3b4c"
        // Retry invocation:    key = "9b2c1d4e-4f6a-8b5c-9d0e-1f2a3b4c5d6e" (different)
        String idempotencyKey = UUID.randomUUID().toString();

        // Store in channel attr so the exception handler can read it for logging.
        // But it is NOT stable across retries — a new UUID is generated here, not re-read.
        stripeClient.createChargeAsync(
            req.getCustomerId(),
            req.getAmount(),
            req.getBillingPeriod(),
            idempotencyKey
        ).addListener(future -> {
            if (future.isSuccess()) {
                ChargeResult result = (ChargeResult) future.getNow();
                // Hand off to AuditHandler downstream
                ctx.fireChannelRead(new BillingComplete(req, result.getChargeId()));
            } else {
                ctx.fireExceptionCaught(future.cause());
            }
        });
    }
}

// AuditHandler — downstream in the pipeline; throws if the DB write fails.
public class AuditHandler extends SimpleChannelInboundHandler<BillingComplete> {
    @Override
    protected void channelRead0(ChannelHandlerContext ctx, BillingComplete event) {
        // Stripe call already completed — ch_A exists in Stripe.
        // If this DB write fails (connection pool exhausted, disk full, deadlock),
        // the exception propagates up the pipeline to BillingRetryHandler.
        auditRepo.recordCharge(event.getChargeId(), event.getCustomerId());
        ctx.fireChannelRead(event);
    }
}

The failure scenario: BillingHandler.channelRead0() receives a BillingRequest for customer cust_123, generates UUID = "3f7a9b2c...", and issues the Stripe API call. Stripe processes the request, creates ch_A (ch_A9f3b), and returns a 201 response. BillingHandler fires BillingComplete to the next handler in the pipeline. AuditHandler attempts to write the charge record to the database. The database connection is at its pool limit; the write blocks for 30 seconds and then throws SQLTimeoutException. The exception propagates up the inbound chain to BillingRetryHandler.exceptionCaught(). The retry fires req back through the pipeline head. BillingHandler.channelRead0() is invoked again, this time generating UUID = "9b2c1d4e...". Stripe has never seen this key. ch_B is created. Customer 123 is charged twice for the same billing period.

The failure is subtle because the exception is thrown by AuditHandler, not by BillingHandler. In unit tests, BillingHandler is tested in isolation with a mock AuditHandler that never throws. The retry path in BillingRetryHandler is tested separately with a mock that returns successfully on the second call. The interaction — that a post-Stripe exception causes a pre-Stripe retry — is only visible in an end-to-end integration test that actually runs Stripe API calls against a test clock and checks idempotency-key uniqueness.

The fix for failure mode 1

The idempotency key must be derived from stable business fields, not from per-invocation runtime state. A content-hash key using customerId, billingPeriod, and a static salt produces the same value on the original invocation and on every retry — whether the retry is triggered by exceptionCaught() re-firing the message or by any other mechanism in the pipeline. The pre-flight database check separates the Stripe call from the audit write: the billing record is claimed in the database before calling Stripe, so a retry that finds an existing record skips the Stripe call entirely:

import org.apache.commons.codec.digest.DigestUtils;

// BillingHandler — safe content-hash key version
public class BillingHandler extends SimpleChannelInboundHandler<BillingRequest> {

    private final StripeClient stripeClient;
    private final BillingRepo billingRepo;

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, BillingRequest req) {
        // Content-hash key — identical on original call and every retry.
        // Must NOT include: UUID.randomUUID() (new per invocation),
        //   System.currentTimeMillis() (different ms per invocation),
        //   Channel.id().asShortText() (new channel on reconnect),
        //   Thread.currentThread().getId() (different EventLoop worker),
        //   req.getCorrelationId() if set at request-receipt time (changes on retry),
        //   ctx.channel().attr(ATTEMPT_COUNT).get() (increments on each retry).
        String idempotencyKey = DigestUtils.sha256Hex(
            req.getCustomerId() + ":" + req.getBillingPeriod() + ":netty-billing"
        ).substring(0, 32);

        // Pre-flight: claim the billing slot in the database before calling Stripe.
        // INSERT ... ON CONFLICT DO NOTHING returns 0 rows if the record already exists
        // (original Stripe call succeeded and committed before the downstream exception).
        // Returns 1 row if this is the first attempt — proceed to call Stripe.
        // The pre-flight is idempotent: same key on retry finds the same record.
        int inserted = billingRepo.insertIfAbsent(
            req.getCustomerId(), req.getBillingPeriod(), idempotencyKey
        );

        if (inserted == 0) {
            // Billing record already committed — original Stripe call succeeded.
            // Retrieve the existing charge ID and propagate BillingComplete downstream.
            String existingChargeId = billingRepo.findChargeId(
                req.getCustomerId(), req.getBillingPeriod()
            );
            ctx.fireChannelRead(new BillingComplete(req, existingChargeId));
            return;
        }

        // No existing record — proceed with Stripe call.
        stripeClient.createChargeAsync(
            req.getCustomerId(), req.getAmount(), req.getBillingPeriod(), idempotencyKey
        ).addListener(future -> {
            if (future.isSuccess()) {
                ChargeResult result = (ChargeResult) future.getNow();
                // Update billing record status from 'pending' to 'completed'.
                billingRepo.markCompleted(
                    req.getCustomerId(), req.getBillingPeriod(), result.getChargeId()
                );
                ctx.fireChannelRead(new BillingComplete(req, result.getChargeId()));
            } else {
                ctx.fireExceptionCaught(future.cause());
            }
        });
    }
}

-- Pre-flight table — insert-before-call ordering
CREATE TABLE billing_records (
    customer_id      TEXT NOT NULL,
    billing_period   TEXT NOT NULL,
    idempotency_key  TEXT NOT NULL,
    status           TEXT NOT NULL DEFAULT 'pending',
    charge_id        TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT billing_records_pk PRIMARY KEY (idempotency_key),
    CONSTRAINT billing_records_uq UNIQUE (customer_id, billing_period)
);

-- Returns 1 row inserted on first call, 0 on retry (record already exists).
INSERT INTO billing_records (customer_id, billing_period, idempotency_key)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id, billing_period) DO NOTHING
RETURNING 1;

With this fix, the AuditHandler database write failure no longer causes a duplicate charge. The retry from BillingRetryHandler calls channelRead0() again with the same content-hash key. The pre-flight check finds the existing pending record (inserted by the first call), returns 0 rows, and the handler retrieves the charge ID that was stored when the original Stripe call succeeded. BillingComplete is fired downstream with the original charge ID. The audit write can be retried independently without re-invoking Stripe.

The fix also handles the case where the original Stripe call never completed — if BillingHandler crashed before createChargeAsync() returned, the pre-flight record is in pending state with no charge_id. The retry proceeds to call Stripe with the same content-hash key. Stripe creates ch_A (the first call never reached Stripe, or was abandoned mid-flight). The billing record is updated with the returned charge ID.

Failure mode 2: channelActive() triggers billing when a connection is established — billing state stored only in channel.attr() is lost when the channel closes — a ChannelFutureListener reconnects via Bootstrap.connect() — the new Channel has no inherited attributes — channelActive() fires on the new channel and triggers billing again — System.currentTimeMillis() at connection time produces ch_B

Metered-API billing systems sometimes trigger a Stripe charge when a client connects: the connection is the billable event (a session start, an agent activation, a WebSocket stream open). The natural Netty hook is channelActive(), which fires when the channel is fully established and connected. The billing handler creates a Stripe charge in channelActive() and stores a “billing-initiated” marker in channel.attr() to prevent double-billing on handler re-entry:

// Billing handler for metered-connection billing (charge on connect)
public class ConnectionBillingHandler extends ChannelInboundHandlerAdapter {

    private static final AttributeKey<Boolean> BILLING_INITIATED =
        AttributeKey.valueOf("billingInitiated");

    private final StripeClient stripeClient;
    private final Bootstrap bootstrap;        // used for reconnect
    private final SocketAddress remoteAddress;

    @Override
    public void channelActive(ChannelHandlerContext ctx) {
        Boolean alreadyBilled = ctx.channel().attr(BILLING_INITIATED).get();

        if (Boolean.TRUE.equals(alreadyBilled)) {
            // Guard condition — prevent double-billing within same channel lifecycle.
            // PROBLEM: channel.attr() is NOT inherited by the new Channel created on reconnect.
            // A new Channel always starts with an empty AttributeMap.
            ctx.fireChannelActive();
            return;
        }

        // UNSAFE: timestamp captured at channelActive() time.
        // Original connection at T=1000ms: key = sha256("cust_123:2026-08:session:1000")
        // Reconnect at T=62000ms:          key = sha256("cust_123:2026-08:session:62000") — ch_B
        long connectedAt = System.currentTimeMillis();
        String customerId = ctx.channel().attr(AttributeKey.valueOf("customerId")).get();
        String billingPeriod = getCurrentBillingPeriod(); // e.g. "2026-08"
        String idempotencyKey = DigestUtils.sha256Hex(
            customerId + ":" + billingPeriod + ":session:" + connectedAt
        ).substring(0, 32);

        // Mark billing initiated on THIS channel's attr.
        // Lost when this channel closes — new Channel on reconnect has no BILLING_INITIATED attr.
        ctx.channel().attr(BILLING_INITIATED).set(Boolean.TRUE);

        stripeClient.createChargeAsync(customerId, SESSION_FEE, idempotencyKey)
            .addListener(future -> {
                if (future.isSuccess()) {
                    ctx.fireChannelActive();
                } else {
                    ctx.close();
                }
            });
    }

    @Override
    public void channelInactive(ChannelHandlerContext ctx) {
        // Auto-reconnect: create a new Channel via Bootstrap.connect().
        // The new Channel has an empty AttributeMap — BILLING_INITIATED is not inherited.
        // channelActive() will fire on the new Channel and find alreadyBilled = null.
        ctx.channel().eventLoop().schedule(() -> {
            bootstrap.connect(remoteAddress).addListener((ChannelFuture future) -> {
                if (!future.isSuccess()) {
                    future.channel().close();
                }
            });
        }, 5, TimeUnit.SECONDS);
        ctx.fireChannelInactive();
    }
}

The failure scenario: a metered billing agent connects at T=0ms. channelActive() fires, generates connectedAt = 1748700000000, builds key sha256("cust_123:2026-08:session:1748700000000")[:32], and calls Stripe. ch_A (ch_A9f3b) is created. BILLING_INITIATED is set to true on the channel’s AttributeMap. The connection drops 60 seconds later due to an intermediate NAT gateway timeout. channelInactive() fires. The event loop schedules a reconnect in 5 seconds. Bootstrap.connect() creates a new NioSocketChannel. The new channel’s AttributeMap is empty — Netty creates a new DefaultAttributeMap for each channel. channelActive() fires on the new channel. alreadyBilled is null. connectedAt is now 1748700065000. sha256("cust_123:2026-08:session:1748700065000")[:32] ≠ sha256("cust_123:2026-08:session:1748700000000")[:32]. Stripe has never seen the new key. ch_B is created. The customer is charged twice for August 2026 within 65 seconds of their first connection.

The guard condition that looks protective — if (Boolean.TRUE.equals(alreadyBilled)) — is only protective within a single channel’s lifetime. It cannot survive the channel close because Netty does not copy attributes from a closed channel to a new channel. The only memory that survives channel lifecycle transitions is external storage: a database, a distributed cache, or a persistent KV store. Channel attributes are ephemeral by design.

The fix for failure mode 2

The billing guard must be in external storage, not in channel attributes. The content-hash key — derived from stable business fields without the per-connection timestamp — is the same on every connection for the same customer in the same billing period. The pre-flight database check before the Stripe call is authoritative across the full billing period regardless of how many times the channel connects and reconnects:

// Safe channelActive() billing — external guard, stable key
public class ConnectionBillingHandler extends ChannelInboundHandlerAdapter {

    private final StripeClient stripeClient;
    private final BillingRepo billingRepo;
    private final Bootstrap bootstrap;
    private final SocketAddress remoteAddress;

    @Override
    public void channelActive(ChannelHandlerContext ctx) {
        String customerId = ctx.channel().attr(AttributeKey.valueOf("customerId")).get();
        String billingPeriod = getCurrentBillingPeriod();

        // Content-hash key — identical on every connection for the same customer + period.
        // Must NOT include: System.currentTimeMillis() (changes on reconnect),
        //   Channel.id().asShortText() (new channel on reconnect),
        //   channel.attr(BILLING_INITIATED) as source of truth (not inherited on reconnect),
        //   connection sequence counter stored in channel.attr() (same problem),
        //   HOSTNAME env var (changes across Kubernetes pods restarting).
        String idempotencyKey = DigestUtils.sha256Hex(
            customerId + ":" + billingPeriod + ":netty-session"
        ).substring(0, 32);

        // Pre-flight: claim the billing slot before calling Stripe.
        // Returns 0 rows if billing already completed for this customer + period.
        // Survives channel lifecycle — lives in the database, not in channel.attr().
        int inserted = billingRepo.insertIfAbsent(customerId, billingPeriod, idempotencyKey);

        if (inserted == 0) {
            // Already billed this period — reconnect is safe, skip Stripe.
            ctx.fireChannelActive();
            return;
        }

        stripeClient.createChargeAsync(customerId, SESSION_FEE, idempotencyKey)
            .addListener(future -> {
                if (future.isSuccess()) {
                    ChargeResult result = (ChargeResult) future.getNow();
                    billingRepo.markCompleted(customerId, billingPeriod, result.getChargeId());
                    ctx.fireChannelActive();
                } else {
                    // Mark failed; next reconnect's pre-flight will find 'pending' and retry Stripe.
                    ctx.close();
                }
            });
    }

    @Override
    public void channelInactive(ChannelHandlerContext ctx) {
        // Reconnect as before — channelActive() on the new Channel will find the billing
        // record in the database and skip Stripe if billing already completed for this period.
        ctx.channel().eventLoop().schedule(() -> {
            bootstrap.connect(remoteAddress);
        }, 5, TimeUnit.SECONDS);
        ctx.fireChannelInactive();
    }
}

The reconnect cycle is now safe for any number of reconnects within the same billing period. The first channelActive() call inserts the billing record and calls Stripe. Every subsequent channelActive() for the same customer in the same billing period finds the existing record and calls ctx.fireChannelActive() without touching Stripe. The guard is durable across pod restarts, channel lifecycle events, and bootstrap reconnects because it lives in the database, not in the channel’s AttributeMap.

The pending state handles the crash-between-insert-and-Stripe case: if the channel closes between the pre-flight insert and the Stripe call, the record is in pending state. On reconnect, insertIfAbsent returns 0 rows (record exists), and the handler reads the charge_id field. If charge_id is null (pending), the handler proceeds to call Stripe with the same content-hash key — this is intentional, as the original call may never have reached Stripe. Stripe’s own idempotency cache will return the result of ch_A if it arrived, or create ch_A fresh if it did not.

Failure mode 3: channel.eventLoop().scheduleAtFixedRate() registers an independent billing timer on every channel in the NioEventLoopGroup — any per-channel value in the idempotency key produces a distinct key per channel — in a multi-channel billing server, N concurrent billing loops each charge the same customer — ch_A through ch_N per billing period

channel.eventLoop().scheduleAtFixedRate() schedules a recurring task on the EventLoop thread that owns the channel. This is a common pattern for heartbeats and keepalives in Netty pipelines. Applied to billing, it looks like a clean way to run periodic billing without a separate scheduling framework:

// Billing handler that schedules periodic billing via the channel's EventLoop.
// Registered in the pipeline for every incoming customer connection.
public class PeriodicBillingHandler extends ChannelInboundHandlerAdapter {

    private final StripeClient stripeClient;
    private final BillingRepo billingRepo;

    @Override
    public void channelActive(ChannelHandlerContext ctx) {
        String customerId = ctx.channel().attr(AttributeKey.valueOf("customerId")).get();

        // Schedule monthly billing — runs on THIS channel's EventLoop thread.
        // Fires every 30 days starting immediately.
        ctx.channel().eventLoop().scheduleAtFixedRate(() -> {

            String billingPeriod = getCurrentBillingPeriod(); // e.g. "2026-08"

            // UNSAFE: Channel.id().asShortText() is unique per Channel instance.
            // In a multi-channel billing server handling 100 concurrent customer connections,
            // each channel has a different Channel.id() — producing 100 distinct keys
            // for the same customer in the same billing period if a customer appears
            // on multiple channels (e.g., agent re-connected, multiple active sessions).
            //
            // Even with a single connection per customer, in a 3-replica Kubernetes Deployment
            // each replica has its own NioEventLoopGroup and its own Channel.id() namespace.
            // A customer routed to replica A and then to replica B after pod churn gets
            // two channels with two different Channel.id() values.
            String channelId = ctx.channel().id().asShortText(); // e.g. "f3a21b9c"
            String idempotencyKey = DigestUtils.sha256Hex(
                customerId + ":" + billingPeriod + ":periodic:" + channelId
            ).substring(0, 32);

            // Replica A: channelId="f3a21b9c" → sha256("cust_123:2026-08:periodic:f3a21b9c")
            // Replica B: channelId="91bc7de1" → sha256("cust_123:2026-08:periodic:91bc7de1")
            // Both fire within milliseconds of each other → ch_A and ch_B.
            stripeClient.createChargeAsync(
                customerId, MONTHLY_FEE, billingPeriod, idempotencyKey
            );
        }, 0, 30, TimeUnit.DAYS);

        ctx.fireChannelActive();
    }
}

The failure scenarios compound quickly. A single server with 100 concurrent customer connections produces 100 independent billing timers via scheduleAtFixedRate(), one per channel. If a customer connects once, that’s one timer per customer — the channel-ID key is unique to their session, not shared, so the single-connection case works. The multi-session failure occurs when a customer connects from two different clients simultaneously (agent re-connect after pod eviction, two active browser tabs, a parallel agent invocation): two channels, two different Channel.id() values, two timers firing on the same billing period, two calls to Stripe with two different keys — ch_A and ch_B.

The multi-replica failure is worse and invisible in single-node testing. In a Kubernetes Deployment with replicas: 3, a billing server running scheduleAtFixedRate() in channelActive() creates a timer on whichever replica the customer’s connection lands. If the customer’s connection migrates across pod restarts — their agent reconnects to a new pod after the original pod is evicted — the new pod creates a new channel with a new Channel.id() and a new timer. If the billing period timer fires on both the old pod (before its timer was garbage-collected) and the new pod, two charges are sent to Stripe with different keys. The failure rate equals the fraction of customers whose connections migrate across pod boundaries within the same billing period.

The failure is also triggered by per-thread identifiers when the billing task is registered without a channel reference. Thread.currentThread().getId() in the idempotency key changes across EventLoop worker threads if the task is rescheduled or if the channel migrates between threads (rare but possible with EventLoopGroup resharding). Instant.now() at schedule-registration time produces a unique timestamp per channel activation. ctx.channel().remoteAddress().toString() changes when the client reconnects from a different port. Each of these patterns produces a unique key per billing invocation and defeats Stripe’s idempotency cache.

The fix for failure mode 3

Periodic billing must not be driven from scheduleAtFixedRate() registered inside channelActive(). Each channel activation creates a new timer with no coordination with timers from other channels, other EventLoop threads, or other Kubernetes replicas. The fix is two-part: use a content-hash key to close the “multiple keys for same billing intent” gap, and move the scheduling to a single coordinated trigger using a PostgreSQL advisory lock or Kubernetes leader election.

// Part 1: Content-hash key — same for all channels and replicas for the same customer + period
// Replace the per-channel key with a stable business-field hash:
String idempotencyKey = DigestUtils.sha256Hex(
    customerId + ":" + billingPeriod + ":netty-billing"
).substring(0, 32);
// Must NOT include: Channel.id().asShortText(), Channel.id().asLongText(),
//   Thread.currentThread().getId(), Instant.now() at schedule-registration time,
//   ctx.channel().remoteAddress().toString(), hostname from InetAddress.getLocalHost(),
//   per-replica UUID generated at JVM startup,
//   EventLoop index from ((SingleThreadEventLoop)ctx.channel().eventLoop()).getIndex().

// Part 2: PostgreSQL advisory lock to serialize billing across all channels and replicas.
// One EventLoop thread acquires the lock; others return immediately without charging.
String idempotencyKey = DigestUtils.sha256Hex(
    customerId + ":" + billingPeriod + ":netty-billing"
).substring(0, 32);

// pg_try_advisory_lock uses a stable hash of customerId + billingPeriod.
// Returns true only on ONE replica/channel at a time.
// The lock is automatically released on PostgreSQL session end (connection close).
try (Connection conn = dataSource.getConnection()) {
    boolean lockAcquired = conn.prepareStatement(
        "SELECT pg_try_advisory_lock(hashtext($1 || ':' || $2))"
    )
    .bind(customerId, billingPeriod)
    .executeQuery()
    .getBoolean(1);

    if (!lockAcquired) {
        // Another channel or replica is handling billing for this customer + period.
        // Return without calling Stripe.
        return;
    }

    // Lock acquired — proceed with pre-flight check and Stripe call.
    int inserted = billingRepo.insertIfAbsent(customerId, billingPeriod, idempotencyKey);
    if (inserted == 0) {
        conn.prepareStatement("SELECT pg_advisory_unlock(hashtext($1 || ':' || $2))")
            .bind(customerId, billingPeriod).execute();
        return;
    }

    stripeClient.createChargeAsync(customerId, MONTHLY_FEE, billingPeriod, idempotencyKey)
        .addListener(future -> {
            if (future.isSuccess()) {
                ChargeResult result = (ChargeResult) future.getNow();
                billingRepo.markCompleted(customerId, billingPeriod, result.getChargeId());
            }
            // Release lock after billing attempt (success or failure).
            // Failure leaves the record in 'pending' — the next scheduled run will retry.
            try {
                conn.prepareStatement("SELECT pg_advisory_unlock(hashtext($1 || ':' || $2))")
                    .bind(customerId, billingPeriod).execute();
            } catch (SQLException ignored) {}
        });
}

For teams running Kubernetes, Kubernetes leader election (via coordination.k8s.io/v1 Lease objects) provides a Netty-native alternative to database advisory locks. One pod holds the billing lease; all others skip their scheduleAtFixedRate() callbacks immediately. The lease holder runs the billing loop for all customers, using the content-hash key for idempotency. On leader handoff, the new leader’s billing run finds existing pre-flight records for customers already charged in the current period and skips them.

The most operationally simple fix is to decouple periodic billing from the Netty ChannelPipeline entirely. A separate Quartz or Spring Scheduler job — running once per period per cluster, not per channel — processes all due customers using content-hash keys and pre-flight checks. The Netty pipeline handles the session management and metered-event billing; the scheduled job handles the periodic billing. This separation means the billing logic is not affected by channel lifecycle events, reconnects, or replica count.

Gap analysis: additional Netty billing failure modes

@ChannelHandler.Sharable with per-request instance fields. Marking a billing handler @ChannelHandler.Sharable allows the same handler instance to be added to multiple channels’ pipelines. If the handler stores per-request state in instance fields (private String currentKey, private int retryCount), two channels assigned to different EventLoop threads can invoke channelRead() concurrently. The currentKey set by channel A’s thread can be overwritten by channel B’s thread before channel A reads it, causing channel A to issue its Stripe call with channel B’s idempotency key. The result is a charge misattribution: customer A is charged with customer B’s idempotency key, which may match a key already used for customer B, causing Stripe to return customer B’s existing charge for customer A’s billing. Fix: keep @ChannelHandler.Sharable handlers stateless; derive all per-request values from the message argument to channelRead(), not from instance fields.

ChannelGroup fan-out with partial failure retry. DefaultChannelGroup.writeAndFlush(msg) sends a billing trigger to all connected channels. A ChannelGroupFutureListener receives the composite future and retries channels whose writes failed. If the billing handler on the receiving channel generates UUID.randomUUID() inside channelRead(), a write-failure-triggered retry on a single channel causes that channel to issue a second Stripe call with a new key. The content-hash fix from failure mode 1 applies here: a stable key means the retry’s Stripe call uses the same key as the original, and Stripe’s cache returns the existing result.

Netty HTTP client retry pipeline for outbound Stripe calls. If the billing system uses a Netty-based HTTP client (Reactor Netty’s HttpClient) with built-in retry on connection failure, the retry re-establishes the TCP connection and re-sends the billing request. If the Stripe-Idempotency-Key header was generated from a per-connection timestamp or the client channel’s Channel.id(), the retry sends a different key on the new connection. The content-hash fix applies: generate the idempotency key before initiating the Reactor Netty request and pass it as a fixed string in the header, never re-computing it inside the retry chain.

Netty WriteBufferWaterMark backpressure causing channelWritabilityChanged() billing trigger. A billing trigger that fires in channelWritabilityChanged() when the channel becomes writable again (after a isWritable() == false backpressure event) can fire multiple times per billing period if the channel oscillates between writable and non-writable states during a high-load burst. Each channelWritabilityChanged() invocation that fires the billing trigger must check the pre-flight database record before calling Stripe. A System.currentTimeMillis() key generated inside the writability callback produces a different key on each oscillation. The content-hash fix and pre-flight check close this.

The governance backstop: per-billing-period vault keys

Content-hash keys and pre-flight database checks are application-layer fixes. They fail if a future code change re-introduces a per-invocation value in the key derivation path (a UUID.randomUUID() added for debugging, a timestamp added for log correlation), if a new developer writes a second billing code path that does not follow the content-hash convention, or if a retry is introduced at a layer above the billing handler that bypasses the pre-flight check.

A vault key with a per-billing-period spend cap is the backstop that survives these application-layer failures. Instead of giving your billing agent the live Stripe secret key, issue it a vault key via Keybrake:

POST /vault/keys
{
  "vendor": "stripe",
  "daily_usd_cap": 10000,
  "cap_basis": "billing_period",
  "billing_period": "2026-08",
  "allowed_endpoints": ["/v1/charges", "/v1/payment_intents"],
  "expires_at": "2026-09-01T00:00:00Z"
}

The vault key is scoped to the August 2026 billing period. The daily_usd_cap is set to your expected total times 1.10 — enough headroom to complete the legitimate billing run, not enough to absorb a full double-charge across all customers. If exceptionCaught() retry loops or channelActive() reconnects trigger duplicate charges that bypass the application-layer guards, the proxy blocks the excess charges at the vault layer and sends an alert. The pipeline bug surfaces as an alert, not as money leaving your account.

Vault keys also give you the audit log that answers the post-incident question: did the Stripe call go out twice, or did the retry fail before reaching the network? The audit table records every proxied request with the idempotency key, the response status, and the wall-clock timestamp — allowing a precise reconstruction of which handler invocations reached Stripe and which were blocked by the pre-flight check or the cap.

Summary

The pattern across all three failure modes is the same as in Akka Streams, Akka Typed EventSourcedBehavior, and Eclipse Vert.x: the framework gives you retry, reconnect, and scheduling primitives that are natural for I/O handling but dangerous for billing, because they re-invoke your billing code at exactly the moment when a per-invocation value in the idempotency key produces a different Stripe key. A content-hash key derived from business fields only — sha256(customerId + ":" + billingPeriod + ":netty-billing")[:32] — is identical across all invocations for the same billing intent, regardless of channel lifecycle, reconnect count, or replica assignment. The pre-flight ON CONFLICT DO NOTHING database check is the authoritative gate. The vault key is the backstop when both fail.