gRPC and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

gRPC’s interceptor model, bidirectional streaming, and built-in hedging policy create three Stripe billing failure modes that each produce a duplicate charge from a different idempotency key: a ClientInterceptor that constructs fresh Metadata with a new UUID.randomUUID() on each retry attempt; a bidirectional streaming billing RPC that reconnects after a stream failure and replays unACK’d messages with a new call-level UUID embedded in the idempotency key; and a ServiceConfig hedging policy that sends concurrent copies of the billing RPC to different backend pods where each handler generates its own UUID on entry.

This post covers all three failure modes with gRPC Java 1.x code, content-hash idempotency keys stable across interceptor retries, stream reconnects, and hedged RPC copies, pre-flight PostgreSQL ON CONFLICT DO NOTHING checks — and per-billing-period vault keys via a spend-cap proxy as a hard backstop. For related failure modes in reactive streaming runtimes, see the Netty Pipeline Handlers and Stripe Integration post. For connection-pool and timeout-retry failure modes in HTTP/2 environments, see the Akka HTTP and Stripe Integration post.

Failure mode 1: ClientInterceptor retries a failed billing RPC by re-invoking the stub — UUID.randomUUID() in the interceptor’s start() method generates a new Metadata key per attempt — the original RPC created ch_A before the server returned UNAVAILABLE — the retry creates ch_B

The idiomatic gRPC Java pattern for client-side retry without enabling the built-in ServiceConfig retry policy is a ClientInterceptor that wraps each ClientCall and re-starts it on retryable status codes. The interceptor overrides interceptCall(), returns a ForwardingClientCall that catches onClose() with a retryable status, and re-invokes the original stub method to create a new ClientCall. The billing idempotency key is typically set in the Metadata passed to ClientCall.start():

// gRPC Java 1.x — billing with ClientInterceptor retry
// UNSAFE: UUID generated inside the interceptor's start() on each retry
import io.grpc.*;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicInteger;

public class RetryInterceptor implements ClientInterceptor {

    private static final Metadata.Key<String> IDEMPOTENCY_KEY =
        Metadata.Key.of("idempotency-key", Metadata.ASCII_STRING_MARSHALLER);

    private final int maxAttempts;

    public RetryInterceptor(int maxAttempts) { this.maxAttempts = maxAttempts; }

    @Override
    public <Req, Resp> ClientCall<Req, Resp> interceptCall(
            MethodDescriptor<Req, Resp> method,
            CallOptions callOptions,
            Channel next) {

        return new ForwardingClientCall.SimpleForwardingClientCall<>(next.newCall(method, callOptions)) {

            final AtomicInteger attempts = new AtomicInteger(0);
            Listener<Resp> responseListener;
            Metadata headers;
            Req requestMessage;

            @Override
            public void start(Listener<Resp> responseListener, Metadata headers) {
                this.responseListener = responseListener;
                this.headers = headers;

                // UNSAFE: UUID generated inside start() on every invocation.
                // Original call (attempt 0): key = "a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d"
                // Retry    (attempt 1): key = "f9e8d7c6-b5a4-4f3e-2d1c-0b9a8f7e6d5c" (different)
                // If the server processed the billing RPC and Stripe created ch_A before
                // returning UNAVAILABLE (network partition mid-response, server GC pause
                // exceeding the deadline, or rolling restart killing the handler goroutine
                // after Stripe responded 201 but before the server sent the gRPC trailer),
                // the retry's new UUID causes Stripe to create ch_B.
                headers.put(IDEMPOTENCY_KEY, UUID.randomUUID().toString());
                super.start(new ForwardingClientCallListener<>(responseListener) {
                    @Override
                    public void onClose(Status status, Metadata trailers) {
                        if (isRetryable(status) && attempts.incrementAndGet() < maxAttempts) {
                            // Re-invoke — generates a new UUID in headers for the next attempt.
                            retry(method, callOptions, next);
                        } else {
                            super.onClose(status, trailers);
                        }
                    }
                }, headers);
            }

            @Override
            public void sendMessage(Req message) {
                this.requestMessage = message;
                super.sendMessage(message);
            }

            private void retry(
                    MethodDescriptor<Req, Resp> method,
                    CallOptions callOptions,
                    Channel next) {
                // Creates a new ClientCall — start() will be called again — new UUID generated.
                ClientCall<Req, Resp> retryCall = next.newCall(method, callOptions);
                retryCall.start(responseListener, headers);  // UUID set fresh in start()
                retryCall.request(1);
                retryCall.sendMessage(requestMessage);
                retryCall.halfClose();
            }
        };
    }

    private boolean isRetryable(Status status) {
        return status.getCode() == Status.Code.UNAVAILABLE
            || status.getCode() == Status.Code.INTERNAL
            || status.getCode() == Status.Code.RESOURCE_EXHAUSTED;
    }
}

The billing stub is configured with this interceptor:

// Billing stub wired with the retry interceptor
BillingServiceGrpc.BillingServiceBlockingStub stub =
    BillingServiceGrpc.newBlockingStub(channel)
        .withInterceptors(new RetryInterceptor(3));

// Call — the interceptor wraps the call and retries on UNAVAILABLE/INTERNAL
ChargeBillingResponse response = stub.chargeBilling(
    ChargeBillingRequest.newBuilder()
        .setCustomerId("cust_123")
        .setAmount(9900)
        .setCurrency("usd")
        .setBillingPeriod("2026-08")
        .build()
);

The failure scenario: the agent calls stub.chargeBilling() for cust_123, August 2026, $99.00. The interceptor’s start() runs. UUID.randomUUID() returns "a1b2c3d4...". The billing RPC is sent to the server. The server’s BillingServiceImpl.chargeBilling() calls Stripe with this key. Stripe begins processing and creates ch_A. Before Stripe returns the 201 response, the gRPC server pod is killed by a Kubernetes rolling restart. The server sends a GOAWAY frame and closes the connection. The gRPC channel delivers Status.UNAVAILABLE to the interceptor’s onClose(). The interceptor calls retry(). start() is invoked on the new ClientCall. A new UUID.randomUUID() returns "f9e8d7c6...". The headers object (which was shared and mutated) now contains the new UUID. The retry RPC reaches a live pod. That pod calls Stripe with "f9e8d7c6...". Stripe has never seen this key. ch_B is created. Customer 123 is charged $99 twice for August 2026.

The failure is additionally triggered by any situation where the server processes the Stripe call but fails to return the gRPC response trailer: a JVM out-of-memory error after calling Stripe but before serializing the response, a network partition that closes the TCP connection after Stripe’s 201 but before the server flushes the HTTP/2 DATA frame, or a deadline exceeded on the server side where the server killed the handler before it could return the response to the client. All of these cause the client to receive a non-OK status with no response — the interceptor interprets them as retryable — and a new UUID is generated on the next attempt.

The fix for failure mode 1

The idempotency key must be computed once at the call site — before the interceptor is invoked — and placed in the Metadata that the interceptor receives. The interceptor reads the caller-supplied key rather than generating its own. If the call site does not supply a key, the interceptor derives one from the request message using a content-hash:

// Safe approach: caller computes the stable key before calling the stub
import java.security.MessageDigest;
import java.nio.charset.StandardCharsets;

public static String stableKey(String customerId, String billingPeriod) {
    try {
        MessageDigest digest = MessageDigest.getInstance("SHA-256");
        byte[] hash = digest.digest(
            (customerId + ":" + billingPeriod + ":grpc-billing")
                .getBytes(StandardCharsets.UTF_8));
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < 16; i++) {
            sb.append(String.format("%02x", hash[i]));
        }
        return sb.toString();  // 32-char hex
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

// Compute once per billing intent — before the stub call
String idempotencyKey = stableKey("cust_123", "2026-08");

Metadata metadata = new Metadata();
metadata.put(IDEMPOTENCY_KEY, idempotencyKey);

// Pass metadata to the stub call via stub.withOption() or a MetadataUtils interceptor
BillingServiceGrpc.BillingServiceBlockingStub stub =
    BillingServiceGrpc.newBlockingStub(channel)
        .withInterceptors(MetadataUtils.newAttachHeadersInterceptor(metadata));

ChargeBillingResponse response = stub.chargeBilling(request);

// Safe RetryInterceptor: reads the caller-supplied key, never generates its own
public class SafeRetryInterceptor implements ClientInterceptor {

    private static final Metadata.Key<String> IDEMPOTENCY_KEY =
        Metadata.Key.of("idempotency-key", Metadata.ASCII_STRING_MARSHALLER);

    @Override
    public <Req, Resp> ClientCall<Req, Resp> interceptCall(
            MethodDescriptor<Req, Resp> method,
            CallOptions callOptions,
            Channel next) {

        return new ForwardingClientCall.SimpleForwardingClientCall<>(next.newCall(method, callOptions)) {

            Listener<Resp> responseListener;
            Metadata headers;
            Req requestMessage;
            String callerKey;  // extracted from caller-supplied metadata, never changed

            @Override
            public void start(Listener<Resp> responseListener, Metadata headers) {
                this.responseListener = responseListener;
                this.headers = headers;

                // Read caller-supplied key — do not overwrite.
                // Must NOT call UUID.randomUUID() here or set any per-invocation value.
                this.callerKey = headers.get(IDEMPOTENCY_KEY);
                if (this.callerKey == null) {
                    // No key supplied — interceptor should reject the call, not generate one.
                    // Generating here would still produce a stable-per-call key if the call
                    // object is re-used, but the safest path is to require callers to supply it.
                    throw new IllegalStateException(
                        "Billing RPC requires Idempotency-Key metadata. " +
                        "Compute stableKey(customerId, billingPeriod) before calling the stub.");
                }

                super.start(new ForwardingClientCallListener<>(responseListener) {
                    @Override
                    public void onClose(Status status, Metadata trailers) {
                        if (isRetryable(status)) {
                            // headers already contains the caller's stable key — no new UUID.
                            retry(method, callOptions, next);
                        } else {
                            super.onClose(status, trailers);
                        }
                    }
                }, headers);
            }

            @Override
            public void sendMessage(Req message) {
                this.requestMessage = message;
                super.sendMessage(message);
            }

            private void retry(MethodDescriptor<Req, Resp> method, CallOptions callOptions, Channel next) {
                ClientCall<Req, Resp> retryCall = next.newCall(method, callOptions);
                // headers still contains the original caller-supplied key — safe to re-use.
                retryCall.start(responseListener, headers);
                retryCall.request(1);
                retryCall.sendMessage(requestMessage);
                retryCall.halfClose();
            }
        };
    }
}

-- Pre-flight table on the server side (the gRPC server checks this before calling Stripe)
CREATE TABLE billing_records (
    customer_id      TEXT        NOT NULL,
    billing_period   TEXT        NOT NULL,
    idempotency_key  TEXT        NOT NULL,
    status           TEXT        NOT NULL DEFAULT 'pending',
    charge_id        TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT billing_records_pk PRIMARY KEY (idempotency_key),
    CONSTRAINT billing_records_uq UNIQUE (customer_id, billing_period)
);

-- Server-side pre-flight: claim the billing slot before calling Stripe
-- Returns the row if inserted (first caller), empty if record already existed
INSERT INTO billing_records (customer_id, billing_period, idempotency_key)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id, billing_period) DO NOTHING
RETURNING idempotency_key;

With the stable content-hash key passed through by the caller, every retry — regardless of which pod handles it, how many UNAVAILABLE responses the interceptor saw, or how many ClientCall.start() invocations occurred — sends the same "a1b2c3d4..." key to the server. The server’s BillingServiceImpl reads the key from Context.current().call().headers().get(IDEMPOTENCY_KEY) (via a server-side ServerInterceptor that populates the Context) and uses it in both the pre-flight database claim and the Stripe call. If ch_A was created by the first attempt before the pod restarted, the retry’s Stripe call returns the cached ch_A response — no second charge.

Failure mode 2: bidirectional streaming billing RPC reconnects after stream failure — replayed messages carry a new stream-level call ID embedded in the idempotency key — ch_B created for customers already charged on the first stream before ACK

A common pattern for high-throughput billing systems on gRPC is a bidirectional streaming RPC: the client sends BillingCommand messages for each customer, and the server sends back BillingResult acknowledgements as charges complete. The client maintains an in-flight queue of unACK’d commands and replays them on reconnect. The problem arises when the idempotency key for each billing command is derived from any attribute of the ClientCall object or the gRPC stream context that changes when the stream is re-established:

// gRPC Java 1.x — bidirectional streaming billing
// UNSAFE: idempotency key includes streamId derived from ClientCall identity
import io.grpc.stub.StreamObserver;

public class BillingStreamClient {

    private static final Metadata.Key<String> IDEMPOTENCY_KEY =
        Metadata.Key.of("idempotency-key", Metadata.ASCII_STRING_MARSHALLER);

    private final BillingServiceGrpc.BillingServiceStub asyncStub;
    private final Queue<BillingCommand> pendingCommands = new ConcurrentLinkedQueue<>();
    private StreamObserver<BillingCommand> requestObserver;
    private volatile String currentStreamId;   // set from ClientCall at stream establishment

    public void connect() {
        // UNSAFE: System.identityHashCode() of the StreamObserver changes on reconnect.
        // The new stream allocates a new object with a different identity hash code.
        // Any per-stream value (call object address, connection sequence counter,
        // Metadata.get("x-grpc-stream-id") set by the server in the initial response,
        // System.currentTimeMillis() at connect time) produces a different streamId.
        currentStreamId = Integer.toHexString(System.identityHashCode(this)) +
                          "-" + System.currentTimeMillis();

        requestObserver = asyncStub.billingStream(new StreamObserver<BillingResult>() {
            @Override
            public void onNext(BillingResult result) {
                // ACK received — remove from pending queue
                pendingCommands.removeIf(cmd -> cmd.getCommandId().equals(result.getCommandId()));
            }

            @Override
            public void onError(Throwable t) {
                // Stream failed — reconnect and replay pending (unACK'd) commands
                scheduleReconnect();
            }

            @Override
            public void onCompleted() {
                scheduleReconnect();
            }
        });

        // Replay any pending commands (unACK'd from prior stream)
        for (BillingCommand cmd : pendingCommands) {
            // UNSAFE: the command was built with the old streamId — but replay wraps
            // it in a new Metadata header with the new currentStreamId.
            requestObserver.onNext(wrapWithStreamContext(cmd));
        }
    }

    public void sendBillingCommand(String customerId, String billingPeriod, long amount) {
        // UNSAFE: idempotency key includes currentStreamId — changes on reconnect.
        // Stream 1 (currentStreamId="a3f7b21c-1748700000000"):
        //   key = sha256("cust_123:2026-08:a3f7b21c-1748700000000")[:32]
        // Stream 2 (currentStreamId="b4e8c32d-1748700065000"):
        //   key = sha256("cust_123:2026-08:b4e8c32d-1748700065000")[:32]  <— different
        String idempotencyKey = sha256(customerId + ":" + billingPeriod + ":" + currentStreamId);

        BillingCommand cmd = BillingCommand.newBuilder()
            .setCommandId(UUID.randomUUID().toString())
            .setCustomerId(customerId)
            .setBillingPeriod(billingPeriod)
            .setAmount(amount)
            .setIdempotencyKey(idempotencyKey)
            .build();

        pendingCommands.add(cmd);
        requestObserver.onNext(cmd);
    }

    private BillingCommand wrapWithStreamContext(BillingCommand cmd) {
        // Rebuild with new streamId — overwrites the original idempotency key from stream 1.
        return cmd.toBuilder()
            .setIdempotencyKey(sha256(cmd.getCustomerId() + ":" +
                               cmd.getBillingPeriod() + ":" + currentStreamId))
            .build();
    }
}

The failure scenario: the client connects and establishes stream 1. currentStreamId = "a3f7b21c-1748700000000". Billing commands for customers 1–50 are sent. The server processes them sequentially. Customers 1–40 complete; their BillingResult ACKs are returned and those commands are removed from pendingCommands. Customers 41–50 are still being processed by the server — ch_41A through ch_50A have been created by Stripe but the BillingResult responses are buffered in the server’s HTTP/2 send buffer when the network partition occurs. The client receives an onError(StatusRuntimeException: UNAVAILABLE). Stream 1 is terminated. The client calls connect() again. currentStreamId = "b4e8c32d-1748700065000". pendingCommands still contains customers 41–50 (their ACKs never arrived). wrapWithStreamContext() rebuilds each command with the new currentStreamId. The new idempotency keys are different from stream 1’s keys. The server receives the replayed commands. Its BillingServiceImpl calls Stripe with the new keys. Stripe has never seen sha256("cust_41:2026-08:b4e8c32d-1748700065000")[:32]. ch_41B through ch_50B are created. Customers 41–50 are charged twice for August 2026.

The same failure occurs when any per-stream attribute ends up in the idempotency key: the gRPC call object’s identity hash code (System.identityHashCode(requestObserver)), a stream sequence counter that resets to 0 on reconnect, the server-assigned stream ID from the HTTP/2 layer (RST_STREAM closes even-numbered client-initiated streams; the reconnect starts a new stream with a higher stream ID), or a timestamp recorded at connect() time. The key does not even need to explicitly embed the stream ID — a System.currentTimeMillis() called at stream establishment produces a value that differs by the reconnect delay, which is always non-zero.

The fix for failure mode 2

The idempotency key must be derived from stable business fields only — the customer ID, billing period, and a static salt — and computed once per billing intent when the command is first created. The key is stored with the command in the pending queue and never recomputed on replay. The stream context, stream ID, connection timestamp, and all other per-stream attributes are excluded:

// Safe bidirectional streaming client — key computed once per billing intent, never recomputed
public class SafeBillingStreamClient {

    private final BillingServiceGrpc.BillingServiceStub asyncStub;
    private final Queue<BillingCommand> pendingCommands = new ConcurrentLinkedQueue<>();
    private StreamObserver<BillingCommand> requestObserver;

    public void connect() {
        requestObserver = asyncStub.billingStream(new StreamObserver<BillingResult>() {
            @Override
            public void onNext(BillingResult result) {
                pendingCommands.removeIf(cmd -> cmd.getCommandId().equals(result.getCommandId()));
            }

            @Override
            public void onError(Throwable t) { scheduleReconnect(); }

            @Override
            public void onCompleted() { scheduleReconnect(); }
        });

        // Replay pending commands — send them UNCHANGED, with the original idempotency key.
        // Do NOT recompute keys or wrap with any stream context.
        for (BillingCommand cmd : pendingCommands) {
            requestObserver.onNext(cmd);  // same command, same key as stream 1
        }
    }

    public void sendBillingCommand(String customerId, String billingPeriod, long amount) {
        // Content-hash key derived from business fields only.
        // Must NOT include: currentStreamId, System.identityHashCode(anything),
        //   System.currentTimeMillis() at command-creation time (changes per call),
        //   UUID.randomUUID() (different per call), connection sequence counter,
        //   HTTP/2 stream ID, server-assigned call ID from response headers.
        String idempotencyKey = stableKey(customerId, billingPeriod);  // sha256(cust:period:grpc-billing)[:32]

        BillingCommand cmd = BillingCommand.newBuilder()
            .setCommandId(UUID.randomUUID().toString())  // correlation ID only — NOT the idempotency key
            .setCustomerId(customerId)
            .setBillingPeriod(billingPeriod)
            .setAmount(amount)
            .setIdempotencyKey(idempotencyKey)  // stable across all stream reconnects
            .build();

        pendingCommands.add(cmd);
        requestObserver.onNext(cmd);
    }
}

// Safe server-side BillingServiceImpl — uses the client-supplied idempotency key
// with a pre-flight DB guard before calling Stripe
@Override
public StreamObserver<BillingCommand> billingStream(StreamObserver<BillingResult> responseObserver) {
    return new StreamObserver<BillingCommand>() {
        @Override
        public void onNext(BillingCommand cmd) {
            String customerId    = cmd.getCustomerId();
            String billingPeriod = cmd.getBillingPeriod();
            String idempotencyKey = cmd.getIdempotencyKey();

            // Pre-flight: claim the billing slot before calling Stripe.
            // If the client replays cmd after a reconnect and the server already
            // processed cmd on stream 1 (charge created, record in DB), the
            // ON CONFLICT DO NOTHING means we skip Stripe and return the cached result.
            boolean inserted = billingRepo.insertIfAbsent(customerId, billingPeriod, idempotencyKey);
            if (!inserted) {
                // Already processed — return cached charge ID without calling Stripe.
                String chargeId = billingRepo.findChargeId(customerId, billingPeriod);
                if (chargeId != null) {
                    responseObserver.onNext(BillingResult.newBuilder()
                        .setCommandId(cmd.getCommandId())
                        .setChargeId(chargeId)
                        .setStatus("deduplicated")
                        .build());
                    return;
                }
                // Record exists but no charge ID yet — concurrent in-flight on another stream.
                // Return ALREADY_EXISTS to let the client retry the poll.
                responseObserver.onError(
                    Status.ALREADY_EXISTS.withDescription("Billing in progress").asRuntimeException());
                return;
            }

            // First caller for this billing intent — proceed with Stripe.
            try {
                String chargeId = stripeClient.createCharge(
                    customerId, cmd.getAmount(), cmd.getCurrency(), billingPeriod, idempotencyKey);
                billingRepo.markCompleted(customerId, billingPeriod, chargeId);
                responseObserver.onNext(BillingResult.newBuilder()
                    .setCommandId(cmd.getCommandId())
                    .setChargeId(chargeId)
                    .setStatus("success")
                    .build());
            } catch (Exception e) {
                billingRepo.markFailed(customerId, billingPeriod);
                responseObserver.onError(Status.INTERNAL.withDescription(e.getMessage()).asRuntimeException());
            }
        }

        @Override public void onError(Throwable t) { /* log */ }
        @Override public void onCompleted() { responseObserver.onCompleted(); }
    };
}

The content-hash key is identical whether the command is sent on stream 1, stream 2, or stream 100. If the server processed the command on stream 1 and created ch_A before the network partition closed the stream, the replayed command on stream 2 hits the pre-flight check, finds the existing billing record, and returns the cached ch_A charge ID — no second Stripe call. The UNIQUE (customer_id, billing_period) constraint is the authoritative gate that survives stream reconnects, server restarts, and even horizontal scaling to multiple billing server pods.

Failure mode 3: ServiceConfig hedging policy sends concurrent copies of the billing RPC to different backend pods — each pod’s billing handler calls UUID.randomUUID() on entry — Stripe sees two requests with different keys — ch_A from pod A and ch_B from pod B are both created

gRPC’s built-in hedging policy (part of the ServiceConfig JSON configuration) sends a second copy of an RPC to a different backend when the first attempt does not respond within hedgingDelay. Unlike retry (which is sequential — one attempt fails before the next starts), hedging is concurrent — both attempts are in-flight simultaneously. This is correct and safe for read operations, but catastrophic for billing when each pod generates a per-request UUID in the billing handler:

// gRPC Java 1.x — channel with hedging ServiceConfig
// UNSAFE with UUID.randomUUID() on the server side
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;

String serviceConfig = "{\n" +
    "  \"methodConfig\": [{\n" +
    "    \"name\": [{\"service\": \"billing.BillingService\"}],\n" +
    "    \"hedgingPolicy\": {\n" +
    "      \"maxAttempts\": 2,\n" +
    "      \"hedgingDelay\": \"3s\",\n" +
    "      \"nonFatalStatusCodes\": [\"UNAVAILABLE\", \"RESOURCE_EXHAUSTED\"]\n" +
    "    }\n" +
    "  }]\n" +
    "}";

ManagedChannel channel = ManagedChannelBuilder.forAddress("billing-service", 50051)
    .defaultServiceConfig(parseJson(serviceConfig))
    .enableRetry()   // required for hedging to activate
    .build();

BillingServiceGrpc.BillingServiceBlockingStub stub =
    BillingServiceGrpc.newBlockingStub(channel);

// This call may be hedged — two simultaneous RPCs sent to different pods
ChargeBillingResponse response = stub.chargeBilling(
    ChargeBillingRequest.newBuilder()
        .setCustomerId("cust_789")
        .setAmount(9900)
        .setCurrency("usd")
        .setBillingPeriod("2026-08")
        .build()
);

// Server-side BillingServiceImpl — UNSAFE: UUID per RPC handler invocation
@Override
public void chargeBilling(ChargeBillingRequest request,
                          StreamObserver<ChargeBillingResponse> responseObserver) {
    // UNSAFE: UUID.randomUUID() generated at handler entry — different per RPC invocation.
    // Pod A (hedging attempt 1): key = "c3d4e5f6-a7b8-4c9d-0e1f-2a3b4c5d6e7f"
    // Pod B (hedging attempt 2): key = "d4e5f6a7-b8c9-4d0e-1f2a-3b4c5d6e7f8a" (sent 3s later)
    // Both RPCs may reach Stripe simultaneously (Stripe's idempotency cache has not yet
    // stored ch_A from pod A's call when pod B's hedged call arrives, because Stripe
    // processes the charge for a few seconds before committing it to the idempotency store).
    // Result: ch_A and ch_B both created, customer charged twice.
    String idempotencyKey = UUID.randomUUID().toString();

    try {
        String chargeId = stripeClient.createCharge(
            request.getCustomerId(), request.getAmount(),
            request.getCurrency(), request.getBillingPeriod(),
            idempotencyKey);
        responseObserver.onNext(ChargeBillingResponse.newBuilder()
            .setChargeId(chargeId)
            .setStatus("success")
            .build());
        responseObserver.onCompleted();
    } catch (StripeException e) {
        responseObserver.onError(Status.INTERNAL.withDescription(e.getMessage()).asRuntimeException());
    }
}

The failure scenario: the agent calls stub.chargeBilling() for cust_789, August 2026. The gRPC channel sends the RPC to Pod A. Pod A’s handler generates UUID = "c3d4e5f6..." and calls Stripe. Stripe begins processing. Pod A is running on a CPU-bound node and does not respond within the hedgingDelay of 3 seconds (Stripe’s processing latency occasionally exceeds 3 seconds on high-volume billing periods). After 3 seconds with no response trailer from Pod A, the gRPC channel sends a second copy of the RPC — with the same ChargeBillingRequest proto payload but a new HTTP/2 stream to Pod B. Pod B’s handler generates UUID = "d4e5f6a7...". Pod B calls Stripe with this key. Stripe has not yet committed ch_A to its idempotency store (ch_A creation is still in-flight). Pod B’s request creates ch_B. Pod A’s Stripe call completes 1 second later and ch_A is committed. The gRPC channel returns whichever response arrived first to the client — the other hedge is cancelled. Customer 789 is charged twice.

The hedging failure is not limited to UUID.randomUUID() in the handler. Any per-RPC-invocation value in the idempotency key triggers it: System.currentTimeMillis() at handler entry (Pod A at T=0, Pod B at T+3s = different milliseconds), request.hashCode() if the proto serialization produces different hash codes across JVMs (non-deterministic in Java), UUID.nameUUIDFromBytes(request.toByteArray()) if the request bytes differ across pods due to proto field ordering in the serialized form, or any thread-local or request-scoped counter that resets per JVM. Hedging makes the concurrent-duplicate scenario not just possible but guaranteed at scale — every billing call that takes longer than hedgingDelay triggers it.

The fix for failure mode 3

The safest fix for hedging is to disable the hedging policy on billing methods and use the retry policy instead. If Stripe’s p99 latency exceeds hedgingDelay on some requests, concurrent hedged copies will always reach the server — no application-layer fix can prevent that without disabling hedging. For methods where hedging is genuinely desired (e.g., read-only balance queries), the content-hash key approach still applies on the server side, but for billing, the correct answer is sequential retry with idempotent keys:

// Safe ServiceConfig: retry policy (sequential), NOT hedging, for billing methods
String safeServiceConfig = "{\n" +
    "  \"methodConfig\": [{\n" +
    "    \"name\": [{\"service\": \"billing.BillingService\",\n" +
    "               \"method\": \"ChargeBilling\"}],\n" +
    "    \"retryPolicy\": {\n" +
    "      \"maxAttempts\": 3,\n" +
    "      \"initialBackoff\": \"1s\",\n" +
    "      \"maxBackoff\": \"30s\",\n" +
    "      \"backoffMultiplier\": 2,\n" +
    "      \"retryableStatusCodes\": [\"UNAVAILABLE\"]\n" +
    "    },\n" +
    "    \"timeout\": \"60s\"\n" +  // long enough to cover Stripe's p99 latency
    "  }]\n" +
    "}";

// Safe server-side handler — content-hash key from caller metadata, pre-flight DB check
@Override
public void chargeBilling(ChargeBillingRequest request,
                          StreamObserver<ChargeBillingResponse> responseObserver) {
    // Extract the caller-supplied idempotency key from gRPC Metadata.
    // The client computes stableKey(customerId, billingPeriod) before calling the stub
    // and attaches it via MetadataUtils.newAttachHeadersInterceptor().
    Metadata headers = ServerInterceptorUtil.getHeaders();  // extracted by a server interceptor
    String idempotencyKey = headers.get(IDEMPOTENCY_KEY);

    if (idempotencyKey == null) {
        // Fallback: derive from request fields — stable across retries and hedged copies,
        // because request fields are identical for both the original and hedged RPC.
        // Still UNSAFE if proto serialization is non-deterministic across JVMs.
        // Prefer requiring callers to supply the key explicitly.
        idempotencyKey = stableKey(request.getCustomerId(), request.getBillingPeriod());
    }

    // Pre-flight: claim the billing slot. If the hedged copy arrives at Pod B
    // while Pod A is still in-flight with Stripe, the INSERT conflicts — Pod B
    // waits and returns the cached result when it becomes available.
    boolean inserted = billingRepo.insertIfAbsent(
        request.getCustomerId(), request.getBillingPeriod(), idempotencyKey);

    if (!inserted) {
        // Hedged copy or retry — billing already claimed by another pod or prior attempt.
        String chargeId = billingRepo.findChargeId(request.getCustomerId(), request.getBillingPeriod());
        if (chargeId != null) {
            responseObserver.onNext(ChargeBillingResponse.newBuilder()
                .setChargeId(chargeId)
                .setStatus("deduplicated")
                .build());
            responseObserver.onCompleted();
        } else {
            // Original call still in-flight on the other pod — return ALREADY_EXISTS;
            // gRPC retry policy will retry; by then, the original should have completed.
            responseObserver.onError(
                Status.ALREADY_EXISTS.withDescription("Billing in progress, retry shortly").asRuntimeException());
        }
        return;
    }

    // First pod to claim the billing slot — proceed with Stripe.
    try {
        String chargeId = stripeClient.createCharge(
            request.getCustomerId(), request.getAmount(),
            request.getCurrency(), request.getBillingPeriod(),
            idempotencyKey);
        billingRepo.markCompleted(request.getCustomerId(), request.getBillingPeriod(), chargeId);
        responseObserver.onNext(ChargeBillingResponse.newBuilder()
            .setChargeId(chargeId)
            .setStatus("success")
            .build());
        responseObserver.onCompleted();
    } catch (StripeException e) {
        billingRepo.markFailed(request.getCustomerId(), request.getBillingPeriod());
        responseObserver.onError(Status.INTERNAL.withDescription(e.getMessage()).asRuntimeException());
    }
}

The pre-flight INSERT ... ON CONFLICT DO NOTHING acts as a cluster-wide mutex for billing. The first pod to arrive — whether from the original call, a retry, or a hedged copy — wins the database row and proceeds with Stripe. All other pods (other hedged copies, retried calls from the client, replayed gRPC retries from the ServiceConfig retry policy) find the row already present and short-circuit to the cached result. The content-hash idempotency key ensures that even if Stripe is called twice (once before the DB guard was added, once after; or once by two pods simultaneously before either commits the billing record), Stripe deduplicates on the key it received first.

Gap analysis: other gRPC failure modes involving Stripe idempotency

The three failure modes above cover the most common gRPC-specific billing bugs. Additional failure modes exist in less common configurations:

• gRPC WaitForReady semantics combined with connection establishment retry — a stub configured with callOptions.withWaitForReady() buffers the RPC until the channel establishes a connection, then re-sends it. If the billing handler on the server creates UUID.randomUUID() per invocation and the connection establishment itself causes a delayed retry (the RPC is sent twice because of a channel state machine race during CONNECTING → READY → IDLE → CONNECTING transitions), ch_B is created. Fix: content-hash key on the server side, with the pre-flight DB check as the authoritative gate.
• gRPC NameResolver round-robin load balancing with pick_first fallback — when the pick-first subchannel drops, the load balancer picks the next available backend and re-sends buffered RPCs. If the billing handler was in-flight on the first backend (Stripe created ch_A), the re-sent RPC reaches the second backend, which generates a new UUID — ch_B. The content-hash key fix applies here without change; the RPC payload is identical regardless of which backend processes it.
• gRPC-Web proxies with automatic HTTP/1.1 fallback retry — Envoy or grpc-web-proxy may retry a failed gRPC-Web request on a different upstream. If the upstream billing service generates per-request UUIDs, the retry creates ch_B. Fix: the gRPC-Web client must supply the idempotency key in the request message proto or as a custom HTTP header that the proxy propagates.
• gRPC server-side streaming billing interrupted mid-stream — client retries from the beginning — a server-side streaming RPC BillingPeriodStream(BillingPeriodRequest) returns (stream BillingResult) charges all customers for a billing period and streams back results as they complete. The server crashes mid-stream after ch_1A through ch_47A are created but before the client receives results for customers 48–100. The client retries the entire streaming RPC. The server has no checkpoint table. Customers 1–47 are recharged with new UUIDs — ch_1B through ch_47B. Fix: server-side checkpoint in the billing_records table; server handler skips customers with existing records; pre-flight ON CONFLICT DO NOTHING is the guard for each customer in the stream.

The vault-key backstop

Content-hash keys and pre-flight database checks close each of these failure modes at the application layer. They require correct implementation in every billing path, every interceptor, every streaming handler, and every reconnect callback. A single missed UUID.randomUUID() in a code path added six months later — inside a new interceptor, inside a new streaming variant, or inside a new pod-specific handler — re-opens the vulnerability without triggering any test or static analysis warning.

A spend-cap proxy at the vendor API layer is the hard backstop that does not depend on application-layer correctness. Keybrake issues a scoped vault key per billing period: vault_key_cust_789_2026_08 with a policy {"vendor":"stripe","period_usd_cap":110,"expires_at":"2026-09-01T00:00:00Z"}. All gRPC billing calls — from ClientInterceptor retries, from bidirectional stream reconnects, from hedged concurrent copies — go through the proxy URL rather than directly to api.stripe.com. The proxy enforces the cap in real time: the 101st dollar in a $99/period billing period is blocked at the network layer before it reaches Stripe, regardless of how many UUID-keyed retry attempts fired from which pod.

The cap is set at expected_total × 1.10 — ten percent over the legitimate billing amount — to allow Stripe’s own idempotency deduplication to absorb the first retry before the cap fires. If a content-hash retry reaches Stripe and Stripe returns the cached ch_A response, the proxied total stays at one charge. If a hedged copy reaches Stripe with a different UUID before ch_A is committed to the idempotency store and ch_B is created, the cap fires on the second charge — before the customer is charged a third or fourth time by additional hedging attempts or by a retry from the client after both hedges complete.

Summary

The pattern across all three failure modes is the same as in Netty Pipeline Handlers, Akka Streams, and Akka HTTP: the framework gives you retry, reconnect, and concurrency primitives that are natural for distributed RPC handling but dangerous for billing, because they re-invoke your billing code at exactly the moment when a per-invocation value in the idempotency key produces a different Stripe key. A content-hash key derived from business fields only — sha256(customerId + ":" + billingPeriod + ":grpc-billing")[:32] — is identical across all invocations for the same billing intent, regardless of which pod handles the RPC, how many concurrent hedging copies are in-flight, or how many times the bidirectional stream has reconnected. The pre-flight ON CONFLICT DO NOTHING database check is the authoritative gate. The vault key is the backstop when both fail.