Apache Camel and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Apache Camel’s onException().maximumRedeliveries() error handler re-invokes the processor on each redelivery attempt. UUID.randomUUID() inside the processor body is a call expression that evaluates per processor invocation, not once at exchange creation time. The initial delivery creates ch_A before a StripeException wrapping a socket timeout; the first redelivery evaluates a new UUID.randomUUID(), causing Stripe to create ch_B. Three Apache Camel-specific Stripe billing failure modes: onException().maximumRedeliveries() re-invokes the processor body with fresh UUID.randomUUID() per redelivery — subtler: including exchange.getProperty(Exchange.REDELIVERY_COUNTER) in the key construction produces a structurally distinct key per delivery attempt; the Camel Kafka component hides Kafka message identity behind exchange.getIn().getMessageId(), which is Camel’s internal UUID regenerated per exchange — a crash between Stripe call and offset commit causes redelivery with a new internal ID, producing ch_B for a customer whose ch_A was already committed; and from(“timer://billing?period=86400000”) fires independently on every pod in a Kubernetes deployment with no built-in cluster coordination — all three pods execute the billing route simultaneously, each generating a distinct UUID.randomUUID() per customer, producing ch_A, ch_B, and ch_C per customer per billing period.

This post covers all three failure modes with Java code (Apache Camel 4.x), content-hash stable keys set as exchange properties at route entry, Kafka message key used as the idempotency key seed, pg_try_advisory_lock() for cluster-wide timer serialization, and pre-flight PostgreSQL ON CONFLICT DO NOTHING as a durable billing mutex — plus per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For Camel’s RabbitMQ component and message-broker retry patterns, see the RabbitMQ and Stripe Integration post. For Spring Batch’s FaultTolerantStep retry and ItemWriter.write() re-invocation, see the Spring Integration and Spring Batch and Stripe Integration post.

Failure mode 1: onException().maximumRedeliveries() re-invokes the processor — UUID.randomUUID() inside the processor body evaluates fresh per redelivery — initial delivery creates ch_A before StripeException — first redelivery creates ch_B

Apache Camel’s error handling model centers on the redelivery policy. When a route processor throws an exception that matches an onException() clause configured with maximumRedeliveries(N), Camel re-invokes the processor on the same Exchange object up to N times. The processor method is called again from the beginning. Any call expression inside the processor body — including UUID.randomUUID() — evaluates fresh on each invocation.

This is not a subtle misuse of the API. It is how Camel error handling is designed to work: the same message, delivered again, processed again. The problem is that “processed again” and “charged again with the same Stripe idempotency key” are not equivalent when UUID.randomUUID() is evaluated inside the processor:

// UNSAFE: UUID.randomUUID() inside Camel Processor.process().
// Camel onException().maximumRedeliveries(3) re-invokes process() on each redelivery.
// The initial invocation and every subsequent redelivery each generate a distinct UUID.

import com.stripe.model.Charge;
import com.stripe.net.RequestOptions;
import com.stripe.param.ChargeCreateParams;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;

import java.util.UUID;

public class StripeChargeProcessor implements Processor {

    @Override
    public void process(Exchange exchange) throws Exception {
        BillingRequest req = exchange.getIn().getBody(BillingRequest.class);

        // UNSAFE: evaluated per process() invocation, not once at exchange creation.
        // Delivery 1: UUID = "7a3f1b2c-4d5e-6f7a-8b9c-0d1e2f3a4b5c" → ch_A
        // Redelivery 1: UUID = "c9d8e7f6-5a4b-3c2d-1e0f-9a8b7c6d5e4f" → ch_B ← duplicate
        // Redelivery 2: UUID = "b2a1c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d" → ch_C ← triplicate
        String idempotencyKey = UUID.randomUUID().toString();

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(req.getAmountCents())
            .setCurrency("usd")
            .setCustomer(req.getCustomerId())
            .build();

        // POST /v1/charges — first call: ch_A committed before socket timeout fires.
        // Camel catches StripeException (wrapping SocketTimeoutException),
        // waits redelivery backoff, calls process() again with same exchange.
        // Second call: new UUID → Stripe creates ch_B.
        Charge.create(params, options);
    }
}

// Route definition:
from("direct:chargeCustomer")
    .onException(com.stripe.exception.StripeException.class)
        .maximumRedeliveries(3)
        .redeliveryDelay(1000)
        .backOffMultiplier(2.0)
        .useExponentialBackOff()
        .handled(true)
        .to("direct:billingError")
    .end()
    .process(stripeChargeProcessor);

// Execution timeline for customer "cust_123":
// 16:00:00.000  process() called — UUID_A → POST /v1/charges (body in flight)
// 16:00:29.998  Stripe commits ch_A (charges.created event)
// 16:00:30.000  SocketTimeoutException fires (30s timeout) → StripeException caught by Camel
// 16:00:31.000  Camel redelivery 1 — process() called again — UUID_B → POST /v1/charges
// 16:00:31.100  Stripe: new key, processes as new charge → ch_B committed
// Result: customer "cust_123" charged twice in same billing period.

The exchange object is the same on redelivery — exchange.getIn().getBody() returns the same BillingRequest. But UUID.randomUUID() is not reading from the exchange; it is a static method call that produces a new random UUID every time it is invoked regardless of what the exchange contains. The processor has no memory across invocations unless the developer explicitly stores state in exchange properties.

Subtler variant: including Exchange.REDELIVERY_COUNTER in the key construction produces a structurally distinct key per delivery attempt by design

Camel sets Exchange.REDELIVERY_COUNTER on the exchange properties before each redelivery attempt. Its value is 0 on the initial delivery, 1 on the first redelivery, 2 on the second, and so on. A developer who tries to build a “stable” idempotency key from exchange state may reach for Exchange.REDELIVERY_COUNTER as a component — reasoning that it identifies the attempt and could disambiguate concurrent deliveries. The result is the opposite of what is intended:

// ALSO UNSAFE: REDELIVERY_COUNTER-based key produces a distinct key per attempt.
// Developer intent: "include attempt number to make key unique per attempt."
// Actual effect: makes a different Stripe idempotency key on every delivery → ch_A, ch_B, ch_C.

public class StripeChargeProcessor implements Processor {

    @Override
    public void process(Exchange exchange) throws Exception {
        BillingRequest req = exchange.getIn().getBody(BillingRequest.class);
        int redeliveryCount = exchange.getProperty(
            Exchange.REDELIVERY_COUNTER, 0, Integer.class
        );

        // UNSAFE: key changes per redelivery — different key = different charge at Stripe.
        // Delivery 1 (count=0): "cust_123:2026-09:0" → sha256 → ch_A
        // Redelivery 1 (count=1): "cust_123:2026-09:1" → sha256 → ch_B ← duplicate
        // Redelivery 2 (count=2): "cust_123:2026-09:2" → sha256 → ch_C ← triplicate
        String idempotencyKey = DigestUtils.sha256Hex(
            req.getCustomerId() + ":" + req.getBillingPeriod() + ":" + redeliveryCount
        ).substring(0, 32);

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(req.getAmountCents())
            .setCurrency("usd")
            .setCustomer(req.getCustomerId())
            .build();
        Charge.create(params, options);
    }
}

// This is worse than UUID.randomUUID(): UUID.randomUUID() at least has a very low
// probability of collision between two distinct customers' keys. The REDELIVERY_COUNTER
// approach guarantees a distinct key per attempt by construction — it is deliberately
// designed to differ on every redelivery, which is the exact opposite of what a
// Stripe idempotency key requires.

The fix is to compute the stable key once, before the exchange reaches the billing processor, and store it in an exchange property. The billing processor reads the key from the exchange property on every invocation. The key never changes regardless of how many times the processor is re-invoked:

// SAFE: stable key set as exchange property at route entry, before any retry logic fires.
// The billing processor reads the key from the exchange property on every invocation.
// Same key on delivery 1, redelivery 1, redelivery 2 → Stripe idempotency cache
// returns ch_A for all invocations after the first.

import org.apache.camel.builder.RouteBuilder;
import org.apache.commons.codec.digest.DigestUtils;

public class BillingRouteBuilder extends RouteBuilder {

    @Override
    public void configure() {
        onException(com.stripe.exception.StripeException.class)
            .maximumRedeliveries(3)
            .redeliveryDelay(1000)
            .backOffMultiplier(2.0)
            .useExponentialBackOff()
            .handled(true)
            .to("direct:billingError");

        from("direct:chargeCustomer")
            // Compute stable key FIRST, before billing processor, before any retry loop.
            // Exchange property persists across all redelivery invocations of process().
            .process(exchange -> {
                BillingRequest req = exchange.getIn().getBody(BillingRequest.class);
                String stableKey = DigestUtils.sha256Hex(
                    req.getCustomerId() + ":" +
                    req.getBillingPeriod() + ":" +
                    "camel-billing"
                ).substring(0, 32);
                exchange.setProperty("stripeIdempotencyKey", stableKey);
            })
            .process(stripeChargeProcessor);
    }
}

// Safe billing processor reads key from exchange property, never generates UUID:
public class StripeChargeProcessor implements Processor {

    @Override
    public void process(Exchange exchange) throws Exception {
        BillingRequest req = exchange.getIn().getBody(BillingRequest.class);

        // SAFE: key was set by the upstream keyComputingProcessor.
        // Same String reference on delivery 1, redelivery 1, redelivery 2.
        String idempotencyKey = exchange.getProperty("stripeIdempotencyKey", String.class);

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(req.getAmountCents())
            .setCurrency("usd")
            .setCustomer(req.getCustomerId())
            .build();

        // Delivery 1: POST /v1/charges key="a3f1b2c4..." → ch_A committed
        // SocketTimeoutException fires → Camel redelivery 1
        // Redelivery 1: POST /v1/charges key="a3f1b2c4..." → Stripe cache hit → returns ch_A
        // No duplicate charge created.
        Charge.create(params, options);
    }
}

// Key placement rule: compute the stable key in a processor that runs BEFORE
// onException retry scope, OR ensure the key-computing processor is also
// covered by the retry scope but produces the same output on every invocation.
// sha256(customerId:billingPeriod:camel-billing) satisfies the latter:
// it is a pure function of its inputs, so re-running it on redelivery returns
// the same value even if the key-computing processor is inside the retry scope.

One practical note: if the key-computing processor is inside the route covered by onException, Camel will re-invoke it on redelivery. A content-hash function like sha256(customerId:billingPeriod:camel-billing) is a pure function of deterministic inputs, so re-running it on redelivery produces the same key — the property is overwritten with the same value. That is safe. But any non-deterministic computation in the key-computing processor — UUID.randomUUID(), System.currentTimeMillis(), a database sequence, a counter incremented on each run — reintroduces the duplicate-key problem regardless of which processor calls it. The key’s value must be the same on every invocation. Content-hash of stable request fields guarantees this; random generation or time-based generation does not.

Using defaultErrorHandler().maximumRedeliveries(N) instead of onException() has the same behavior: the route’s processors are re-invoked on redelivery. The key-computation rule applies in both cases. Camel’s deadLetterChannel() fires only after redeliveries are exhausted and does not affect the in-retry processor re-invocation pattern.

Failure mode 2: Camel Kafka consumer hides Kafka message identity behind exchange.getIn().getMessageId() — crash between Stripe call and offset commit causes redelivery with a new Camel-internal message ID — processor generates fresh UUID.randomUUID() for redelivered exchange — ch_B while ch_A already committed

The Camel Kafka component’s default configuration commits the Kafka consumer offset after the exchange completes successfully. This means the commit happens after all route processors have returned without throwing. If the application crashes, is killed by a pod eviction, or throws an uncaught exception after the billing processor completes but before Camel’s auto-commit sends the offset to the Kafka broker, the message is redelivered on the next consumer start.

The redelivered message arrives on a new Exchange object. A processor that calls exchange.getIn().getMessageId() will get a new value: Camel generates a fresh UUID for the internal message ID on every new Exchange created from an incoming message. The Kafka message itself has a stable identity — its partition, offset, and key — but getMessageId() is not that identity; it is Camel’s own abstraction layer identifier, and it changes on every re-delivery:

// UNSAFE: using exchange.getIn().getMessageId() as Stripe idempotency key seed.
// Camel Kafka consumer creates a new Exchange per message delivery.
// getMessageId() returns a new Camel-generated UUID on every Exchange, not a
// stable identifier derived from the Kafka partition+offset+key.

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.commons.codec.digest.DigestUtils;

public class KafkaBillingProcessor implements Processor {

    @Override
    public void process(Exchange exchange) throws Exception {
        BillingEvent event = exchange.getIn().getBody(BillingEvent.class);

        // UNSAFE: getMessageId() is Camel's internal UUID — changes per Exchange object.
        // Original delivery: messageId = "0a1b2c3d-4e5f-6a7b-8c9d-0e1f2a3b4c5d"
        //   → sha256("cust_123:0a1b2c3d...") → key_A → POST /v1/charges → ch_A committed
        //   App crashes between ch_A commit and Kafka offset commit.
        // Redelivery on restart: new Exchange → messageId = "f7e6d5c4-b3a2-1f0e-9d8c-7b6a5f4e3d2c"
        //   → sha256("cust_123:f7e6d5c4...") → key_B → POST /v1/charges → ch_B ← duplicate
        String messageId = exchange.getIn().getMessageId();
        String idempotencyKey = DigestUtils.sha256Hex(
            event.getCustomerId() + ":" + messageId
        ).substring(0, 32);

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getCustomerId())
            .build();
        Charge.create(params, options);
    }
}

// Route:
from("kafka:billing-events?groupId=billing-group&autoOffsetReset=earliest")
    .process(kafkaBillingProcessor);

// Execution timeline:
// Pod 1 running. Kafka message key="cust_123:2026-09", partition=2, offset=14177.
// 14:22:00.000  Exchange created — messageId = "0a1b2c3d-..." (Camel-internal UUID)
// 14:22:00.500  POST /v1/charges — ch_A committed in Stripe ledger
// 14:22:00.501  Pod 1 killed (OOM eviction) — offset 14177 NOT yet committed to Kafka
// Pod 2 starts. Kafka delivers partition=2 offset=14177 again (uncommitted).
// 14:22:45.000  Exchange created — messageId = "f7e6d5c4-..." (NEW Camel-internal UUID)
// 14:22:45.500  POST /v1/charges — key_B ≠ key_A → Stripe creates ch_B ← duplicate

The Kafka consumer offset commit behavior is controlled by the autoCommitEnable and allowManualCommit Camel Kafka component options. With the default autoCommitEnable=true, Camel commits the offset after the exchange completes. The crash window between the billing processor completing and the offset commit reaching the broker is small but non-zero, particularly under pod eviction conditions where the JVM is killed before it can flush in-flight network writes.

Subtler variant: exchange.getIn().getHeader(KafkaConstants.OFFSET) combined with partition number changes between Kafka partition rebalances — offset 14177 on partition 2 before rebalance becomes offset 14177 on partition 0 after rebalance for a different customer

A developer who discovers that getMessageId() is Camel-internal may switch to building the idempotency key from KafkaConstants.OFFSET and KafkaConstants.PARTITION. This is more stable — offset and partition do identify the Kafka message position and will be the same on crash-recovery redelivery. But using offset alone, or offset combined with partition number, is unsafe across Kafka partition rebalances: partition 2 offset 14177 is a position in partition 2’s log; partition 0 offset 14177 is a position in partition 0’s log. These may reference completely different messages for different customers if the topic’s partition count changes (rare) or if the same offset appears in a different partition after a consumer group rebalance changes which partition each consumer handles. A better seed is the Kafka message key, which is set by the producer and is stable across all redeliveries, reconnections, and partition rebalances:

// SAFE: idempotency key seeded from the Kafka message key, not the Camel-internal
// message ID, not partition+offset alone.
// The producer sets the Kafka message key at publish time from deterministic
// billing fields. The same key is delivered on every redelivery of the same message.

import org.apache.camel.component.kafka.KafkaConstants;

public class KafkaBillingProcessor implements Processor {

    @Override
    public void process(Exchange exchange) throws Exception {
        BillingEvent event = exchange.getIn().getBody(BillingEvent.class);

        // SAFE: Kafka message key set by producer as "customerId:billingPeriod".
        // Stable across crash-recovery redelivery, partition rebalance, consumer restart.
        // KafkaConstants.KEY header is set by the Camel Kafka component from the
        // Kafka ConsumerRecord's key field — not a Camel-generated value.
        String kafkaKey = exchange.getIn().getHeader(KafkaConstants.KEY, String.class);

        if (kafkaKey == null || kafkaKey.isBlank()) {
            // Message was published without a key — fall back to content-hash of
            // deterministic event fields. Never fall back to UUID.randomUUID() or
            // exchange.getIn().getMessageId().
            kafkaKey = event.getCustomerId() + ":" + event.getBillingPeriod();
        }

        // sha256(kafkaKey + ":camel-kafka-billing")[:32] is stable across redeliveries
        // because kafkaKey is determined at message production time, not at consumption time.
        String idempotencyKey = DigestUtils.sha256Hex(
            kafkaKey + ":camel-kafka-billing"
        ).substring(0, 32);

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getCustomerId())
            .build();
        Charge.create(params, options);
    }
}

// Producer sets the Kafka message key deterministically:
// ProducerRecord<String, BillingEvent> record = new ProducerRecord<>(
//     "billing-events",
//     customerId + ":" + billingPeriod,  // ← message key: stable, producer-controlled
//     billingEvent
// );

// With this key:
// Original delivery: kafkaKey = "cust_123:2026-09"
//   → sha256("cust_123:2026-09:camel-kafka-billing") → "a3f1b2c4..." → ch_A
// Crash — offset not committed.
// Redelivery: kafkaKey = "cust_123:2026-09" (same Kafka message, same key field)
//   → sha256("cust_123:2026-09:camel-kafka-billing") → "a3f1b2c4..." → Stripe cache hit → ch_A
// No duplicate charge.

A pre-flight database guard provides durable protection beyond Stripe’s 24-hour idempotency cache TTL. The database insert runs before the Stripe API call and catches redeliveries that arrive more than 24 hours after the original charge, when Stripe’s cache has expired:

// Pre-flight guard: INSERT ... ON CONFLICT (customer_id, billing_period) DO NOTHING
// Runs before Stripe API call. Returns 0 rows inserted if customer already billed.
// Durable across pod restarts, Kafka consumer group changes, and Stripe cache expiry.

public class KafkaBillingProcessor implements Processor {

    private final DataSource dataSource;

    public KafkaBillingProcessor(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    @Override
    public void process(Exchange exchange) throws Exception {
        BillingEvent event = exchange.getIn().getBody(BillingEvent.class);
        String kafkaKey = exchange.getIn().getHeader(KafkaConstants.KEY, String.class);
        if (kafkaKey == null || kafkaKey.isBlank()) {
            kafkaKey = event.getCustomerId() + ":" + event.getBillingPeriod();
        }
        String idempotencyKey = DigestUtils.sha256Hex(
            kafkaKey + ":camel-kafka-billing"
        ).substring(0, 32);

        // Pre-flight guard: returns false if already billed (no row inserted).
        boolean shouldCharge = insertBillingAttempt(
            event.getCustomerId(), event.getBillingPeriod(), idempotencyKey
        );

        if (!shouldCharge) {
            // Already billed — skip Stripe call, mark exchange as handled.
            exchange.getIn().setHeader("BillingSkipped", true);
            return;
        }

        RequestOptions options = RequestOptions.builder()
            .setIdempotencyKey(idempotencyKey)
            .build();
        ChargeCreateParams params = ChargeCreateParams.builder()
            .setAmount(event.getAmountCents())
            .setCurrency("usd")
            .setCustomer(event.getCustomerId())
            .build();
        Charge charge = Charge.create(params, options);

        // Record charge ID for reconciliation.
        markBillingComplete(event.getCustomerId(), event.getBillingPeriod(), charge.getId());
    }

    private boolean insertBillingAttempt(String customerId, String period, String key) {
        try (Connection conn = dataSource.getConnection();
             PreparedStatement ps = conn.prepareStatement(
                 "INSERT INTO billing_attempts " +
                 "(customer_id, billing_period, idempotency_key, created_at) " +
                 "VALUES (?, ?, ?, NOW()) " +
                 "ON CONFLICT (customer_id, billing_period) DO NOTHING"
             )) {
            ps.setString(1, customerId);
            ps.setString(2, period);
            ps.setString(3, key);
            int rows = ps.executeUpdate();
            return rows == 1; // 1 = inserted (proceed to charge); 0 = already exists (skip)
        }
    }
}

The ON CONFLICT (customer_id, billing_period) DO NOTHING constraint requires a unique index on (customer_id, billing_period) in the billing_attempts table. Once a row is inserted for a given customer and billing period, all subsequent calls — from redelivered Kafka messages, from crash-recovery replays, from a second consumer accidentally assigned the same partition — find the row already present and skip the Stripe API call entirely. The constraint is enforced by PostgreSQL’s MVCC, so two concurrent connections racing to insert the same row will result in exactly one insert succeeding and the other returning 0 rows.

Failure mode 3: from(“timer://billing?period=86400000”) on Kubernetes replicas:3 fires on all three pods simultaneously — TOCTOU race on customer billing query — each pod generates distinct UUID.randomUUID() per customer — ch_A, ch_B, and ch_C per customer per billing period

Apache Camel’s timer consumer — from("timer://billing?period=86400000") — uses a java.util.Timer scheduled in the JVM. There is no cross-JVM coordination mechanism in the default timer component. When a Kubernetes Deployment sets replicas: 3, three independent JVM processes start, each initializing its own CamelContext with its own timer scheduler. All three timers fire within the same scheduling window, all three pods execute the billing route, and all three processors call UUID.randomUUID() per customer independently.

The from("timer://...") declaration has no knowledge of other pods. This is not a Camel bug — it is the correct behavior for a single-JVM timer. The bug is deploying a timer-triggered billing route without cluster-aware coordination in a multi-pod environment:

// UNSAFE: timer route on Kubernetes replicas:3.
// Three independent CamelContext instances each fire their own timer.
// All three pods execute billing route within milliseconds of each other.

public class BillingTimerRouteBuilder extends RouteBuilder {

    @Override
    public void configure() {
        // This route runs on EVERY pod. replicas:3 means 3 concurrent billing runs.
        from("timer://monthlyBilling?period=86400000&delay=5000")
            .routeId("monthly-billing-timer")
            .process(exchange -> {
                // Fetch customers due for billing — TOCTOU: all 3 pods run this query
                // simultaneously, before any pod has written billing_started records.
                List<Customer> customers = customerRepository.findDueForBilling();

                for (Customer customer : customers) {
                    // UNSAFE: independent UUID per pod per customer.
                    // Pod 1 thread: UUID_A for cust_123 → ch_A committed
                    // Pod 2 thread: UUID_B for cust_123 → ch_B committed (concurrent)
                    // Pod 3 thread: UUID_C for cust_123 → ch_C committed (concurrent)
                    String idempotencyKey = UUID.randomUUID().toString();

                    RequestOptions options = RequestOptions.builder()
                        .setIdempotencyKey(idempotencyKey)
                        .build();
                    ChargeCreateParams params = ChargeCreateParams.builder()
                        .setAmount(customer.getAmountCents())
                        .setCurrency("usd")
                        .setCustomer(customer.getStripeId())
                        .build();
                    Charge.create(params, options);
                }
            });
    }
}

// Even with a stable content-hash key (sha256(customerId:billingPeriod:camel-timer)):
// Pod 1: sha256("cust_123:2026-09:camel-timer") → key_stable → ch_A (committed)
// Pod 2: sha256("cust_123:2026-09:camel-timer") → key_stable → Stripe: 409 IdempotencyError
//   if Pod 1 is still processing (same key, concurrent request — Stripe rejects)
// Pod 3: sha256("cust_123:2026-09:camel-timer") → key_stable → same race
//
// The stable key prevents triple-charging but may cause Stripe 409 errors
// and does not prevent the three pods from all making concurrent Stripe API calls.
// The pre-flight advisory lock prevents all three from even reaching the Stripe call.

A stable content-hash key (covered in failure mode 1’s fix) prevents the duplicate charge in the stable-key path: Stripe’s idempotency cache returns ch_A for all concurrent requests with the same key. But it does not prevent the three pods from making concurrent Stripe API calls. Stripe responds with HTTP 409 for concurrent requests with the same idempotency key that are still in-flight, which Camel’s error handler may then retry with redelivery. The correct fix is to prevent more than one pod from executing the billing route at all, not to rely on Stripe’s cache to deduplicate concurrent requests:

// SAFE OPTION 1: pg_try_advisory_lock() at route entry — prevents multi-pod execution.
// Only the pod that acquires the lock runs the billing route. Other pods' process()
// calls return immediately without making any Stripe API calls.

public class BillingTimerRouteBuilder extends RouteBuilder {

    private final DataSource dataSource;

    public BillingTimerRouteBuilder(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    @Override
    public void configure() {
        from("timer://monthlyBilling?period=86400000&delay=5000")
            .routeId("monthly-billing-timer")
            .process(exchange -> {
                String billingPeriod = YearMonth.now().toString(); // e.g. "2026-09"
                // Advisory lock key: stable hash of the billing context.
                // All 3 pods compute the same lockKey for the same billing period.
                long lockKey = Math.abs(
                    ("camel-monthly-billing:" + billingPeriod).hashCode()
                );

                try (Connection conn = dataSource.getConnection();
                     PreparedStatement ps = conn.prepareStatement(
                         "SELECT pg_try_advisory_lock(?)"
                     )) {
                    ps.setLong(1, lockKey);
                    ResultSet rs = ps.executeQuery();
                    rs.next();
                    boolean acquired = rs.getBoolean(1);

                    if (!acquired) {
                        // Another pod holds the lock — billing already in progress.
                        // Skip this timer firing. Lock holder will complete the run.
                        exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
                        return;
                    }

                    // This pod holds the lock. Proceed with billing.
                    try {
                        runBillingRun(billingPeriod);
                    } finally {
                        // Release advisory lock when billing run completes.
                        // Advisory locks on a connection are released when
                        // the connection closes or when pg_advisory_unlock() is called.
                        try (PreparedStatement unlock = conn.prepareStatement(
                             "SELECT pg_advisory_unlock(?)")) {
                            unlock.setLong(1, lockKey);
                            unlock.execute();
                        }
                    }
                }
            });
    }

    private void runBillingRun(String billingPeriod) throws Exception {
        List<Customer> customers = customerRepository.findDueForBilling(billingPeriod);

        for (Customer customer : customers) {
            // Pre-flight guard: skip if already billed.
            String stableKey = DigestUtils.sha256Hex(
                customer.getId() + ":" + billingPeriod + ":camel-timer-billing"
            ).substring(0, 32);

            boolean inserted = insertBillingAttempt(customer.getId(), billingPeriod, stableKey);
            if (!inserted) continue;

            RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(stableKey)
                .build();
            ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(customer.getAmountCents())
                .setCurrency("usd")
                .setCustomer(customer.getStripeId())
                .build();
            Charge.create(params, options);
        }
    }
}

Safer alternative: replace timer with a queue-based trigger and Kubernetes replicas:1 for the billing worker

The most reliable fix is architectural: do not use a timer route for cluster-wide billing coordination. Instead, publish a billing trigger message to a queue (Kafka, SQS, RabbitMQ) from a separate scheduler (cron job, Kubernetes CronJob, database-backed scheduler) and consume it with a Camel Kafka or SQS route on a deployment with replicas:1. Consumer group semantics ensure only one consumer instance processes each trigger message. The queue acts as the cluster-wide coordination mechanism, eliminating the need for advisory locks:

// SAFE OPTION 2: queue-triggered billing with replicas:1 consumer.
// CronJob (or a Quartz-based scheduler with pg_try_advisory_lock) publishes one
// billing trigger message per billing period. Camel Kafka consumer at replicas:1
// processes exactly one trigger message per billing period.

// Kubernetes CronJob (publishes trigger message, does not run billing):
// schedule: "0 0 1 * *"  (first of each month at midnight)
// command: kafka-console-producer --topic billing-triggers --message "2026-09"

// Camel consumer (replicas:1 Deployment — single consumer, single billing run):
public class BillingTriggerRouteBuilder extends RouteBuilder {

    @Override
    public void configure() {
        from("kafka:billing-triggers?groupId=billing-trigger-group&autoOffsetReset=earliest")
            .routeId("billing-trigger-consumer")
            .process(exchange -> {
                String billingPeriod = exchange.getIn().getBody(String.class);
                // Single consumer instance — no concurrent execution across pods.
                // Still use stable key + pre-flight guard as defense in depth
                // (protects against accidental duplicate trigger messages).
                runBillingRun(billingPeriod);
            });
    }
}

// Kubernetes Deployment for billing consumer:
// spec:
//   replicas: 1          ← single instance — no timer race
//   selector: { billing-consumer }
//   template:
//     containers:
//       - name: billing-consumer
//         image: billing-service:latest
//
// The Camel timer route is removed entirely from this deployment.
// Timer-based scheduling responsibility moves to the Kubernetes CronJob,
// which runs in a separate Job pod (also replicas:1 by default).

The queue-based approach separates two concerns that the timer route conflates: when to run billing (scheduling) and how to run billing (processing). The CronJob handles scheduling and publishes exactly one trigger per billing period. The Camel consumer handles processing and is guaranteed single-instance by replicas:1. The pg_try_advisory_lock() approach from option 1 is appropriate when refactoring to queue-based triggering is not immediately feasible, but it adds a database dependency to every timer firing and a risk of lock leak if the advisory lock connection closes before the billing run completes. For production billing systems, the queue-based architecture is more robust.

Why idempotency keys from Stripe’s 24-hour cache are not the last line of defense

Stripe’s idempotency cache matches requests by key within a 24-hour sliding window. A request with key K submitted at 14:00:00 UTC will match cache for requests with the same key K submitted before 14:00:00 UTC the following day. After 24 hours, the key expires from Stripe’s cache. A new request with the same key after expiry is treated as a fresh charge.

For monthly billing, a crash-recovery scenario where the redelivered Kafka message arrives more than 24 hours after the original message was first processed will bypass Stripe’s idempotency cache entirely — the stable key matches nothing in the cache, and Stripe creates a new charge. The pre-flight ON CONFLICT (customer_id, billing_period) DO NOTHING guard is immune to Stripe’s cache TTL because it checks your own database, which retains billing records indefinitely. The advisory lock is also immune because it prevents the billing processor from reaching the Stripe API call at all when a billing run is already in progress or already completed for the billing period.

These three guards form a layered defense:

Implementation checklist for Apache Camel and Stripe billing

• Never call UUID.randomUUID() inside a Camel Processor.process() method that is covered by onException().maximumRedeliveries() or defaultErrorHandler().maximumRedeliveries(). Camel re-invokes process() on each redelivery attempt. The stable idempotency key must be computed in an upstream processor and stored in an exchange property before the billing processor executes. The billing processor reads the key from the exchange property, not from UUID.randomUUID().
• Do not include exchange.getProperty(Exchange.REDELIVERY_COUNTER) in the idempotency key. REDELIVERY_COUNTER is 0 on first delivery, 1 on first redelivery, 2 on second — including it in the key construction guarantees a distinct key per delivery attempt. The key must be stable across all delivery attempts for the same logical billing operation.
• Do not use exchange.getIn().getMessageId() as the Stripe idempotency key seed in Camel Kafka consumer routes. getMessageId() returns a Camel-generated UUID for the exchange object, not the Kafka message key. It changes on every redelivered exchange. Use exchange.getIn().getHeader(KafkaConstants.KEY, String.class) as the key seed — it is the producer-assigned Kafka message key, which is stable across crash-recovery redeliveries. Ensure the producer sets the Kafka message key from deterministic billing fields (customerId + ":" + billingPeriod).
• Never deploy a billing timer route (from(“timer://...”)) to a Kubernetes Deployment with replicas > 1 without cluster-wide coordination. Camel’s timer runs independently in each JVM. All pods fire within the same scheduling window. Add pg_try_advisory_lock() at the route’s first processor step, or replace the timer route with a queue-based trigger consumed by a replicas:1 Deployment.
• Use a stable content-hash key of the form sha256(customerId:billingPeriod:camel-billing)[:32], not sha256(customerId:billingPeriod:redeliveryCount)[:32]. The key must produce the same value for the same logical billing operation regardless of how many times the route has been invoked. Redelivery count, attempt number, timestamp, and random values all violate this requirement.
• Add a pre-flight INSERT ... ON CONFLICT (customer_id, billing_period) DO NOTHING before the Stripe API call. The pre-flight guard catches duplicate billing attempts that arrive after Stripe’s 24-hour idempotency cache expires — delayed Kafka redeliveries, long-delayed job restarts, and manual retry requests from operators. The database row is permanent; the Stripe cache is not.
• Add a vault key spend cap at expected_total × 1.10 per billing period. The cap fires at the proxy layer regardless of what Camel error handler configuration, Kafka consumer offset setting, or Kubernetes replica count is deployed. It bounds the financial damage if any of the above failure modes fires in a configuration path that was not tested.