AI agent observability: what tracing tools miss about vendor API spend

What standard observability tools record for vendor API calls

When your agent calls https://api.stripe.com/v1/payment_intents, an OpenTelemetry auto-instrumentation span records:

Span: POST https://api.stripe.com/v1/payment_intents
  duration_ms: 342
  http.status_code: 200
  http.method: POST
  http.url: https://api.stripe.com/v1/payment_intents
  error: false
  trace_id: 4d8f1a2b3c9e0f7a
  parent_span_id: 7f3e2c1b9a4d8f0e

What it doesn't record: the amount field from the response body (which tells you how many dollars just moved), the Stripe PaymentIntent.id that would let you find this charge in the Stripe dashboard, which agent run or workflow execution triggered this call, or how this call's cost compares to the budget allocated for this run. The span tells you the call succeeded in 342ms. It doesn't tell you it charged $149.99.

The four observability questions agent operations teams need

Why vendor dashboards don't fill the gap

The Stripe dashboard shows total charges, but it doesn't know which of your agents made each charge. Stripe metadata can carry agent identifiers — if you instrument every Stripe API call to include metadata: { agent_run_id: "run_123" } — but this requires modifying every service that calls Stripe, maintaining the instrumentation across deploys, and building custom reports in the Stripe Radar or Sigma products to aggregate by metadata field. The Twilio dashboard shows SMS cost but has no per-agent breakdown. Resend has no per-call cost breakdown at all (flat rate billing). And none of the vendor dashboards give you a cross-vendor view: how much did the billing agent spend across Stripe, Twilio, and Resend combined for the 3am run?

The minimum audit log schema for agent spend observability

The data you need is a structured event for each vendor API call with cost and agent attribution:

-- Minimum audit log schema
CREATE TABLE agent_spend_audit (
  id            TEXT PRIMARY KEY,
  vault_key_id  TEXT NOT NULL,        -- which scoped key made this call
  agent_run_label TEXT,               -- "billing-agent/run_abc123" from vault key
  vendor        TEXT NOT NULL,        -- 'stripe', 'twilio', 'resend'
  endpoint      TEXT NOT NULL,        -- 'POST /v1/payment_intents'
  cost_usd      DECIMAL(10,4),        -- parsed from vendor response
  vendor_txn_id TEXT,                 -- Stripe PaymentIntent.id, Twilio SID, Resend email_id
  policy_verdict TEXT,                -- 'allowed', 'cap_exhausted', 'endpoint_blocked'
  called_at     TIMESTAMPTZ NOT NULL,
  http_status   INTEGER
);

-- Index for per-run queries
CREATE INDEX idx_audit_run_label ON agent_spend_audit(agent_run_label, called_at);
-- Index for anomaly detection
CREATE INDEX idx_audit_called_at ON agent_spend_audit(called_at, vendor);

Four SQL queries that answer the observability questions

-- 1. Per-run cost summary (last 24h)
SELECT
  agent_run_label,
  vendor,
  SUM(cost_usd) AS total_cost_usd,
  COUNT(*) AS call_count,
  COUNT(*) FILTER (WHERE policy_verdict = 'cap_exhausted') AS cap_hits
FROM agent_spend_audit
WHERE called_at > NOW() - INTERVAL '24 hours'
  AND policy_verdict = 'allowed'
GROUP BY agent_run_label, vendor
ORDER BY total_cost_usd DESC;

-- 2. Fleet-wide spend by agent today
SELECT
  split_part(agent_run_label, '/', 1) AS agent_name,
  SUM(cost_usd) AS total_cost_usd,
  COUNT(DISTINCT split_part(agent_run_label, '/', 2)) AS run_count
FROM agent_spend_audit
WHERE called_at >= CURRENT_DATE
GROUP BY 1
ORDER BY total_cost_usd DESC;

-- 3. Per-vendor daily trend (30d)
SELECT
  DATE_TRUNC('day', called_at) AS day,
  vendor,
  SUM(cost_usd) AS daily_spend_usd
FROM agent_spend_audit
WHERE called_at > NOW() - INTERVAL '30 days'
GROUP BY 1, 2
ORDER BY 1, 2;

-- 4. Spend anomaly detection (>3x 30-day hourly baseline)
WITH hourly AS (
  SELECT
    DATE_TRUNC('hour', called_at) AS hour,
    SUM(cost_usd) AS hourly_spend
  FROM agent_spend_audit
  WHERE called_at > NOW() - INTERVAL '30 days'
  GROUP BY 1
),
baseline AS (
  SELECT AVG(hourly_spend) AS avg_hourly
  FROM hourly
  WHERE hour < NOW() - INTERVAL '2 hours'
)
SELECT
  h.hour,
  h.hourly_spend,
  b.avg_hourly,
  ROUND(h.hourly_spend / NULLIF(b.avg_hourly, 0), 1) AS multiplier
FROM hourly h, baseline b
WHERE h.hour >= NOW() - INTERVAL '3 hours'
  AND h.hourly_spend > b.avg_hourly * 3
ORDER BY h.hour DESC;

How cost is parsed from vendor responses

The proxy reads vendor API responses before forwarding them to the caller and extracts cost signals from response fields:

How Keybrake fits

Keybrake is the proxy positioned between your agent services and vendor APIs. Every call that flows through the proxy is logged to the audit table with cost_usd parsed from the vendor response, agent_run_label from the vault key's metadata, and vendor_txn_id from the vendor's response (Stripe PaymentIntent.id, Twilio SID). No custom span attributes, no SDK changes in your agent code, no post-hoc data pipeline joining APM telemetry with vendor dashboards. The dashboard at keybrake.com/app surfaces today's spend per vendor, recent calls with cost breakdown, and cap-hit rate. The audit log is queryable SQL for teams that want to build custom reports or wire alerting on anomaly detection queries.

Get early access

Related questions

Further reading

• AI agent spend reporting — the four SQL reporting queries that answer per-run, fleet-wide, trend, and anomaly questions from the proxy audit log.
• AI agent policy enforcement — how runtime spend caps complement observability: observability tells you what happened, caps prevent it from happening in the first place.
• AI agent audit trail — the compliance and forensic requirements that the proxy audit log satisfies beyond just spend visibility.
• AI agent cost management — the relationship between observability (seeing what happened) and cost management (controlling what can happen).