Merchant Routing Engine · Architecture

Kalman Filter-Based
LLM Request Router

A cost-minimizing reverse proxy that predicts token prices and quota exhaustion using dual Kalman filter families, then routes every request to the cheapest viable provider in real time. Price is the primary signal — argmin(effective_price).

$0.001
Cheapest Rate /M
99.7%
Savings vs Seed
527M
Tokens Replayed
50,354
Decisions
Full Pipeline client → provider → log
CLIENT GATEWAY PROXY KEY SELECT FAILOVER PRICING DISPATCH PROVIDERS LOGGING Signal / Matrix Message client request hermes-gateway port 9098 · POST /v1/chat/completions zai_proxy.py — THE ORCHESTRATOR port 9099 · DQ05 primary, T470 secondary KEYS LOADER (ours, friend) HARDWARE PROBE (udevadm/ssh) QUOTA FETCHER (5min TTL) SQLite (zai_usage.db) STEP 1: best_key() — 4 Phases Phase 1: Predict quota exhaustion Kalman burn-rate forecast Phase 2: Lock keys predicted to exhaust soon Phase 3: Unlock keys whose quota window has reset Phase 4: Pick cheapest unlocked key (health as tie-breaker) → "ours" | "friend" | None if None STEP 2: LiveRouter.select_failover() emergency fallback only · enabled by file flag PriceKalman predicts true $/M from costs ConsumptionKalman predicts quota burn for failovers CPVO Calculator adjusts price for output quality → RoutingOptimizer.route() → cheapest viable STEP 3: PRICING ENGINE (pure functions) effective = base × peak(3×) × scarcity × health × pace MIN_EFFECTIVE_PRICE = $0.001/M floor (ADR-004) STEP 4: DISPATCH GATE (/v1/dispatch_gate) Dim 1: Hardware → Dim 2: Quota → Dim 3: Price → can_dispatch: true/false + recommended_model z.ai "ours" glm-5.2 · api.z.ai $0.001/M z.ai "friend" glm-5.2 · shared key $0.029/M Ollama Cloud local · $100/mo flat $0.024/M PPQ / OpenRouter per-token · deepseek-v4 $0.14/M DeepInfra last resort $1.30/M POST-REQUEST LOGGING ShadowLogger live vs shadow ProfitTracker savings vs next-best ConsumptionKalman .update(tokens) token_audit billed vs actual Kalman feedback loop → next prediction more accurate
Core Principle (ADR-001)
Price is the primary routing signal. Provider selection is always argmin(effective_price) among viable providers — no hardcoded cascade, no peak-hour routing directive. Peak hours affect routing only through the price multiplier, never through an if-statement.
02 — KALMAN FILTERS

Two Independent Filter Families

Two completely independent Kalman filter families operate per-provider, each tuned to its physics (ADR-002). They never share state. One estimates cost, the other predicts consumption — together they power the routing decision.

PriceKalman src/price_kalman.py
INPUT Observed cost /M tokens 2-STATE MODEL constant-velocity · F = [[1,1],[0,1]] base_rate $/M · smoothed cost velocity $/M/cycle · trend Q = diag(1e-6) · R = 1e-4 · H = [1, 0] cost drifts slowly → tiny process noise OUTPUTS base_rate (smoothed $/M) velocity (trend direction) effective_price() — applies multipliers CONVERGED RATES (50,354 decisions) ours $0.001/M friend $0.029/M ollama $0.024/M

Estimates the smooth component of effective cost per million tokens. Peak multiplier, scarcity, and health are deterministic functions applied on top — they are NOT Kalman inputs (ADR-003).

ConsumptionKalman src/consumption_kalman.py
INPUT Token usage per period 3-STATE MODEL constant-accel · F = [[1,dt,½dt²],[0,1,dt],[0,0,1]] burn_rate tok/period 0th deriv velocity tok/period² 1st deriv acceleration tok/period³ 2nd deriv Auto-tuned via from_history(): R = empirical variance · Q = R × 1e-3 H = [1, 0, 0] — only burn_rate observed OUTPUTS predict_horizon(n) — projected burn rate predict_cumulative(n) — total tokens will_exhaust(quota, n) → (bool, periods) uncertainty — √(P[0,0]) LIVE DATA (from proxy /quota) ours: 359K tph · friend: 19.8M tph friend 5h window: 95% used · LOCKED

Provider-agnostic predictor of token consumption and quota exhaustion. Knows nothing about price, cost, peak hours, or health. The will_exhaust() output feeds the lock decisions in Phase 1 & 2 of best_key().

03 — EFFECTIVE PRICE

The Five Multipliers

The Kalman-smoothed base rate gets multiplied by four deterministic functions (ADR-008). All multipliers are pure and side-effect-free. Any infinite multiplier makes the provider effectively unreachable.

Formula price_kalman.effective_price()
base_rate × peak_mult × scarcity × health × pace = effective_price
base_rate
from PriceKalman · $/M
The smoothed cost estimate. Converges from seed rates to amortized reality over thousands of observations.
peak_mult
1.0× → 3.0×
Instant step function during UTC 06:00–09:59 (Beijing peak). Only affects z.ai keys. Never smoothed (ADR-003).
scarcity
1.0× → 2.0×
Linear ramp from 50% quota usage to 100%. Formula: 1 + max(0, (used_pct − 50) / 50)
health
1.0× → 1.5× → 3× → 10× → ∞
Graduated penalty based on failure count. Circuit breaker at >10 failures = +∞ (unreachable).
pace
0.5× → 3.0×
Predictive pacing. clamp(ratio², 0.5, 3.0). Attracts traffic when underutilizing, slows when exhausting.
Health Penalty Scale graduated
1.0× 0 failures No penalty
1.5× 1–2 failures Soft (transient)
3.0× 3–5 failures Moderate
10× 6–10 failures Severe
+∞ >10 / breaker Unreachable
04 — REQUEST LIFECYCLE

Single Request, Step by Step

A swim-lane view showing how one request moves through the system, with decision points, branch conditions, and the Kalman feedback loop that makes each subsequent prediction more accurate.

Lifecycle Flow 7 steps · with branches
1 Request arrives at hermes-gateway (:9098) POST /v1/chat/completions → forwarded to proxy :9099 2 best_key() runs 4-phase selection Kalman prediction → locks → recovery → health filter key found? YES NO → failover path LiveRouter.select_failover() RoutingOptimizer → all providers External Failover (hardcoded) DeepInfra → PPQ → OpenRouter 3 Pricing Engine computes effective_price base × peak × scarcity × health × pace 4 Routing Optimizer evaluates all providers 5 gates: quality → health → exhaustion → scarcity → price → sort by effective_price → cheapest viable 5 Dispatch Gate (if hardware task) hardware → quota → price → can_dispatch? can dispatch? HOLD QUEUE / RETRY YES 6 Forward to chosen provider — response handling 200 + content ✓ 200 + reasoning → inject 200 empty → failover 401/403/429/402 401→exhausted · 429→backoff · 402→unfunded · all trigger next provider 7 Post-Request Logging & Kalman Update ShadowLogger live vs shadow ProfitTracker savings async Kalman.update() serving key only token_audit billed vs actual ↻ FEEDBACK LOOP Kalman state evolves next prediction sharper LEGEND Main flow Decision Success Error / Hold Feedback Animated dashed lines = active data flow · Solid = structural
05 — PROVIDER ECONOMICS

Cost Comparison

All providers ranked by converged effective price per million tokens. Flat-rate z.ai keys dominate — the Kalman filter amortizes the monthly fee across actual volume, driving effective rates to fractions of a cent.

Cost per Million Tokens log scale
Provider Details
ProviderType$/MNotes
oursFlat €155/mo$0.001Cheapest — high volume
friendFlat shared$0.02921% penalty over ours
ollamaFlat $100/mo$0.024No peak hours
openrouterPer-token$0.135deepseek-v4-flash
ppqPer-token$0.140deepseek-v4-flash
deepinfraPer-token$1.300Last resort only
Seed → Converged Evolution
Cost per Million Tokens ($/M, log scale) ours seed $0.31 $0.001 friend seed $0.375 $0.029 ollama seed $0.50 $0.024 ppq fixed $0.14 $0.14 (no Kalman)

Flat-rate providers converge dramatically as volume amortizes the fixed fee. Per-token providers have published prices — no Kalman needed.

06 — DISPATCH GATE

Three-Dimension Decision

A pure, side-effect-free function exposed at GET /v1/dispatch_gate. Governs whether hardware-dependent tasks should proceed. Checks hardware first, then quota sufficiency with predictive margins, then price as informational.

Gate Dimensions sequential · short-circuit
1 HARDWARE Binary · checked first none → always (2× margin) board → /dev/ttyACM* (4×) dual_board → 2+ boards (6×) dq05 → SSH reachable (3×) ↓ unavailable → HOLD ↓ available → proceed + override 2 QUOTA Predictive · hardware-scaled margin required = budget × margin + concurrent_burn_extra find key: remaining ≥ headroom AND used_pct < 95% AND healthy if fail → flash downgrade (×0.3) ↓ sufficient → PROCEED ↓ insufficient → HOLD 3 PRICE Informational + override peak 3× reported, never blocks scarcity_override if hw present "a board in hand beats waiting" → can_dispatch: true/false + recommended_model + price
Task Budget Multipliers
Task TypeModelBudget Mult
mechanicalglm-4.5-flash0.25×
codingglm-5.21.0×
researchglm-5.22.5×
reviewglm-5.20.5×
docsglm-4.5-flash0.5×
Live Snapshot
"can_dispatch": true,
"reason": "sufficient headroom",
"recommended_model": "glm-5.2",
"effective_price_per_m": 0.002,
"scarcity_factor": 2.0,
"recommended_provider": "friend"
07 — CONVERGED REPLAY

Historical Validation

Replay of 50,354 historical routing decisions consuming 527M tokens. The Kalman filters converged to rates 1,000× cheaper than seed estimates while maintaining 99.1% agreement with the seed-rate optimizer.

$0.55
Converged Replay Cost
vs $10,743.95 live actual
99.7%
Savings vs Seed Rate
seed estimate was $11,996.63
100%
Routed to "ours"
under converged rates
Replay Metrics
MetricValue
Decisions replayed50,354
Tokens consumed527M
Live cost (actual spend)$10,743.95
Shadow cost (seed-rate estimate)$11,996.63
Converged replay cost$0.55
Converged vs Seed savings99.7%
Live vs Converged agreement31.8%
Seed vs Converged agreement99.1%
Routing under converged rates100% → "ours"
08 — ARCHITECTURE DECISIONS

Key ADRs

The design is codified in Architecture Decision Records. These capture the why behind each non-obvious choice.

ADR-001
Price-First Routing
Price is the primary routing signal. Selection is always argmin(effective_price).
ADR-002
Multi-Kalman Separation
Separate filters per concern: cost, consumption, demand. They never share state.
ADR-003
Deterministic Peak Multiplier
Peak hours are an instant step function, NOT Kalman-smoothed.
ADR-004
Effective Price Positivity
Price is always > 0 ($0.001 floor). +∞ means unreachable.
ADR-005
Three-Layer Actor Separation
Kalman (predict) → Pricing (deterministic) → Router (argmin).
ADR-006
Shadow Mode Validation
Run optimizer in read-only parallel before going live.
ADR-008
Deterministic Multipliers
All non-cost factors applied post-Kalman as pure functions.
ADR-009
Scarcity Multiplier
Linear ramp starting at 50% quota usage (1× → 2×).
Production Deployment
DQ05 (primary proxy, port 9099) · T470 (secondary) · CVM Server publishes unified snapshots via Nostr NIP-28 + HTTP. Kill switches: .enable_live_routing, .key_disabled_ours. All shadow/logging paths are wrapped in try/except — they can never break production routing.