How valite Works

The gateway is a transparent reverse proxy. The first path segment of each request names the provider (/openai/…, /anthropic/…), and everything after it is forwarded to that provider byte-for-byte — streaming, tool calls, and all. Because nothing is re-implemented, anything the provider's API can do works through valite unchanged.

Logging happens off the hot path: the response is relayed to you first and parsed for the log afterwards, so the gateway adds zero time-to-first-token. Token counts and the served model come from the provider's own usage metadata, including on streamed responses.

A callsite is a distinct place in your code that calls a model — the same prompt template, tools, and output shape, with different values filled in. valite discovers callsites by fingerprinting request bodies: volatile values are masked, and the stable structure that remains identifies the callsite.

Callsites are the unit of every decision valite makes. A candidate model is proven (or rejected) per callsite, and a routing rule only ever rewrites the model for callsites where the evidence passed.

A workflow is a unit of work in your product — an agent task, a document pipeline, a support reply. A run is one execution of it. valite groups calls into runs so it can measure what actually matters: cost per run and quality per run, not per isolated API call.

You attribute calls to workflows via a path segment in the base URL, an x-valite-workflow header, or W3C traceparent propagation — see Workflows & Runs. Once a workflow reaches its required number of complete runs (100 by default, configurable), the pipeline has enough baseline to work with.

Analysis runs as seven jobs over the call log, each writing a durable artifact. Later stages read earlier artifacts, so the chain of evidence from raw traffic to routing decision is inspectable at every step:

profile   → what does this workflow cost today, per run?
goldenset → freeze a stratified, representative sample per callsite
calibrate → A/A-test the judge on your own traffic before trusting it
replay    → re-send golden requests to candidates; the judge grades each
plan      → cheapest candidate whose worse-rate clears the ceiling, per callsite
validate  → judge live routed calls against fresh incumbent regenerations
report    → human-readable scorecard of the whole run

Each job is started and polled over HTTP — see the Jobs API. Calibrate, replay, and validate spend real provider money, so they are budget-gated: without an explicit budget the job fails fast with a cost projection instead of running.

Load balancers make reactive, per-request decisions against live traffic. valite makes deterministic, pre-proven decisions: routing rules are computed offline from replay evidence, reviewed by you, and applied explicitly. On the data plane a rule is a simple lookup — matched callsites get their model field rewritten, everything else passes through untouched.