Latency-based LLM routing

sturnus is a sidecar that speaks the OpenAI API and sends each request to whichever of your providers is fastest and healthy right now. One static binary. No Redis, no database, no control plane.

Install Source on GitHub

RustMIT

Requests flock to the fastest healthy provider. Slower ones keep a small share as a probe, so a recovered provider wins traffic back on its own.

The only code change needed is the base URL

- client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
+ client = OpenAI(base_url="http://127.0.0.1:4000/v1", api_key="unused")

response = client.chat.completions.create(
    model="fast",  # alias, resolved to the fastest candidate
    messages=[{"role": "user", "content": "Hello"}],
)

The model name alias (fast in the example above) maps to a list of interchangeable provider and model pairs in a small TOML file. Traffic is dynamically redistributed across them based on their live latency.

How routing works

Measure. Every response feeds two exponentially weighted moving averages per candidate: time to first chunk, and success rate.
Score. Effective latency is calculated as latency divided by success rate. Slow or flaky providers score worse.
Weight. Each candidate gets a share proportional to (best / its_score)^k. The best takes most of the traffic. Worse ones keep a shrinking share, floored at 1%.
Probe by default. That floor ensures every provider gets a live measure. When a slow or erroring provider recovers, it wins traffic back automatically.

Each sidecar routes from what it observes locally, so there is no shared state to run and nothing to keep in sync across pods.

What you get

Transparent passthrough: Only the model field is rewritten. The rest of the body is forwarded byte for byte, keeping key order, number precision and formatting. Streaming responses are relayed as chunks arrive.
Session affinity without state: Every response carries an x-session-affinity header. Send it back to pin a multi-turn conversation to the same provider, across any pod. The pin breaks itself if that provider starts failing.
Memory bounded: Request buffers are capped per request and in aggregate. Bursts beyond the budget are shed with 429 and Retry-After instead of an OOM kill.
Providers: OpenAI, Anthropic, Groq, Azure OpenAI, Google AI Studio, any OpenAI-compatible server, and Vertex AI with GKE Workload Identity and automatic token refresh.
Observability: Prometheus metrics for requests, time to first chunk, latency and errors, labelled by alias, provider and model. Structured JSON logs with W3C traceparent propagation. A /status endpoint that shows the live scores. An optional second listener so metrics can be scraped without exposing the proxy.
Small and auditable: A single static binary published as a scratch container for amd64 and arm64. MIT licensed. Runs entirely inside your infrastructure.

Install

Run with docker or test locally via cargo.

sturnus needs a config.toml: start from config.example.toml and add your providers.

Docker

docker run -v ./config.toml:/config.toml \
  -p 4000:4000 \
  ghcr.io/sturnus-dev/sturnus:latest

Cargo

cargo install sturnus
sturnus --config config.toml

Binary

Static builds for Linux and macOS, x86_64 and aarch64, are attached to every release.

A minimal config

listen = "127.0.0.1:4000"

[provider.openai]
base_url = "https://api.openai.com/v1"
api_key = "${OPENAI_API_KEY}"

[provider.vertex]
vertex_ai = { project_id = "my-gcp-project", location = "us-central1" }

[model]
fast = [
  { provider = "openai", model = "gpt-4o-mini" },
  { provider = "vertex", model = "google/gemini-2.5-flash" },
]

On Kubernetes, run it as a native sidecar bound to loopback. The full configuration reference, endpoint list and metrics table are in the README.