One gate for every model you call.
AirLLM sits between everything you build and every model you use. Point your apps, agents, and internal tools at one endpoint — it routes each request to OpenAI, Anthropic, Groq, or a model you run yourself, falls back automatically when one is down, meters spend per key, and scans what goes through for secrets on the way.
Every request in — one policy, one bill, one place to watch. Every request out — wherever it actually needs to go.
Every provider means another integration
Add a model and you're back in the SDKs: a different auth scheme, different rate limits, different error shapes, no shared record of who called what. Multiply that across every app and agent your team ships, and nobody has one place to see spend, enforce a policy, or catch a secret before it leaves the building — until the provider that's pinned everywhere has an outage, and every one of those integrations goes down with it.
If you've used OpenRouter or LiteLLM for this, the idea is familiar — one API in front of many providers. AirLLM is the same shape, run on infrastructure you own: a single Go binary, your Postgres, your Redis, your keys never leaving your network.
What sits behind that one endpoint
A single Go service speaks the OpenAI and Anthropic wire formats your apps, agents, and tools already use — so nothing downstream has to change, only what's behind the endpoint does. It authenticates by API key, enforces per-key model policy and rolling usage limits, routes to upstream providers with balancing and fallback, and meters tokens and cost. It can scan prompts for secrets and PII before they leave, capture traffic for audit, and feed reviewer-corrected captures back into the detector. A console, served by the same binary, handles keys, usage, and administration.
Three ways to run it
All public-clean — secrets are supplied at deploy time, never committed.
Dev / mock
Local development against a mock provider — no real keys needed.
make compose-up
Standalone prod
Single host: persistent data, a permanent admin, optional auto-HTTPS.
make gen-secrets make compose-prod-up
Kubernetes
HPA/KEDA autoscaling, secrets by reference, ArgoCD-friendly.
helm install airllm deploy/helm/airllm \
--set image.repository=\
ghcr.io/ipsupport-llc/ipsupport-airllm
What it does once it's running
Dual ingress
OpenAI (/v1/chat/completions, /v1/models) and Anthropic (/v1/messages) — non-streaming and SSE, translated across protocols when a client and its upstream don't match.
Routing and balancing
An alias catalog with priority fallback tiers and per-provider concurrency; explicit provider/model passthrough when a key's policy allows it. Every provider at capacity means a 429, never a crash.
Policy and limits
Per-role allowed models, snapshotted onto each key at issue. Rolling 5-hour, 24-hour, and 7-day windows, tracked in both tokens and dollars.
Secret and PII scanning
A deterministic layer (regex and entropy) plus an opt-in BERT model for fuzzier matches — off, flagged, redacted, or blocked, your choice. Prompts only, by design.
Capture, review, and audit
Traffic captured redacted-by-default, a second pass that hunts for what the first one missed, a review queue, JSONL export for retraining, and every transcript view logged under its own auditor role.
Observability
Prometheus metrics, per-component latency, sparklines right in the console, and Grafana dashboards checked into the repo as code.
Autoscaling
The secret-scanning sidecar pool scales on its own saturation signal — HPA or KEDA, including scale-to-zero, on Kubernetes.
Your own auth
Generic OIDC (Okta, Zitadel, anything that speaks it) or local accounts. Provider credentials sealed with AES-GCM; nothing secret ever sits in the repo.
Try it in three steps
make compose-up — builds and starts Postgres, Redis, and the gateway on 127.0.0.1:8080.# OpenAI-compatible curl http://127.0.0.1:8080/v1/chat/completions \ -H "Authorization: Bearer <your-token>" \ -d '{"model":"mock-gpt","messages":[{"role":"user","content":"hello"}]}'