Skip to content
SPECIFICTION
Get listed

OpenZiti LLM Gateway

github.com

WindowsmacOSLinuxAPISelf-hosted
Visit github.com
Spec score 7.1 #27 of 33 in LLM gateways

Spec sheet

Purpose
It is an OpenAI-compatible API proxy that routes requests to OpenAI, Anthropic, and compatible backends such as Ollama and vLLM.Read 10 Oct 2026 · github.com
Intended users
The README describes routing across local inference servers and remote providers, including GPU servers behind NAT and clients using OpenAI-compatible libraries.Read 10 Oct 2026 · github.com
Platforms
The README says prebuilt binaries are available for Linux, macOS, and Windows, and also documents installation with Go or from source.Read 10 Oct 2026 · github.com
Deployment
The project describes deployment as one Go binary and one YAML file, with no database, message queue, or sidecar required.Read 10 Oct 2026 · github.com
Integrations
Documented backends include OpenAI, Anthropic, Ollama, vLLM, llama-server, and SGLang; clients use an OpenAI-compatible API.Read 10 Oct 2026 · github.com
License
The repository identifies the project as Apache 2.0 licensed.Read 10 Oct 2026 · github.com
Headquarters
Charlotte, North Carolina, United StatesRead 28 Sep 2026 · github.com
Anthropic support
The gateway translates between OpenAI format and Anthropic’s Messages API.Read 9 Oct 2026 · github.com
Client authentication
Optional virtual API keys can be restricted to models using glob patterns; the health and metrics endpoints remain unauthenticated.Read 10 Oct 2026 · github.com
Load balancing
It supports weighted round-robin across inference endpoints, health checks, and failover.Read 10 Oct 2026 · github.com
Metrics
When enabled, its Prometheus endpoint reports request, latency, token, routing, provider-error, and endpoint-health metrics.Read 10 Oct 2026 · github.com
Network security
The project describes using zrok over OpenZiti to connect gateway and backend across NAT or network boundaries without firewall rules or port forwarding.Read 9 Oct 2026 · github.com
Observability
When enabled, Prometheus-exported metrics include request counts, latency, token counts, routing decisions, provider errors, and endpoint health.Read 9 Oct 2026 · github.com
Provider integrations
The project documents routing to OpenAI, Anthropic, Ollama, vLLM, llama-server, SGLang, and other OpenAI-compatible backends.Read 9 Oct 2026 · github.com
Provider translation
It translates between the OpenAI request format and Anthropic’s Messages API.Read 10 Oct 2026 · github.com
Routing
Optional semantic routing uses keyword heuristics, embedding similarity, and an LLM classifier to choose a model when the request omits one.Read 9 Oct 2026 · github.com
Semantic routing
An optional cascade of keyword heuristics, embedding similarity, and an LLM classifier can select a model when the request omits one.Read 10 Oct 2026 · github.com
Streaming
It supports server-sent event streaming for its providers.Read 10 Oct 2026 · github.com
Tracing
Optional tracing logs request metadata and truncated message content, with a configurable maximum content length.Read 9 Oct 2026 · github.com
Zero-trust networking
With zrok over OpenZiti, the gateway can expose itself or reach backends without firewall rules or port forwarding.Read 10 Oct 2026 · github.com

Where Specifiction places it

Alternatives to OpenZiti LLM GatewayAll alternatives →