Inference Gateway

Spec sheet

Purpose

Inference Gateway is an open-source, cloud-native proxy that provides one OpenAI-compatible API for multiple hosted and local LLM providers.Read 10 Oct 2026 · docs.inference-gateway.com

Product

Inference Gateway is an open-source, cloud-native proxy that presents supported hosted and local LLM providers through one OpenAI-compatible API.Read 9 Oct 2026 · docs.inference-gateway.com

Deployment

The getting-started guide documents Docker, Docker Compose examples, and Kubernetes deployment.Read 10 Oct 2026 · docs.inference-gateway.com

Privacy

The project states that it has no analytics or telemetry phoning home and can be self-hosted on-premises, in the cloud, or air-gapped.Read 10 Oct 2026 · docs.inference-gateway.com

License

The documentation states that Inference Gateway is released under the Apache-2.0 license.Read 9 Oct 2026 · docs.inference-gateway.com

Agent coordination

Its Agent-to-Agent support lets agents discover capabilities, delegate tasks, and stream results within a conversation.Read 10 Oct 2026 · docs.inference-gateway.com

Agent workflows

The product supports Agent-to-Agent coordination and describes defining agents in YAML with a CLI that generates Go, Rust, or TypeScript projects.Read 9 Oct 2026 · docs.inference-gateway.com

Authentication

Authentication supports OIDC identity providers, with JWT validation against the issuer’s JWKS.Read 10 Oct 2026 · docs.inference-gateway.com

Desktop platforms

The desktop app page lists Linux, macOS, and Windows assets for amd64 and arm64; it also says releases are not signed with Apple Developer or Windows code-signing certificates.Read 9 Oct 2026 · docs.inference-gateway.com

Footprint

The gateway is described as an approximately 10.8 MB static binary designed to scale horizontally with Kubernetes HPA.Read 10 Oct 2026 · docs.inference-gateway.com

Kubernetes

A Kubernetes Operator manages gateways, agents, MCP servers, and chat orchestrators declaratively as cluster resources.Read 10 Oct 2026 · docs.inference-gateway.com

License and support

The project is released under Apache-2.0, with GitHub Discussions for questions and an issue tracker for bugs and feature requests.Read 10 Oct 2026 · docs.inference-gateway.com

MCP integration

It can discover tools from connected MCP servers and execute tool calls server-side.Read 10 Oct 2026 · docs.inference-gateway.com

Observability

Built-in observability includes Prometheus metrics, OTLP tracing, structured JSON logs, and reference Grafana dashboards.Read 10 Oct 2026 · docs.inference-gateway.com

Provider coverage

The documentation lists providers including OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, Nvidia, and llama.cpp.Read 10 Oct 2026 · docs.inference-gateway.com

Provider switching

The gateway lets applications switch providers by changing configuration, without an application redeploy, and can route model aliases to pools of upstream deployments.Read 9 Oct 2026 · docs.inference-gateway.com

Routing

The gateway supports model routing so a stable model alias can point to a pool of upstream deployments.Read 10 Oct 2026 · docs.inference-gateway.com

SDKs

Official SDKs are available for Python, TypeScript, Go, and Rust, with streaming support over Server-Sent Events.Read 10 Oct 2026 · docs.inference-gateway.com

Where Specifiction places it

• Tool Calling#13 of 217.5
• LLM Gateway Software#24 of 347.5
• LLM gateways#30 of 337.0