vLLM Semantic Router

Spec sheet

Purpose

vLLM Semantic Router is an open, programmable decision layer that selects or combines models for each call according to policy.Read 10 Oct 2026 · github.com

Product

vLLM Semantic Router describes itself as an open, programmable decision layer for models and compute.Read 11 Oct 2026 · vllm-sr.ai

Support

The site links to GitHub Discussions as a community resource.Read 11 Oct 2026 · vllm-sr.ai

Agent use

Agent harnesses connect to the Router inference endpoint while retaining responsibility for the agent loop, tools, and task state.Read 10 Oct 2026 · github.com

API compatibility

The Quickstart demonstrates requests through an OpenAI-style /v1/chat/completions endpoint.Read 10 Oct 2026 · vllm-sr.ai

Client protocols

The documentation lists OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages as client-facing protocols.Read 11 Oct 2026 · vllm-sr.ai

Data retention

The security guide says replay, response cache, memory, response history, service logs, and provider logs can retain request-derived data, and recommends setting retention and deletion policies for enabled stores.Read 11 Oct 2026 · vllm-sr.ai

Deployment choices

The documentation describes cloud, data center, edge, and enterprise hybrid deployment environments.Read 10 Oct 2026 · vllm-sr.ai

Deployment limits

The Router chooses a model or model pool, while the inference platform handles replica placement, batching, autoscaling, cache locality, and accelerator scheduling.Read 10 Oct 2026 · vllm-sr.ai

Deployment locations

The site says users configure where inference runs, including cloud, private, or edge environments.Read 11 Oct 2026 · vllm-sr.ai

Installation

The quickstart lists Linux, macOS, or WSL2, Python 3.10 or newer, and Docker as requirements, with Podman available as a Linux fallback.Read 11 Oct 2026 · vllm-sr.ai

Local and remote models

It can keep eligible sensitive or offline traffic on local models and allow selected requests to escalate to approved remote models.Read 10 Oct 2026 · vllm-sr.ai

Model backends

The site names vLLM, SGLang, and OpenAI API among the model backends it can route to.Read 11 Oct 2026 · vllm-sr.ai

Model routing

It can route requests by task, capability, modality, context length, and tool requirements across configured model backends.Read 10 Oct 2026 · vllm-sr.ai

Open source license

The public GitHub repository lists an Apache-2.0 license.Read 10 Oct 2026 · github.com

Playground

The project provides an online Playground at app.vllm-sr.ai/playground.Read 10 Oct 2026 · github.com

Privacy policy

The site describes routing private domain knowledge to open models on on-premises GPUs while allowing policy-eligible requests to use external frontier models.Read 11 Oct 2026 · vllm-sr.ai

Project team

The project team page lists five maintainers responsible for project-wide technical and operational stewardship.Read 11 Oct 2026 · vllm-sr.ai

Protocol limitations

The compatibility matrix says custom free-form tools are not supported for Anthropic Messages, and audio input is supported only for Chat Completions.Read 11 Oct 2026 · vllm-sr.ai

Requirements

The Quickstart lists Linux, macOS, or WSL2, Python 3.10 or newer, and Docker, with Podman available as a Linux fallback.Read 10 Oct 2026 · vllm-sr.ai

Routing

It selects a model backend per request using signals that can include context, intent, safety, preferences, and system state.Read 11 Oct 2026 · vllm-sr.ai

Routing algorithms

The site lists 16 algorithms: 11 selection algorithms and 5 loopers.Read 11 Oct 2026 · vllm-sr.ai

Routing modes

It offers a Router mode for routing chat to model backends and an Engine mode that disables routing so the application owns the next step.Read 11 Oct 2026 · vllm-sr.ai

Routing priorities

Operators can set quality, latency, and cost priorities for model selection.Read 10 Oct 2026 · vllm-sr.ai

Security boundary

The documentation says the Router complements identity, network segmentation, secrets management, retention policy, and provider governance rather than replacing them.Read 10 Oct 2026 · vllm-sr.ai

Security guidance

The security guide advises authenticating public inference endpoints, removing internal control and identity headers at a trusted proxy, and keeping management and storage ports private.Read 11 Oct 2026 · vllm-sr.ai

Support and community

The project links to GitHub Discussions and describes maintainers as responsible for technical escalations and releases.Read 10 Oct 2026 · vllm-sr.ai

Where Specifiction places it

• LLM gateways#26 of 337.3
• LLM Gateway Software#31 of 347.2