Sensei

Spec sheet

Purpose

Sensei is an open-source qualification engine for testing, evaluating, and certifying AI agents across professional skills.Read 9 Oct 2026 · sensei.sh

Intended users

Sensei is designed for people evaluating AI agents on real-world professional tasks using standardized benchmarks, scenarios, and measurable KPIs.Read 9 Oct 2026 · github.com

Integrations

The repository documents HTTP, stdio, OpenAI-compatible, and LangServe adapters for connecting agents.Read 9 Oct 2026 · github.com

License

Sensei is distributed under the MIT License.Read 9 Oct 2026 · github.com

Agent connection

The site shows agents connecting to Sensei over HTTP or stdio.Read 4 Oct 2026 · sensei.sh

Built-in suites

The site lists built-in suites for SDR, customer support, content writing, bartender, dungeon master, and cat interview roles.Read 9 Oct 2026 · sensei.sh

CI/CD

The repository lists CI/CD integration among Sensei's capabilities.Read 9 Oct 2026 · github.com

Community library

The home page describes a library of 70+ community-built professional evaluation suites.Read 4 Oct 2026 · sensei.sh

Custom suites

Users can create their own evaluation suites and publish them to the community marketplace.Read 9 Oct 2026 · sensei.sh

Evaluation

It evaluates agents across task execution, reasoning, and self-improvement.Read 4 Oct 2026 · sensei.sh

Evaluation layers

Sensei evaluates task execution, reasoning, and self-improvement through feedback.Read 9 Oct 2026 · sensei.sh

LLM judging

Sensei supports LLM-based and comparative judging, with OpenAI shown as an example provider.Read 9 Oct 2026 · github.com

Marketplace

The marketplace lets users discover, download, and publish AI-agent evaluation suites.Read 9 Oct 2026 · sensei.sh

Marketplace categories

Marketplace categories include engineering, sales, marketing, product, design, support, testing, analytics, compliance, and content.Read 4 Oct 2026 · sensei.sh

Reports

The repository documents machine-readable JSON results and terminal reports.Read 9 Oct 2026 · github.com

Scoring

Its evaluation pipeline produces a score from 0 to 100 and a badge decision.Read 9 Oct 2026 · sensei.sh

Security policy

The security policy supports version 0.x.x and asks vulnerability reporters to contact the maintainer by email rather than opening a public issue.Read 9 Oct 2026 · github.com

Security response

The policy says reports are acknowledged within 48 hours, initially assessed within one week, and targeted for a patch within 30 days of confirmation.Read 9 Oct 2026 · github.com

Test suites

Built-in suites cover roles including SDR, customer support, content writer, bartender, dungeon master, and cat interview.Read 4 Oct 2026 · sensei.sh

Where Specifiction places it

• AI Agent Evaluation Tools#28 of 286.6