Skip to content
SPECIFICTION
Get listed

PromptPex

microsoft.github.io

WebWindowsmacOSLinuxSelf-hosted
Visit microsoft.github.io
Spec score 7.2 #21 of 28 in AI Prompt Generators

What it is

PromptPex is an open-source tool for developers who keep AI prompts in their codebase and want to examine how models follow prompt requirements. It treats prompts as functions, extracts output rules expressed in natural language, and generates unit tests to check whether a model follows those rules. It can also evaluate prompt performance across models, using an LLM to judge whether outputs meet stated requirements. Test generation can be shaped with scenarios, expansions, custom instructions, and effort levels. The CLI accepts Markdown or text templates, Prompty files, and GitHub Models prompt YAML files. Setup lists OpenAI, Azure OpenAI, GitHub Models, and Ollama as supported providers. PromptPex is packaged as an npm CLI and can run in Docker; its setup guide also describes a local web interface. Documented integrations include OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions. Microsoft is identified as the maker. The project is intended for research, and its maker advises against downstream use without additional evaluation of robustness, safety, harm, and bias. Vague prompts can lead to suboptimal results, and rule extraction may be less accurate with complex or idiomatic language.

Spec sheet

Purpose
PromptPex is a tool for developers to explore and test AI model prompts by treating prompts as functions and generating test inputs.Read 8 Oct 2026 · microsoft.github.io
Deployment
The project README gives a Docker command for running PromptPex, and the setup guide describes a local web interface.Read 8 Oct 2026 · github.com
Integrations
PromptPex documents exports and integrations for OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions.Read 8 Oct 2026 · microsoft.github.io
Support
The transparency note invites feedback and reports of unexpected or offensive behavior by email to listed project contacts.Read 8 Oct 2026 · microsoft.github.io
Maker
The documentation footer identifies Microsoft as the maker of PromptPex.Read 8 Oct 2026 · microsoft.github.io
Automatic rule extraction
PromptPex extracts output rules written in natural language in a prompt, such as a requirement to format output as JSON.Read 8 Oct 2026 · github.com
CLI and Docker
PromptPex is packaged as an npm command-line tool and can also be run in Docker.Read 8 Oct 2026 · microsoft.github.io
Configurable generation
Test generation can be configured with scenarios, test expansions, and custom instructions.Read 8 Oct 2026 · microsoft.github.io
Feedback
The transparency note invites feedback and collaboration by email and says the team will update the repository with mitigations for reported or identified issues.Read 8 Oct 2026 · microsoft.github.io
Intended audience
The project describes its intended users as developers who keep prompts in their codebase.Read 8 Oct 2026 · microsoft.github.io
Intended use
The released open-source library is intended for research, and the maker says it should not be used in downstream applications without additional evaluation of robustness, safety, harm, and bias.Read 8 Oct 2026 · microsoft.github.io
Limitations
The maker says vague prompts can produce suboptimal results and rule extraction can vary in accuracy, especially with complex or idiomatic language.Read 8 Oct 2026 · microsoft.github.io
Model evaluation
PromptPex can evaluate prompt performance across models, and uses an LLM to assess whether model outputs meet specified requirements.Read 8 Oct 2026 · github.com
Model providers
The setup guide says users can configure OpenAI, Azure OpenAI, or Ollama credentials.Read 8 Oct 2026 · microsoft.github.io
Prompt formats
The CLI supports Markdown or text templates, Prompty files, and GitHub Models prompt YAML files.Read 8 Oct 2026 · microsoft.github.io
Rule extraction
It can extract output rules written in natural language in a prompt.Read 8 Oct 2026 · github.com
Supported model providers
The CLI setup lists OpenAI, Azure OpenAI, GitHub Models, and Ollama among supported LLM providers.Read 8 Oct 2026 · microsoft.github.io
Test generation
It generates unit test cases designed to check whether a prompt makes a model follow its output rules.Read 8 Oct 2026 · github.com
Test generation controls
Generation can be configured with scenarios, test expansions, custom instructions, and effort levels.Read 8 Oct 2026 · microsoft.github.io

Strong points and limits

Strong points

  • Generates tests from natural-language output rules
  • Supports evaluation across models
  • Can run through npm or Docker
  • Documents integrations with evaluation tools and GitHub Actions

Limits

  • Intended for research, not downstream use without further evaluation
  • Vague prompts can produce suboptimal results
  • Rule extraction may vary in accuracy

Who it suits

PromptPex is aimed at developers who maintain prompts in their codebase and want to test model adherence to output requirements. It is intended for research rather than downstream application use without further evaluation.

Verdict

PromptPex provides a structured way to generate checks for prompt requirements and evaluate model responses. Its research framing and stated extraction limitations matter when considering how to use its results.

Where Specifiction places it

Alternatives to PromptPexAll alternatives →