Overview

Promptfoo brings prompt evaluation and red-team testing together, with support for hosted and local models. Founded in 2024 and based in San Francisco, it says it is now part of OpenAI while remaining open source. Community runs locally or self-hosted; Enterprise adds managed cloud deployment, and On-Premise runs on the customer’s infrastructure.

Key features

Evaluate prompts across models

Promptfoo evaluates prompts across multiple models and provides a web view for comparing outputs. Support for 60+ providers includes OpenAI, Anthropic, Google, and local models such as Ollama. That range is useful when a team needs to compare providers in one evaluation workflow rather than commit to a single model ecosystem.

Community includes all LLM evaluation features, custom metrics, LLM-as-a-judge, safety evaluations, human review workflows, and CI/CD integration. Integrations include GitHub Actions, GitLab CI, Jenkins, Azure Pipelines, CircleCI, Bitbucket Pipelines, Splunk, Burp Suite, Google Sheets, and MCP. This makes the free plan capable of supporting development and security workflows; teams should confirm that its specific integrations match their tooling.

Probe for risks

Red-team runs simulate users to look for prompt injections, jailbreaks, data leaks, business-rule violations, and insecure agent tool use. Community allows 10k probes per month, with each probe counting as one request to the target system. That is a meaningful allowance for smaller-scale or periodic testing, but frequent testing across multiple targets can make the cap a constraint. Enterprise offers custom red-teaming limits and continuous monitoring for teams needing more capacity or ongoing checks.

Installation and security

Promptfoo can be installed with npm, npx, or Homebrew. It requires Node.js 22.22.0 or newer and recommends Node.js 24 LTS, a notable prerequisite for teams standardizing on an older runtime. Homebrew supports Mac and Linux. The site displays SOC 2 and ISO 27001 certification badges.

Pricing

Community: 0.00 USD per free. This plan includes all LLM evaluation features, all model providers and integrations, local or self-hosted deployment, community support, and 10k red-team probes per month. It is the clear fit for individual developers and teams whose testing stays within that allowance and who do not need enterprise support or managed deployment.

Enterprise: custom pricing. Pricing is based on team size and needs. The plan adds custom red-teaming limits, team sharing, continuous monitoring, SSO, API access, managed cloud deployment, priority support, and SLA guarantees. It suits organizations that need shared workflows, ongoing monitoring, or a managed service; the trade-off is a custom-priced commitment rather than a published rate.

On-Premise: custom pricing. It includes Enterprise features plus deployment on the customer’s infrastructure, complete data isolation, a dedicated runner, and an assigned deployment engineer. This is the appropriate option where infrastructure control and isolation are central requirements, but it carries custom pricing.

Platforms

Promptfoo supports API, Linux, macOS, self-hosted, web, and Windows. Its deployment choices span local and self-hosted use through Community, managed cloud with Enterprise, and customer-infrastructure deployment with On-Premise. The broad range accommodates different operating and hosting preferences, though installation requires the stated Node.js version.

Who it's for

Promptfoo is a strong fit for developers and AI teams that need both model evaluation and adversarial testing, particularly when they use several hosted or local providers and want evaluation in CI/CD. Community is suited to teams comfortable with local or self-hosted operation and a fixed probe allowance. Organizations needing SSO, continuous monitoring, SLAs, managed cloud, or full infrastructure isolation should consider a paid plan. It is less suitable for teams unable to meet the Node.js requirement or whose red-team volume exceeds Community’s monthly cap without custom limits.

Pros and cons

Broad evaluation coverage at no cost: Community includes all evaluation features, providers, and integrations, alongside custom metrics, safety evaluation, and human review workflows.
Evaluation and security testing in one product: Teams can compare model outputs and probe for risks including insecure agent tool use without separating those tasks across tools.
Flexible deployment: Local, self-hosted, managed cloud, and customer-infrastructure options serve different operational requirements.
Probe cap on the free plan: Community’s 10k monthly probes may limit frequent or broad red-team campaigns.
Runtime prerequisite: Node.js 22.22.0 or newer may add setup work for teams on older versions.
Custom-priced higher tiers: Enterprise and On-Premise provide organizational controls and deployment options, but pricing is not a fixed published amount.

Alternatives

DeepEval is a free, Apache 2.0-licensed open-source LLM evaluation framework with local and CI/CD test runners; choose it when evaluation is the priority and a standalone framework is enough.

Maxim AI offers a free Developer plan for up to 3 seats, one workspace, 10k logs per month, and three-day data retention; it may suit a small team seeking a bounded hosted workspace.

Whisper provides MIT-licensed open-source code and model weights for local installation with Python and ffmpeg; choose it when that package is what you need.

Giskard offers a free open-source library for local deployment, with a basic LLM vulnerability scan and RAG evaluation report; it is an option for teams focused on those checks.

Braintrust has a free Starter plan with 1 GB processed data, 10,000 scores, 14-day retention, and unlimited users, projects, and datasets; choose it when those evaluation allowances fit your workflow.

Galileo offers a Pro plan at 100.00 USD per month, billed yearly, with 50,000 traces per month, standard RBAC, advanced analytics, and dedicated Slack support; it may fit teams seeking those paid analytics and support provisions.

garak is a free open-source tool; consider it as an alternative if you want a free tool.

Parea AI offers a free plan at 0.00 USD per month with up to two team members, 3k logs per month, one-month retention, and 10 deployed prompts; it may suit small teams whose needs match those limits.

For a broader comparison, see LLM Evaluation Tools, AI Prompt Management Software, AI Red Teaming Tools, AI Security Testing Tools, AI LLM Evaluation Tools, and AI Agent Evaluation Tools.

Verdict

Choose Promptfoo if you want model evaluation and red-team testing together, broad provider support, and local or self-hosted use without a software charge. Its main appeal is the breadth of Community; its main caveats are the 10k monthly probe limit, the Node.js prerequisite, and custom pricing for enterprise deployment and controls. Teams that need more probes or organizational support should request Enterprise pricing, while infrastructure isolation points to On-Premise.

	Apastra	Promptfoo
Free plan	Yes	Yes
Paid from	Free	Free
Platforms	self-hosted	api, Linux, macOS, self-hosted, Web, Windows
Prompt version history	Yes
Prompt testing	Yes
Deployment controls	Yes
Free plan		Yes
Deployment options		both
Custom metrics		Yes
LLM-as-a-judge		Yes
Safety evaluations		Yes
Human review workflows		Yes
CI/CD integration		Yes

Apastra vs Promptfoo

Apastra

Promptfoo