Project Moonshot vs Promptfoo

Project Moonshot is an open-source toolkit for evaluating the safety and reliability of large language models and applications. It runs benchmark tests using open-source datasets and metrics, and generates adversarial prompts to probe possible vulnerabilities and misuse. Users can build custom evaluations with their own datasets, optional prompt templates, metrics, and grading scales. Results appear in interactive HTML reports and downloadable raw JSON. Moonshot provides a Web UI, interactive command-line interface, library APIs, and Web APIs, with configurable connections to model providers including OpenAI, Anthropic, Together, and Hugging Face. Users can also create connectors for other models or applications hosted on custom servers. The toolkit implements benchmarks recommended in IMDA’s Starter Kit for safety testing LLM-based applications. Moonshot is free beta software under Apache License 2.0. Installation uses pip and requires Python 3.11; the Web UI additionally requires Node.js 20.11.1 LTS or later, and test assets are needed to run tests. The installation guide recommends Chrome for the Web UI and notes possible installation difficulties on x86 Macs with moonshot-data dependencies.

Promptfoo helps developers and enterprises evaluate prompts and test AI applications for security risks. It compares prompt outputs across multiple models in a web view. Red-team testing simulates users to look for prompt injections, jailbreaks, data leaks, business rule violations and insecure agent tool use. Its getting-started guide lists more than 60 providers, including OpenAI, Anthropic, Google and local models such as Ollama. Integrations include CI systems such as GitHub Actions, GitLab CI, Jenkins, Azure Pipelines, CircleCI and Bitbucket Pipelines, plus Splunk, Burp Suite, Google Sheets and MCP. Installation is available through npm, npx or Homebrew. Promptfoo requires Node.js 22.22.0 or newer and recommends Node.js 24 LTS. The free Community plan includes all LLM evaluation features and 10k red-team probes per month, and can run locally or be self-hosted. Enterprise includes managed cloud deployment; On-Premise runs on customer infrastructure. Promptfoo says it is based in San Francisco, is now part of OpenAI and remains open source.