MLflow Prompt Optimization vs PromptWizard

MLflow Prompt Optimization is an open-source, automated service for improving prompts through evaluation and iteration. It evaluates prompts against supplied data, identifies failure patterns, and generates revised variants. The `mlflow.genai.optimize_prompts` API provides a shared interface for optimization methods including GEPA and Metaprompting. You can provide training data, scorers, custom scorers, and aggregation functions, then save resulting prompts as new Prompt Registry versions. Runs, metrics, and traces support comparison and rollback. The workflow integrates with LangChain, LangGraph, OpenAI Agent, Pydantic AI, CrewAI, AutoGen, and custom frameworks, and works with any LLM provider. It supports API, self-hosted, and web access, with prompt variables, testing, and multiple-model support. GEPA is recommended for clear metrics and quality-critical work; documentation suggests 50–100 labeled examples, or at least 100 records for GEPA. The free Apache 2.0 option can be self-hosted or managed through cloud providers.

PromptWizard is a free, MIT-licensed open-source framework for automated prompt and example optimization. It generates prompt variations, scores them, critiques their outcomes, and refines instructions over repeated rounds. It can also optimize in-context examples and synthesize examples suited to a task. The described workflows cover prompt optimization without examples, with generated examples, or with training data. Listed datasets include GSM8k, SVAMP, AQUARAT, and Instruction Induction (BBII). Custom datasets use JSONL samples with question and answer fields, and require task-specific answer extraction and evaluation functions, configuration, and data files. PromptWizard can generate chain-of-thought reasoning for in-context examples, with an option to turn that off to reduce prompt length. Installation uses the GitHub repository as a Python package in development mode, with instructions for Windows, macOS, and Linux. Model access setup supports OpenAI API keys and Azure OpenAI endpoints, so access requires credentials for those services. The README reports average optimization times of around 20–30 minutes in experiments on the listed datasets, with timing dependent on the dataset, and notes that human supervision can help tune outputs.