Pythia

Spec sheet

Purpose

Pythia is an automated prompt optimization engine that iteratively evaluates prompts, analyzes failures, and synthesizes improvements.Read 11 Oct 2026 · clai-group.github.io

Intended users

The project presents Pythia for clinical prompt optimization and also gives a general example involving home-buying inquiries.Read 11 Oct 2026 · github.com

Support

The project links to its GitHub repository issues for questions and issue reporting.Read 11 Oct 2026 · clai-group.github.io

Backends

The repository README says Pythia includes Gemini and Ollama backends and can use another backend whose LLM is called with an .invoke() method.Read 5 Oct 2026 · github.com

Clinical focus

The project describes its optimization as clinical and sensitive to sensitivity and specificity.Read 11 Oct 2026 · clai-group.github.io

Configurable thresholds

The README says sensitivity and specificity thresholds are configurable and default to 0.75, while the priority metric defaults to specificity.Read 5 Oct 2026 · github.com

Data requirements

The documented workflow requires development and validation datasets, an output directory, an SOP, and an initial prompt.Read 11 Oct 2026 · github.com

Dataset format

The README recommends CSV datasets, with each CSV representing a person and rows representing visits or notes, using “visit” and “Ground Truth” columns.Read 5 Oct 2026 · github.com

Error analysis

A specialist LLM analyzes failures in aggregate, identifies root causes, and proposes improvement hypotheses.Read 11 Oct 2026 · clai-group.github.io

Evaluation

Pythia evaluates prompts against a held-out dataset using configurable metrics and tracks results across iterations.Read 11 Oct 2026 · clai-group.github.io

Framework

The site says Pythia uses LangGraph stateful agentic workflows.Read 5 Oct 2026 · clai-group.github.io

History and export

Pythia logs prompt versions, scores, and controller decisions, and the site says users can compare runs, export results, and roll back to earlier checkpoints.Read 5 Oct 2026 · clai-group.github.io

Hosting and privacy

The project describes Pythia as locally hosted and privacy-preserving, with optimization running on open-source models within the institution.Read 11 Oct 2026 · clai-group.github.io

Iteration control

A controller can continue, backtrack to a stronger prior prompt, or halt when the prompt has converged.Read 11 Oct 2026 · clai-group.github.io

Local hosting

The maker describes Pythia as locally hosted and privacy-preserving, with optimization running on open-source models within the institution.Read 5 Oct 2026 · clai-group.github.io

Model backends

The repository says Pythia includes Gemini and Ollama backends and can use another LLM backend that provides an invoke() method.Read 11 Oct 2026 · github.com

Model scale

The site says optimization can run on models ranging from one billion to one trillion parameters.Read 5 Oct 2026 · clai-group.github.io

Prompt improvements

An improver agent generates and ranks multiple candidate prompt rewrites before evaluation.Read 11 Oct 2026 · clai-group.github.io

Run history

Pythia logs prompt versions, scores, and controller decisions, and supports comparing runs, exporting results, and rolling back to checkpoints.Read 11 Oct 2026 · clai-group.github.io

Usage limit

The repository warns that Pythia is API-call heavy and says fewer iterations or a smaller dataset can reduce token costs and computation.Read 11 Oct 2026 · github.com

Workflow

Its workflow evaluates prompts on a held-out dataset, analyzes failure patterns, proposes prompt rewrites, and controls whether to continue, backtrack, or stop.Read 5 Oct 2026 · clai-group.github.io

Where Specifiction places it

• AI Prompt Generators#25 of 286.7