Skip to content
SPECIFICTION
Get listed

Future AGI AI Evaluation SDK

futureagi.com

From $250/mo Free plan Open source WebWindowsmacOSLinuxAPISelf-hosted
Visit futureagi.com
Spec score 9.3 #2 of 28 in AI Agent Evaluation Tools

What it is

Future AGI AI Evaluation SDK is an open-source platform for evaluating, simulating, optimizing, monitoring, and protecting AI agents. Its evaluation options include heuristic, code, LLM-as-judge, and agentic methods, with templates and CI/CD support. traceAI instrumentation converts LLM calls, tool use, retrieval, and chain steps into OpenTelemetry spans. Simulation supports text and voice tests using personas and scenarios; the Free plan includes 1M text simulation tokens and 60 voice minutes per month. The integrations page lists 79 integrations and SDK packages for Python, TypeScript, and Java, including OpenAI, Anthropic, Google GenAI, Vertex AI, AWS Bedrock, and Azure OpenAI. The Apache 2.0 platform offers Docker and SDK self-hosting options. Future AGI says Docker Compose self-hosting keeps traces, datasets, evaluations, and model calls within the customer's network. The platform is presented for startups and enterprise teams building and operating AI agents, with API, web, self-hosted, Linux, macOS, and Windows options. Free usage pauses at its cap; pay-as-you-go usage continues and may incur overage charges.

Spec sheet

What it does
Future AGI is an AI reliability platform for simulating, evaluating, protecting, observing, optimizing and routing AI agents.Read 15 Oct 2026 · futureagi.com
Purpose
Future AGI describes itself as an open-source platform for simulating, evaluating, optimizing, monitoring, and protecting AI agents.Read 29 Sep 2026 · futureagi.com
Audience
The site presents the product for startups and enterprise teams building and operating AI agents.Read 29 Sep 2026 · futureagi.com
Integrations
Future AGI documents integrations with OpenAI, LangChain, LlamaIndex and more than 30 frameworks.Read 15 Oct 2026 · docs.futureagi.com
Security
Future AGI states SOC 2 Type II, ISO 27001, GDPR, CCPA and HIPAA certification, with ISO 42001 in progress.Read 15 Oct 2026 · futureagi.com
Support
The pricing page lists community support for Free and email support for Pay-as-you-go; Scale includes a Slack channel.Read 29 Sep 2026 · futureagi.com
License
The public platform repository identifies the core as Apache License 2.0 and self-hostable.Read 15 Oct 2026 · github.com
Headquarters
San Francisco, California, United States; Bengaluru, Karnataka, IndiaRead 28 Sep 2026 · futureagi.com
Company
Future AGI Inc. is privately held and headquartered in San Francisco.Read 15 Oct 2026 · futureagi.com
API scope
The API reference covers simulations, evaluations, scenarios, personas, datasets, annotations, test runs and execution analytics.Read 15 Oct 2026 · docs.futureagi.com
Built-in evaluators
The documentation lists 132 ready-made evaluators covering quality, safety, retrieval, format and related categories.Read 15 Oct 2026 · docs.futureagi.com
Documentation depth
The docs include Python SDK references, a complete REST API reference, integration guides, cookbooks and an MCP server.Read 15 Oct 2026 · docs.futureagi.com
Documentation limitation
Future AGI describes its documentation as a work in progress and asks readers to submit feedback on pages.Read 15 Oct 2026 · docs.futureagi.com
Enterprise deployment
The enterprise page offers managed cloud, private cloud in the customer's AWS, GCP, or Azure account, and air-gapped on-premise deployment.Read 29 Sep 2026 · futureagi.com
Evaluation
Its evaluation product includes heuristic, code, LLM-as-judge, and agentic evaluations, with templates and CI/CD support.Read 29 Sep 2026 · futureagi.com
Evaluation layer
The evaluation system scores AI responses against user-defined quality goals and supports text and voice inputs.Read 15 Oct 2026 · docs.futureagi.com
Founded and size
LinkedIn lists Future AGI as founded in 2024 with 11–50 employees.Read 15 Oct 2026 · linkedin.com
Free-plan limit
The pricing FAQ says free-plan usage pauses at its cap, while pay-as-you-go usage continues and incurs overage charges.Read 29 Sep 2026 · futureagi.com
Funding
Preqin reports a $1.6 million February 2025 pre-seed round co-led by Powerhouse Ventures and Snow Leopard Ventures.Read 15 Oct 2026 · preqin.com
Limit: deployment maturity
Another independent review cites nightly-release warnings, rough edges and no official Kubernetes or Helm support at its review date.Read 15 Oct 2026 · andrew.ooo
Limit: Gartner complaints
The Gartner review page lists UI and console issues, a security gap and performance impact among dislikes.Read 15 Oct 2026 · gartner.com
Limit: integrations and exports
G2 reviewers ask for more BI exports and annotation-tool integrations.Read 15 Oct 2026 · g2.com
Limit: self-host operations
An independent review reports that self-hosting can involve roughly 17 Docker containers and substantial ClickHouse/Kafka operations work.Read 15 Oct 2026 · hysenlabs.com
Limit: setup and usability
G2 reviewers report trial-and-error during setup, unclear initial documentation and reduced intuitiveness for nontechnical users.Read 15 Oct 2026 · g2.com
Limit: tracing coverage
A G2 reviewer says some classic machine-learning models could not be traced.Read 15 Oct 2026 · g2.com
Local metrics
The SDK documentation says more than 76 local metrics run without a Future AGI API key.Read 15 Oct 2026 · docs.futureagi.com
Notable news: self-hosting
The October 13, 2026 changelog announced a signed Helm chart and a standalone self-host deployment of about three containers using roughly 1 GB idle memory.Read 15 Oct 2026 · futureagi.com
Open source
The homepage describes the platform as Apache 2.0 licensed and provides Docker, Python SDK, and Node SDK self-hosting options.Read 29 Sep 2026 · futureagi.com
Praise: error detection
G2 reviewers repeatedly praise catching hallucinations, toxicity, policy violations and edge-case failures before production.Read 15 Oct 2026 · g2.com
Praise: evaluation workflow
G2 reviewers describe the evaluation framework as intuitive for model selection, with one reporting double-digit improvement in online product metrics.Read 15 Oct 2026 · g2.com
Praise: Product Hunt
Product Hunt reviewers praise automated quality checks, custom metrics and rapid detection of unknown model problems.Read 15 Oct 2026 · producthunt.com
Praise: Trustpilot reviewer
The sole Trustpilot reviewer praises the fluid interface and one-stop coverage of the generative-AI lifecycle.Read 15 Oct 2026 · trustpilot.com
Provider connections
The platform can connect OpenAI, Anthropic, AWS Bedrock, Azure OpenAI and custom model endpoints.Read 15 Oct 2026 · docs.futureagi.com
Provider integrations
Featured integrations include OpenAI, Anthropic, Google GenAI, Vertex AI, AWS Bedrock, and Azure OpenAI.Read 29 Sep 2026 · futureagi.com
Rating on G2
4.8 out of 5 from 12 reviews.Read 15 Oct 2026 · g2.com
Rating on Gartner Peer Insights
3.0 out of 5 from 1 rating.Read 15 Oct 2026 · gartner.com
Rating on Product Hunt
4.9 out of 5 from 9 reviews.Read 15 Oct 2026 · producthunt.com
Rating on Trustpilot
3.7 out of 5 from 1 review.Read 15 Oct 2026 · trustpilot.com
REST API
The public REST API is hosted at api.futureagi.com and uses X-Api-Key plus X-Secret-Key headers.Read 15 Oct 2026 · docs.futureagi.com
SDK package
The Python package is installed with pip install ai-evaluation and exposes one evaluate() function for local metrics, cloud metrics and LLM judging.Read 15 Oct 2026 · docs.futureagi.com
Security controls
The security page states AES-256 encryption at rest, TLS 1.2 or newer in transit, no training on customer data, annual penetration tests and US or EU data residency.Read 15 Oct 2026 · futureagi.com
Self-host operating systems
The changelog says the installer works on macOS, Linux and Windows through Git Bash, WSL or PowerShell.Read 15 Oct 2026 · futureagi.com
Self-host requirements
Standalone self-hosting requires Docker with 2 vCPUs, 4 GB memory and 10 GB disk; distributed deployment requires at least 4 vCPUs, 12–16 GB memory and 20 GB disk.Read 15 Oct 2026 · docs.futureagi.com
Self-hosting
Future AGI says its Docker Compose self-hosted deployment keeps traces, datasets, evaluations, and model calls within the customer's network.Read 29 Sep 2026 · docs.futureagi.com
Simulation
Simulation includes text and voice testing with personas and scenarios, with a free allowance of 1M text tokens and 60 voice minutes per month.Read 29 Sep 2026 · futureagi.com
Tracing
Its traceAI instrumentation turns LLM calls, tool use, retrieval, and chain steps into OpenTelemetry spans.Read 29 Sep 2026 · futureagi.com
Tracing integrations
The integrations page lists 79 TraceAI integrations across Python, TypeScript and Java, including voice, vector database, MCP and workflow tools.Read 15 Oct 2026 · futureagi.com

Plans and prices

Free Free $0 /month 50 GB storage/mo · 2K AI credits/mo · 100K gateway requests/mo · 100K cache hits/mo · 1M text simulation tokens/mo · 60 min voice simulation/mo · 30-day data retention · Community support Read 29 Sep 2026 · futureagi.com
Free Free $0 platform fee; usage blocked after allowance 50 GB storage · 2,000 AI credits · 100,000 gateway requests · 100,000 cache hits · 1,000,000 text-simulation tokens · 60 voice-simulation minutes · 50,000 tracing events · 30-day retention Read 15 Oct 2026 · docs.futureagi.com
Pay-as-you-go Free $0 platform fee; overage billed Free allowances included · usage continues past allowance · 30-day retention Read 15 Oct 2026 · docs.futureagi.com
Pay-as-you-go Free Starts at $0 /mo + usage Everything in Free · Usage-based after free tier · Volume discounts at scale · 30-day data retention · Email support · Billing alerts and spending caps Read 29 Sep 2026 · futureagi.com
Boost $250 a month $250 /mo 90-day data retention · 5 knowledge bases · 10 annotation queues · 15 monitors · SOC 2 Type II · OAuth SSO · Audit logs · 99.5% SLA · 48hr email support Read 29 Sep 2026 · futureagi.com
Boost $250 a month $250.00/month; usage billed separately 90-day retention · 500 API requests/minute · 20,000 ingestion events/minute Read 15 Oct 2026 · docs.futureagi.com
Scale $750 a month $750 /mo Everything in Boost · 1-year retention · Unlimited queues and monitors · Review workflow · Inter-annotator agreement · HIPAA BAA · SAML SSO + SCIM · 99.9% SLA · 24hr email · Slack channel Read 29 Sep 2026 · futureagi.com
Scale $750 a month $750.00/month; usage billed separately 365-day retention · 2,000 API requests/minute · 100,000 ingestion events/minute Read 15 Oct 2026 · docs.futureagi.com
Enterprise $2,000 a month $2,000 /mo Everything in Scale · Custom retention · ABAC · Data masking · Dedicated support engineer + CSM · Training sessions · Architecture review · Financial SLA · Custom rate limits Read 29 Sep 2026 · futureagi.com
Enterprise $2,000 a month $2,000.00/month; usage billed separately 7-year retention · unlimited API rate · unlimited ingestion rate Read 15 Oct 2026 · docs.futureagi.com

Every Future AGI AI Evaluation SDK plan and price →

Strong points and limits

Strong points

  • Evaluation methods include heuristic, code, LLM, and agentic
  • CI/CD support and evaluation templates
  • Self-hosted deployment keeps data within the customer's network
  • Free plan includes text and voice simulation allowances

Limits

  • Free usage pauses at its cap
  • Pay-as-you-go usage can incur overage charges
  • Free plan has 30-day data retention

Who it suits

It suits startup and enterprise teams building or operating AI agents who need evaluation, simulation, and monitoring tools. Teams that want data to remain within their network can consider its self-hosted deployment options.

Future AGI AI Evaluation SDK in depth

Read the full write-up

Overview

Future AGI AI Evaluation SDK is an open-source platform for evaluating, tracing, and simulating AI-agent workflows. It suits teams that want those capabilities together, especially when self-hosting matters. Its breadth is the draw; the free tier’s capped usage and short retention are meaningful constraints.

Rather than focusing only on model-output checks, the platform spans evaluations, trace ingestion, simulation, monitoring, and protection. That makes it a stronger fit for teams operating agents across a development lifecycle than for someone who needs only a lightweight local test runner.

Key features

Evaluation and tracing

Evaluations include heuristic, code-based, LLM-as-judge, and agentic methods, supported by templates and CI/CD workflows. Tool-call checks, safety evaluations, and regression runs give teams ways to assess agent behavior as it changes. This mix is useful when evaluation needs to cover more than a single scoring method, though the plan choice still determines how much data can be retained and how teams can collaborate.

TraceAI converts LLM calls, tool use, retrieval, and chain steps into OpenTelemetry spans. That provides a view across agent activity, while trace ingestion supports bringing traces into the platform. The integrations catalog has 79 entries, with SDK packages for Python, TypeScript, and Java. Featured providers include OpenAI, Anthropic, Google GenAI, Vertex AI, AWS Bedrock, and Azure OpenAI.

Simulation and deployment

Text and voice simulations use personas and scenarios, giving teams a way to exercise interactions beyond fixed evaluation cases. The Free plan’s 1M text tokens and 60 voice minutes per month provide a defined starting allowance, but teams with heavier simulation workloads will need to account for usage-based billing or a higher plan.

The platform is Apache 2.0 licensed and offers Docker, Python SDK, and Node SDK self-hosting options. Its Docker Compose deployment keeps traces, datasets, evaluations, and model calls inside the customer’s network. That is a practical advantage for teams that need data to remain in their environment; enterprise customers can also choose private cloud in their AWS, GCP, or Azure account, or air-gapped on-premise deployment.

Pricing

Future AGI’s freemium pricing is flexible at entry, but the free allowance has hard boundaries. The Free plan costs $0 /month and includes 50 GB storage per month, 2K AI credits per month, 100K gateway requests per month, 100K cache hits per month, 1M text simulation tokens per month, 60 voice simulation minutes per month, 30-day retention, and community support. Usage pauses when the free cap is reached, so it is best for evaluation and modest workloads rather than uninterrupted production use.

Pay-as-you-go starts at $0 /mo + usage and includes everything in Free, usage-based billing after the free tier, volume discounts at scale, 30-day retention, email support, billing alerts, and spending caps. It is the more suitable option when work must continue after the free quota, but it does not extend retention. Both entry plans give up the longer history and operational controls of the fixed-price tiers.

Boost costs $250 /mo and raises retention to 90 days, with five knowledge bases, 10 annotation queues, 15 monitors, SOC 2 Type II, OAuth SSO, audit logs, a 99.5% SLA, and 48-hour email support. It fits teams needing defined operational controls and a longer trace history without Scale’s collaboration and compliance package.

Scale costs $750 /mo and includes everything in Boost, one-year retention, unlimited queues and monitors, review workflow, inter-annotator agreement, HIPAA BAA, SAML SSO plus SCIM, a 99.9% SLA, 24-hour email support, and a Slack channel. It is aimed at teams with sustained review operations and stricter service or identity requirements.

Enterprise costs $2,000 /mo and adds custom retention, ABAC, data masking, a dedicated support engineer and CSM, training, architecture review, a financial SLA, and custom rate limits. It is the clearest fit for organizations that need those controls and deployment choices; smaller teams may find Scale sufficient.

Platforms

The supported platforms are API, Linux, macOS, self-hosted, web, and Windows. SDK packages cover Python, TypeScript, and Java, while self-hosting options include Docker, Python SDK, and Node SDK. This range suits teams spanning local development, API workflows, and private deployments.

Who it's for

Future AGI is aimed at startups and enterprise teams building and operating AI agents. It is especially relevant when evaluation, tracing, simulation, and deployment control need to sit in one platform. Teams with simple test-only needs or little tolerance for usage caps may be better served by a narrower tool.

Pros and cons

Pro: Multiple evaluation methods, regression runs, safety checks, and tool-call checks cover varied agent-testing needs.

Pro: OpenTelemetry-based traces and 79 integrations connect observability with a broad set of SDKs and providers.

Pro: Self-hosting and private or air-gapped enterprise deployment support data-control requirements.

Con: Free usage pauses at the cap, making it unsuitable for workloads that must continue without interruption.

Con: Free and pay-as-you-go plans retain data for only 30 days; longer retention starts with Boost at $250 /mo.

Con: The richer review, identity, and service commitments are concentrated in Scale and Enterprise, at $750 /mo and $2,000 /mo respectively.

Alternatives

AI Agent Evaluation Tools is the broader category page for comparing tools in this area.

Choose W&B Weave if its free plan’s 1 GB monthly ingestion, 5 GB storage, and evaluation, tracing, and scorers package better matches your expected volume. Noveum is an alternative with a free allowance of 2.5K credits, 1M spans, 2 GB storage, three members, and 30-day retention.

Opik offers an open-source core for observability and evaluation that users can download and run locally. Choose DeepEval instead when a free, Apache 2.0 local and CI/CD evaluation test runner is all you need.

Promptfoo offers a free Community plan with 10k red-team probes per month, evaluation features, provider integrations, and local or self-hosted operation. Google Cloud Agent Evaluation is a pay-as-you-go alternative priced per input and output character for computation-based metrics, with separate charges for model-based metrics.

Maxim AI offers a free Developer plan limited to three seats, one workspace, 10k logs per month, and three-day retention. Strands Evals is a free, open-source Python SDK and CLI; model-provider and trace-provider access may require separate credentials.

Verdict

Future AGI is a strong choice for AI-agent teams that need evaluations, traces, and text or voice simulation together, with self-hosting available when data control is central. Its main advantage is that breadth; its main drawback is the sharp distinction between capped free usage and paid plans for continued use, longer retention, and stronger operational commitments. Choose it when that combined workflow matters, and look elsewhere for a test-only tool or a free tier that can absorb your ongoing workload.

Verdict

Future AGI combines evaluation, tracing, simulation, and self-hosting options for AI-agent teams. The free tier has defined usage and retention limits, while paid plans start at $0 per month plus usage.

Where Specifiction places it

Alternatives to Future AGI AI Evaluation SDKAll alternatives →