Skip to content
SPECIFICTION
Get listed

Google Cloud Agent Evaluation

docs.cloud.google.com

Free plan WebAPI
Visit docs.cloud.google.com
Spec score 7.9 #15 of 28 in AI Agent Evaluation Tools

What it is

Google Cloud Agent Evaluation helps measure and improve agent performance, safety, and quality. Its workflow runs from defining evaluation cases through inference, behavior-trace capture, metric calculation, result analysis, and optimization. It can generate varied, multi-turn test scenarios from agent instructions and tool definitions, and intercept tool calls to supply custom behavior, mocked data, or simulated errors such as HTTP 503 responses and latency spikes. Prebuilt or custom raters score traces with reference-based Exact Match or reference-free Helpfulness metrics. Teams can also evaluate production-traffic traces or external logs without a managed test environment. Prompt optimization identifies failure points and proposes focused system-instruction changes. The service supports tool-call checks, safety evaluations, regression runs, trace ingestion, and custom models. Access is through API or web, with notebook launch options for Colab, Colab Enterprise, Agent Platform Workbench, and GitHub. Pricing is usage-based: $0.00003 per 1k input characters and $0.00009 per 1k output characters for computation-based metrics; model-based metrics also incur underlying autorater model costs. New Google Cloud customers get $300 in trial credits.

Spec sheet

Purpose
Agent evaluation measures and helps improve agents' performance, safety, and quality.Read 2 Oct 2026 · docs.cloud.google.com
Integrations
Google provides evaluation notebook launch options for Colab, Colab Enterprise, Agent Platform Workbench, and GitHub.Read 2 Oct 2026 · docs.cloud.google.com
Compliance
Google Cloud states that its services undergo independent verification of security, privacy, and compliance controls and achieve certifications against global standards.Read 2 Oct 2026 · cloud.google.com
Support
Google Cloud Basic Support includes documentation, community forums, billing assistance, and Active Assist; customers can upgrade for tailored technical support.Read 2 Oct 2026 · cloud.google.com
Headquarters
Mountain View, California, United StatesRead 28 Sep 2026 · docs.cloud.google.com
Founded
1998Read 28 Sep 2026 · docs.cloud.google.com
Coding assistant support
Evaluation skills for Gemini CLI or other AI coding assistants provide workflows, dataset schemas, metric guidance, and failure analysis steps.Read 2 Oct 2026 · docs.cloud.google.com
Environment simulation
It can intercept tool calls to inject custom behavior, mocked data, or simulated errors such as HTTP 503 errors and latency spikes.Read 2 Oct 2026 · docs.cloud.google.com
Evaluation workflow
The workflow defines evaluation cases, runs inferences, captures behavior traces, computes metrics, analyzes results, and optimizes the agent.Read 2 Oct 2026 · docs.cloud.google.com
Metrics
Prebuilt or custom raters score traces, including reference-based Exact Match and reference-free Helpfulness metrics.Read 2 Oct 2026 · docs.cloud.google.com
Production traces
Evaluation can score traces captured from production traffic or external logs without a managed test environment.Read 2 Oct 2026 · docs.cloud.google.com
Prompt optimization
Prompt optimization identifies failure points and iteratively proposes targeted updates to system instructions.Read 2 Oct 2026 · docs.cloud.google.com
Quota limit
The documented default quotas include 1,000 evaluation service requests per project per region per minute and 20 concurrent evaluation runs per project per region.Read 2 Oct 2026 · docs.cloud.google.com
Synthetic scenarios
It can automatically generate diverse, multi-turn synthetic test scenarios from agent instructions and tool definitions.Read 2 Oct 2026 · docs.cloud.google.com
Third-party models
The console tutorial says the Gen AI evaluation service can evaluate Anthropic and Llama partner models through Agent Platform Model Garden.Read 2 Oct 2026 · docs.cloud.google.com
Trial credits
New Google Cloud customers get $300 in free credits to run, test, and deploy workloads.Read 2 Oct 2026 · docs.cloud.google.com

Plans and prices

Pay-as-you-go usage Free $0.00003 per 1k input characters and $0.00009 per 1k output characters for computation-based metrics; model-based metrics are charged for underlying autorater model prediction costs. Pricing is usage-based; model-based metric charges depend on dataset input tokens and autorater output · Third-party model evaluation also incurs model inference charges Read 2 Oct 2026 · cloud.google.com

Every Google Cloud Agent Evaluation plan and price →

Strong points and limits

Strong points

  • Generates multi-turn synthetic scenarios from instructions and tools
  • Scores production traces and external logs
  • Offers reference-based and reference-free metrics
  • Includes automated prompt optimization
  • Available through API and web

Limits

  • No free plan
  • Usage charges vary by metric type
  • Default quota limits concurrent runs and requests

Who it suits

It suits teams evaluating agents across test scenarios and production traces, including teams that need automated prompt improvement or simulated tool behavior. It also supports evaluation of Anthropic and Llama partner models through Agent Platform Model Garden, as described in the console tutorial.

Verdict

Google Cloud Agent Evaluation covers scenario creation, trace scoring, safety checks, regression runs, and prompt optimization in one evaluation workflow. It has a free trial with $300 in credits for new customers, but costs depend on usage and model-based metric charges.

Where Specifiction places it

Alternatives to Google Cloud Agent EvaluationAll alternatives →