Skip to content
SPECIFICTION
Get listed

AgentBench

github.com

LinuxAPISelf-hosted
Visit github.com
Spec score 7.8 #10 of 29 in LLM Evaluation Tools

Spec sheet

Purpose
AgentBench is a benchmark for evaluating the reasoning and decision-making abilities of LLMs acting as agents.Read 9 Oct 2026 · arxiv.org
Deployment
The project documents a one-command setup for AgentBench FC tasks using Docker Compose.Read 9 Oct 2026 · github.com
Support
The project invites users to join its Slack for questions and collaboration, and lists a Google Group for questions about AgentBench FC.Read 9 Oct 2026 · github.com
License
The repository is licensed under Apache License 2.0.Read 9 Oct 2026 · github.com
Founded
2023Read 28 Sep 2026 · github.com
Configuration
The configuration system uses YAML with extended import, default, and overwrite keywords.Read 9 Oct 2026 · github.com
Current tasks
The current function-calling version supports alfworld, dbbench, knowledgegraph, os_interaction, and webshop tasks.Read 9 Oct 2026 · github.com
Current version
The repository says its current version is AgentBench FC, which uses function-calling prompts and integrates with AgentRL.Read 9 Oct 2026 · github.com
Environments
The original benchmark includes eight environments, including operating systems, databases, knowledge graphs, house-holding, web shopping, and web browsing.Read 9 Oct 2026 · github.com
Extensibility
Users can add a task environment by implementing the Task class and specifying it in the configuration.Read 9 Oct 2026 · github.com
FC task coverage
AgentBench FC supports fully containerized deployment for alfworld, dbbench, knowledgegraph, os_interaction, and webshop.Read 9 Oct 2026 · github.com
Hardware limit
The README warns that the webshop environment requires approximately 16 GB of RAM to start.Read 9 Oct 2026 · github.com
Knowledge graph dependency
The KnowledgeGraph task depends on an online service that the README says is unstable, and it documents local deployment as an option.Read 9 Oct 2026 · github.com
Known issue
The README warns that the current alfworld implementation leaks memory and disk space until its task worker is restarted.Read 9 Oct 2026 · github.com
Maintainer
The THUDM GitHub profile identifies the group as THUKEG and lists FIT Building, Tsinghua University as its location.Read 9 Oct 2026 · github.com
Model interfaces
The framework supports local models served with FastChat and says models offered only through an API can be connected by implementing an interface in the Agent Client.Read 9 Oct 2026 · github.com
Model setup
The original quick start configures an OpenAI API key and gives gpt-3.5-turbo-0613 as its example agent.Read 9 Oct 2026 · github.com
Original environments
The original benchmark covers eight environments: operating system, database, knowledge graph, digital card game, lateral thinking puzzles, household tasks, web shopping, and web browsing.Read 9 Oct 2026 · github.com
Other agents
The README says users can replace the default agent by changing configuration parameters.Read 9 Oct 2026 · github.com
Requirements
The README recommends Python 3.9 for dependency installation and requires Docker for the original quick-start setup.Read 9 Oct 2026 · github.com
Research release
The repository describes AgentBench as an ICLR 2024 work.Read 9 Oct 2026 · github.com
Resource limits
The current webshop environment requires about 16 GB of RAM to start, and the README warns that alfworld can leak memory and disk space until its task worker is restarted.Read 9 Oct 2026 · github.com

Where Specifiction places it

Alternatives to AgentBenchAll alternatives →