WebArena

Spec sheet

Purpose

WebArena is a standalone, self-hostable web environment and benchmark for building autonomous agents.Read 11 Oct 2026 · webarena.dev

License

The WebArena code repository identifies its license as Apache-2.0.Read 11 Oct 2026 · github.com

Demo environments

The demos include social forum, online shopping, content management, collaborative software development, map, wiki, calculator, and scratchpad sites.Read 5 Oct 2026 · webarena.dev

Embedded resources

The environment includes tools and knowledge resources as independent websites.Read 5 Oct 2026 · webarena.dev

Environment

It recreates websites from four popular categories with functionality and data modeled on real-world equivalents, and includes tools and knowledge resources as separate websites.Read 11 Oct 2026 · webarena.dev

Evaluation

Evaluation checks information-seeking answers against annotated references and can programmatically check intermediate execution states.Read 11 Oct 2026 · webarena.dev

Evaluation limit

The repository instructs users to reset the environment after evaluating its 812 examples.Read 11 Oct 2026 · github.com

Evaluation requirement

The repository says to set up your own WebArena websites for correct evaluation and describes the demo sites as browsing-only.Read 5 Oct 2026 · github.com

Example tasks

Shown tasks include creating a repository and checking the status and arrival time of an order.Read 5 Oct 2026 · webarena.dev

External model key

The repository’s evaluation instructions require exporting a valid OpenAI API key before launching evaluation.Read 5 Oct 2026 · github.com

Included sites

The demos include a social forum, online shopping, content management, collaborative software development, a map, a wiki, a calculator, and a scratchpad.Read 11 Oct 2026 · webarena.dev

Licensing

The WebArena website states that it is licensed under Creative Commons Attribution-ShareAlike 4.0 International.Read 5 Oct 2026 · webarena.dev

Model use

The repository's example evaluation command uses GPT-3.5 Turbo and requires an OpenAI API key.Read 11 Oct 2026 · github.com

Observation formats

Agents can observe a page through its URL and content, represented as a screenshot, HTML DOM tree, or accessibility tree.Read 5 Oct 2026 · webarena.dev

Observation options

Observations include a page URL and content, represented as a screenshot, HTML DOM tree, or accessibility tree.Read 11 Oct 2026 · webarena.dev

Related framework

The repository describes AgentLab as supporting parallel experiments through BrowserGym, benchmark integration including VisualWebArena, unified leaderboard reporting, and improved handling of environment edge cases.Read 11 Oct 2026 · github.com

Research affiliations

The page lists Carnegie Mellon University and Inspired Cognition as author affiliations.Read 5 Oct 2026 · webarena.dev

Research provenance

The project page lists Carnegie Mellon University and Inspired Cognition affiliations for the paper's authors.Read 11 Oct 2026 · webarena.dev

Self-hosting

The project provides Docker resources and a pre-installed Amazon Machine Image for hosting the benchmark websites.Read 11 Oct 2026 · github.com

Setup

The canonical repository README specifies Python 3.10 or newer and installation of its requirements and Playwright browser.Read 5 Oct 2026 · github.com

Setup requirement

The repository installation instructions specify Python 3.10 or later and Playwright.Read 11 Oct 2026 · github.com

Simulated websites

It creates websites across four popular categories with functionality and data modeled on real-world equivalents.Read 5 Oct 2026 · webarena.dev

Task input

The benchmark evaluates agents that turn high-level natural-language commands into web interactions.Read 5 Oct 2026 · webarena.dev

Task validation

WebArena provides annotated programs to programmatically validate each task’s functional correctness.Read 5 Oct 2026 · webarena.dev

Tasks

The benchmark evaluates agents that translate realistic natural-language instructions into concrete web interactions.Read 11 Oct 2026 · webarena.dev

Validation

Annotated programs are provided to validate task completion programmatically.Read 11 Oct 2026 · webarena.dev

Where Specifiction places it

• LLM Evaluation Tools#26 of 296.8