Skip to content
SPECIFICTION
Get listed

Crawlr

crawlr.ai

From $8/mo Free plan WebAPI
Visit crawlr.ai
Spec score 9.0 #1 of 34 in SEO Crawler Software

What it is

Crawlr is a website crawler and technical SEO audit tool that finds technical issues, crawl inefficiencies and site architecture problems. It discovers URLs through internal links, sitemaps, canonical links and JavaScript. Crawl settings include robots.txt compliance, rate limiting, inclusion and exclusion rules, and depth controls. Its analysis covers response codes, indexability, metadata, headings, word counts, and internal and external links. Results can be explored through interactive graphs, directory trees, 3D site views and content-cluster visualizations. Crawls can run daily, weekly or on custom schedules, with email notifications on completion. The API can start crawls, check their status, return structured JSON, access historical data and send webhook events. The Free plan is free and includes 500 tokens monthly, one concurrent crawl, CSV and JSON exports, and 15-day history. Paid plans start at $8/mo (annual). The Starter plan lists 5,000 tokens monthly, five concurrent crawls and a 5,000-page maximum per crawl. Unauthenticated sandbox crawls are limited to 25 pages.

Spec sheet

What it does
Crawlr is a cloud technical SEO crawler that finds technical issues, crawl inefficiencies and site-architecture problems.Read 15 Oct 2026 · crawlr.ai
Product
Crawlr is a website crawler and technical SEO audit tool that identifies technical SEO issues, crawl inefficiencies, and site architecture problems.Read 28 Sep 2026 · crawlr.ai
Intended users
The pricing page describes Starter as suited to single domains and portfolios, Standard to growing teams and agencies, and Growth to high-volume AI and SEO pipelines.Read 28 Sep 2026 · crawlr.ai
Deployment
The official pages describe Crawlr as cloud-based and provide no self-hosting option.Read 15 Oct 2026 · crawlr.ai
Free plan
YesRead 20 Sep 2026 · crawlr.ai
Paid from
8Read 20 Sep 2026 · crawlr.ai
API
The API supports starting crawls, checking status, retrieving structured JSON results, accessing historical data, and webhook events.Read 28 Sep 2026 · crawlr.ai
Integrations
The pricing page lists Google PageSpeed Insights for the Starter plan and webhook notifications (Beta) for Growth.Read 28 Sep 2026 · crawlr.ai
Security
The API uses API key authentication, and the documentation advises keeping keys out of client-side code, public repositories, and logs.Read 28 Sep 2026 · crawlr.ai
Support
The pricing comparison lists priority support and dedicated support as plan features.Read 28 Sep 2026 · crawlr.ai
AI-oriented output
Crawlr provides token-aware markdown summaries intended for LLM and RAG ingestion.Read 15 Oct 2026 · crawlr.ai
API access
YesRead 20 Sep 2026 · crawlr.ai
API capabilities
The API can start crawls, poll status, retrieve JSON results, access historical crawls and emit webhook events.Read 15 Oct 2026 · crawlr.ai
API design
The maker describes a REST API using API-key authentication, standard HTTP methods, documented JSON structures, rate limits and descriptive errors.Read 15 Oct 2026 · crawlr.ai
Company listing
A LinkedIn page named Crawlr Labs describes a privately held technology company with 11-50 employees focused on competitor monitoring; the page does not identify crawlr.ai's SnappyTech maker.Read 15 Oct 2026 · linkedin.com
Crawl controls
Crawl settings include robots.txt compliance, rate limiting, include and exclude rules, and crawl depth controls.Read 28 Sep 2026 · crawlr.ai
Crawl graphs
Interactive 2D force graphs, 3D graphs, tree views and filtered views visualize URL relationships and site structure.Read 15 Oct 2026 · crawlr.ai
Data collected
The service collects account details, billing information, usage logs, device information and crawl data such as URLs, content, metadata, links, status codes and performance metrics.Read 15 Oct 2026 · crawlr.ai
Data retention
Personal data is retained while an account is active or needed for service delivery, while crawl-data retention follows the subscription plan.Read 15 Oct 2026 · crawlr.ai
DIQ score
Its Domain Intelligence Quotient combines indexability, technical health, content quality, link structure and mobile or speed signals into a score from 0 to 100.Read 15 Oct 2026 · crawlr.ai
Documentation depth
The API page provides endpoint-level capabilities, data-format descriptions, use cases and authentication, rate-limit and error-handling notes, but no separate public SDK is stated.Read 15 Oct 2026 · crawlr.ai
Exports
The Free plan includes CSV and JSON exports; paid plans add LLM-ready markdown exports and higher-tier bulk export capabilities.Read 28 Sep 2026 · crawlr.ai
Full-site crawling
It crawls entire websites, including subdomains, parameterized URLs and XML-sitemap URLs.Read 15 Oct 2026 · crawlr.ai
HTTP analysis
The crawler tracks 2xx, 3xx, 4xx and 5xx responses and reports redirect chains, loops, soft 404s and broken links.Read 15 Oct 2026 · crawlr.ai
Independent listing status
G2 says Crawlr has not yet been reviewed and lacks enough reviews to provide buying insight.Read 15 Oct 2026 · g2.com
Independent pricing availability
G2 states that pricing details are unavailable on its listing and directs buyers to the vendor site.Read 15 Oct 2026 · g2.com
Independent product description
G2 describes Crawlr as a free web-scraping tool for extracting and managing website data, which differs from crawlr.ai's technical-SEO positioning.Read 15 Oct 2026 · g2.com
Independent seller identity
G2 attributes the listed Crawlr product to Crawlr Labs.Read 15 Oct 2026 · g2.com
Indexability analysis
It checks noindex, nofollow and canonical directives and flags conflicts such as noindex combined with another canonical target.Read 15 Oct 2026 · crawlr.ai
Integration
Google PageSpeed Insights is listed as the named third-party integration for performance analysis.Read 15 Oct 2026 · crawlr.ai
Licence
The opened official pages do not state an open-source licence or publish source code.Read 15 Oct 2026 · crawlr.ai
Link analysis
It measures internal and outbound links, anchor-text distribution, link depth, orphan pages and broken links.Read 15 Oct 2026 · crawlr.ai
Metadata analysis
Crawlr extracts titles, descriptions, H1-H6 headings, word counts and content-quality signals.Read 15 Oct 2026 · crawlr.ai
Orphan-page detection
The product identifies pages with no internal links pointing to them.Read 15 Oct 2026 · crawlr.ai
Privacy sharing
The policy says personal information is not sold and may be shared with service providers, for legal requirements, business transfers or with consent.Read 15 Oct 2026 · crawlr.ai
Rating on G2
0 out of 5 from 0 reviewsRead 15 Oct 2026 · g2.com
Sandbox
A no-signup sandbox can test URLs in Snapshot, Crawl or Map modes.Read 15 Oct 2026 · crawlr.ai
Sandbox limit
The API documentation says unauthenticated sandbox crawls are rate limited and have a maximum of 25 pages.Read 28 Sep 2026 · crawlr.ai
Scalability
The maker says its distributed cloud crawler can handle sites from a few pages to millions of URLs.Read 15 Oct 2026 · crawlr.ai
Scheduled crawls
Paid plans support daily, weekly or custom recurring crawls, post-deployment hooks and completion email notifications.Read 15 Oct 2026 · crawlr.ai
Screenshot capture
The feature list includes full-page screenshot capture for documentation and comparison.Read 15 Oct 2026 · crawlr.ai
Security stated
The privacy policy says data is encrypted in transit and at rest, authentication is secured and security assessments are performed regularly.Read 15 Oct 2026 · crawlr.ai
SEO analysis
The tool analyzes response codes, indexability signals, metadata, headings, word counts, and internal and external links.Read 28 Sep 2026 · crawlr.ai
Smart crawl controls
Crawls can respect or override robots.txt, apply rate limits, include or exclude URL patterns, handle parameters and restrict crawl depth.Read 15 Oct 2026 · crawlr.ai
Structured export
It exports structured JSON containing page-level details, issues, metadata, aggregate insights and link-graph data.Read 15 Oct 2026 · crawlr.ai
URL discovery
The crawler discovers URLs through internal links, sitemaps, canonicals, and JavaScript.Read 28 Sep 2026 · crawlr.ai
Visualizations
Crawlr offers interactive graphs, directory trees, 3D site exploration, and content cluster visualizations.Read 28 Sep 2026 · crawlr.ai

Plans and prices

Free Free monthly 500 tokens/month · 1 concurrent crawl · 500 pages/crawl · 15-day crawl history · full crawling · SEO intelligence and DIQ score · CSV/JSON exports Read 15 Oct 2026 · crawlr.ai
Starter $8 a month yearly, 2 months free 5,000 tokens/month · 5 concurrent crawls · 5,000 pages/crawl · 3-month crawl history · scheduled crawls · PageSpeed Insights · 8 visualizations Read 15 Oct 2026 · crawlr.ai
Standard $73 a month yearly, 2 months free 35,000 tokens/month · 25 concurrent crawls · 10,000 pages/crawl · 1-year crawl history · LLM-ready markdown · advanced indexability · priority queues Read 15 Oct 2026 · crawlr.ai
Growth $291 a month yearly, 2 months free 100,000 tokens/month · 100 concurrent crawls · 25,000 pages/crawl · 1-year crawl history · webhook notifications beta · bulk exports · dedicated account manager · custom crawl frequency Read 15 Oct 2026 · crawlr.ai
Enterprise Priced on request custom pricing Unlimited pages · custom crawl logic · dedicated resources · SLA Read 15 Oct 2026 · crawlr.ai

Every Crawlr plan and price →

Strong points and limits

Strong points

  • Finds URLs through links, sitemaps, canonicals and JavaScript.
  • Offers crawl depth and inclusion controls.
  • Visualizes site structure with graphs and directory trees.
  • API supports structured results, history and webhook events.
  • Free plan includes CSV and JSON exports.

Limits

  • Unauthenticated sandbox crawls are limited to 25 pages.
  • Free plan provides 500 tokens per month.
  • Paid plans start at $8/mo (annual).

Who it suits

Crawlr suits site owners and SEO practitioners who need to inspect crawl behavior, site structure and page signals. Its plan descriptions also identify single domains, portfolios, growing teams, agencies and high-volume SEO or AI pipelines as intended users.

Verdict

Crawlr combines configurable crawling, SEO analysis, visualizations and API access. The Free plan provides a starting tier, while paid plans add capacity and export capabilities.

Where Specifiction places it

Alternatives to CrawlrAll alternatives →