Crowl

Spec sheet

Product

Crowl is an open-source SEO crawler made by SEOs for SEOs.Read 9 Oct 2026 · crowl.tech

Intended users

The maker describes Crowl as designed for SEO professionals and invites contributors to join its volunteer community.Read 9 Oct 2026 · crowl.tech

Platforms

The maker says Crowl works with Windows, Mac OS and Linux.Read 9 Oct 2026 · crowl.tech

Free plan

YesRead 21 Sep 2026 · crowl.tech

Integrations

The documentation describes CSV files and MySQL as output options; it does not state integrations with third-party SEO services.Read 9 Oct 2026 · crowl.tech

Support

The maker provides a contact form for questions and suggestions and links to Buy Me a Coffee for voluntary project support.Read 9 Oct 2026 · crowl.tech

License

Crowl is distributed under GNU GPL v3, which allows private or commercial use provided modified or redistributed code is shared under the same license.Read 9 Oct 2026 · crowl.tech

Additional crawl features

Release notes list exclusion patterns, proxies, rotating user-agents, HTTP authentication, request limits, duplicate-link removal and custom XPath extractors among Crowl features.Read 9 Oct 2026 · crowl.tech

Authentication

Crowl supports HTTP basic authentication settings for crawling environments that require a username and password.Read 9 Oct 2026 · crowl.tech

Crawl control

Crowl can stop and resume crawls.Read 9 Oct 2026 · crowl.tech

Crawl controls

Crowl supports stopping and resuming crawls and allows a custom user-agent.Read 9 Oct 2026 · crowl.tech

Crawl data

Crowl extracts links and anchor texts, and can extract page content.Read 9 Oct 2026 · crowl.tech

Customization

The maker describes Crowl as customizable and says crawl settings are configured in project-specific INI files.Read 9 Oct 2026 · crowl.tech

Exports

Crowl provides CSV export by default and supports storing crawl data in MySQL.Read 9 Oct 2026 · crowl.tech

Exports and storage

The maker lists CSV export and MySQL storage as built-in output options.Read 9 Oct 2026 · crowl.tech

Language detection

Crowl can identify the language of crawled page content using a FastText model, which must be downloaded separately.Read 9 Oct 2026 · crowl.tech

Maintenance

The release notes list version 0.3.1 dated July 27, 2021, as the latest release shown there.Read 9 Oct 2026 · crowl.tech

Platform

Crowl works with Windows, Mac OS, and Linux; its installation guide says it works best on UNIX-like systems, including Linux and macOS.Read 9 Oct 2026 · crowl.tech

Request settings

Configuration includes custom user agents, optional proxies, robots.txt compliance, request delays, and concurrent request settings.Read 9 Oct 2026 · crowl.tech

Support and community

The project invites community contributions, provides a contact form, and links to optional supporter funding.Read 9 Oct 2026 · crowl.tech

Technology

Crowl is developed using Python and Scrapy, and the maker says it works best with Python 3.6 or later.Read 9 Oct 2026 · crowl.tech

URL limit

The feature list says Crowl supports unlimited URLs; configuration also offers MAX_REQUESTS, defaulting to 0 for no limit.Read 9 Oct 2026 · crowl.tech

Warranty

The product page states that Crowl is provided without warranty.Read 9 Oct 2026 · crowl.tech

Where Specifiction places it

• SEO Crawler Software#25 of 347.6
• SEO tools#227 of 2457.0