Crawl Lab vs SEO Log File Analyzer by Kunal Dabi

Crawl Lab is a free web toolbox for analyzing server logs and AI crawler activity. It accepts Apache and Nginx common or combined logs, including compressed.gz files, and reports crawled pages, crawl frequency, wasted crawl budget, and bot errors. The analyzer recognizes GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Amazonbot, and other crawlers, checking their authenticity against official IP ranges. Processing occurs in the browser, so log files and IP addresses stay on your machine and are not uploaded or stored. No registration is required. Streaming reads allow analysis of several gigabytes or a full month of logs without a server-imposed size limit. Crawl Lab targets technical SEO practitioners concerned with SEO and generative-engine visibility. Its results show what crawlers fetched, not whether pages were indexed or how they rank. An llms.txt generator and AI citation audit are listed as coming soon. The free plan costs nothing, with no paid tier.

SEO Log File Analyzer by Kunal Dabi is a free web tool for examining Apache and Nginx access logs. It reports crawler activity, crawl-budget use, indexability, bot traffic, and response-time patterns. The analyzer shows requests to indexable HTML, assets, and low-value paths such as /wp-admin/. It tracks crawlers including Googlebot, Bingbot, Yahoo, DuckDuckBot, ChatGPT, Perplexity, AhrefsBot, SemrushBot, Applebot, and Bytespider. Sitemap URLs can be compared with bot-crawled URLs to find uncrawled pages. Reports include status-code and content-type breakdowns, per-bot charts, crawl coverage, and an action plan, with URL and error lists exportable to CSV. Response times are measured at p50, p95, and p99. Multiple.log,.txt, or extensionless files can be combined. The Free Log File Analyzer plan costs nothing, is free forever, and requires a daily access code.