CrawlGenius Log Analyzer vs Log File Bot Traffic Cost Analyzer

CrawlGenius Log Analyzer analyzes server logs to show how search engines and AI crawlers interact with a website. Its SEO analysis identifies crawl budget waste, HTTP errors, orphan pages, redirect and canonical issues, and fake bots. AI and LLM crawlers are tracked separately from Googlebot, with comparisons between AI and SEO crawling. Dashboards cover crawl coverage, freshness, time to first visit, new URLs, and URLs crawled but never clicked.

Logs can be supplied through a WordPress plugin, server agent, file upload, Amazon S3, or FTP/sFTP deposit; FTP/sFTP is unavailable on the Free plan. Paid plans include a REST API and MCP server. The pricing page says AI crawlers are checked against more than 3,500 official IP ranges and spoofed hits rejected at ingestion. The company says data is hosted in the EU, processed under GDPR, and visitor IPs are truncated before storage; it does not currently hold SOC 2 or ISO 27001 certification. The web-based product is aimed at SEO professionals. The Free plan is free and covers 10,000 log lines monthly, one project, and a seven-day visible window. Starter is €49 a month and Agency is €149 a month, excluding VAT. A guided 30-day trial uses real customer logs and requires no credit card or commitment.

Log File Bot Traffic Cost Analyzer examines server logs to summarize AI bot traffic, estimate costs, measure crawl budget, and prepare edge rules. It lists 127 bot signatures and 18 browser patterns, with bandwidth, IPs, and time to first byte by category. Verification uses vendor IP data, TLS, cloud ASN heuristics, and behavior; heuristic results are labeled low confidence, and optional reverse DNS through DoH is listed. Cost estimates use measured bytes and CDN pricing presets, with copyable rules for Cloudflare, Fastly, and AWS. Other modules cover crawl traps, rendering gaps, bot-versus-human performance, traffic bursts, 404 clusters, a GSC overlay, and security anomalies. Inputs include JSON, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended, AWS ALB, Cloudflare Logpush, and gzip files. The project describes processing as client-side, with logs remaining on the user's machine; its only stated network request is for a demo file the user chooses to load. Browser analysis under 500 MB is sampled triage, CLI analysis from 500 MB to 50 GB is described as exact, and BigQuery is for 50 GB and above. It is free, MIT-licensed, and supports CSV and PDF downloads plus generated policy files.