Log File Bot Traffic Cost Analyzer vs Screaming Frog SEO Log File Analyser

Log File Bot Traffic Cost Analyzer examines server logs to summarize AI bot traffic, estimate costs, measure crawl budget, and prepare edge rules. It lists 127 bot signatures and 18 browser patterns, with bandwidth, IPs, and time to first byte by category. Verification uses vendor IP data, TLS, cloud ASN heuristics, and behavior; heuristic results are labeled low confidence, and optional reverse DNS through DoH is listed. Cost estimates use measured bytes and CDN pricing presets, with copyable rules for Cloudflare, Fastly, and AWS. Other modules cover crawl traps, rendering gaps, bot-versus-human performance, traffic bursts, 404 clusters, a GSC overlay, and security anomalies. Inputs include JSON, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended, AWS ALB, Cloudflare Logpush, and gzip files. The project describes processing as client-side, with logs remaining on the user's machine; its only stated network request is for a demo file the user chooses to load. Browser analysis under 500 MB is sampled triage, CLI analysis from 500 MB to 50 GB is described as exact, and BigQuery is for 50 GB and above. It is free, MIT-licensed, and supports CSV and PDF downloads plus generated policy files.

Screaming Frog SEO Log File Analyser is a desktop application for examining access logs and understanding search engine and AI bot activity. It installs locally and keeps log event data in a local database. It accepts Apache, W3C Extended, and Amazon Elastic Load Balancing logs, and can verify supported bots to identify spoofed requests. Reports cover crawled URLs, crawl frequency, response errors, redirects, bytes downloaded, and response times. Users can match imported data containing a URLs column against log activity, including crawls, directives, or external link data. Log files can be added by drag and drop, with analysis exported to CSV or Excel. The software can process millions of events, though capacity depends on hard drive space. It runs on Linux, macOS, and Windows. The free plan allows 1,000 log event lines and one project. The paid licence costs $139 per year and includes unlimited events and projects, subject to storage capacity. Free technical support is listed for both versions.