Log File Bot Traffic Cost Analyzer vs SEO Log Auditor

Log File Bot Traffic Cost Analyzer examines server logs to summarize AI bot traffic, estimate costs, measure crawl budget, and prepare edge rules. It lists 127 bot signatures and 18 browser patterns, with bandwidth, IPs, and time to first byte by category. Verification uses vendor IP data, TLS, cloud ASN heuristics, and behavior; heuristic results are labeled low confidence, and optional reverse DNS through DoH is listed. Cost estimates use measured bytes and CDN pricing presets, with copyable rules for Cloudflare, Fastly, and AWS. Other modules cover crawl traps, rendering gaps, bot-versus-human performance, traffic bursts, 404 clusters, a GSC overlay, and security anomalies. Inputs include JSON, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended, AWS ALB, Cloudflare Logpush, and gzip files. The project describes processing as client-side, with logs remaining on the user's machine; its only stated network request is for a demo file the user chooses to load. Browser analysis under 500 MB is sampled triage, CLI analysis from 500 MB to 50 GB is described as exact, and BigQuery is for 50 GB and above. It is free, MIT-licensed, and supports CSV and PDF downloads plus generated policy files.

SEO Log Auditor analyzes server access logs to show how search bots crawl a site. It accepts nginx and Apache logs and provides seven views: crawl budget, orphan URLs, status waste, stale pages, performance, bot verification, and parameter traps. Reports can break down requests by user agent and highlight bot-to-human ratios, compare bot-accessed URLs with sitemap and internal-link signals, and rank frequently crawled non-200 responses. Other views flag URLs not visited by a major crawler in 30 or more days and surface slow pages by server response time. Bot verification compares declared user agents with known reverse-DNS signatures to help identify Googlebot or Bingbot impersonation. Parameter analysis surfaces URL patterns with high variation, such as session IDs, filters, and sorting. The app runs locally on a laptop without a database or cloud dependency; its maker describes it as having no telemetry or log upload. A repository workflow also describes Grafana/Loki exports in JSON, CSV, or plain text, plus a sitemap URL; direct Loki API streaming is a roadmap item. It uses Python, Streamlit, and Pandas, and is MIT-licensed.