Crawler Log Analyser vs Log File Bot Traffic Cost Analyzer
Crawler Log Analyser
github.comLog File Bot Traffic Cost Analyzer
github.comSide by side
| Figure | Crawler Log Analyser | Log File Bot Traffic Cost Analyzer |
|---|---|---|
| Price from | — | Free |
| Free plan | — | Yes |
| Free trial | — | — |
| Open source | — | — |
| Web | — | Yes |
| Linux | Yes | Yes |
| Self-hosted | Yes | Yes |
| Plans published | 0 | 1 |
| Spec score · SEO Log File Analysis Tools | 7.7 (#9) | 7.7 (#13) |
| Intended users | The maker positions it for operators of self-hosted static sites who need to inspect real crawler behavior in nginx access logs. | The README positions it for quick private log triage, including cases where DPDP/RBI logs cannot leave the machine. |
| Purpose | The project describes itself as operational SEO observability for self-hosted static sites running nginx. | Analyzes server logs to break down AI bot traffic and estimated costs, quantify crawl budget, and generate edge rules. |
Plans
In their own words
Crawler Log Analyser
Crawler Log Analyser has no write-up on Specifiction yet; its figures are above.
Log File Bot Traffic Cost Analyzer
Log File Bot Traffic Cost Analyzer examines server logs to summarize AI bot traffic, estimate costs, measure crawl budget, and prepare edge rules. It lists 127 bot signatures and 18 browser patterns, with bandwidth, IPs, and time to first byte by category. Verification uses vendor IP data, TLS, cloud ASN heuristics, and behavior; heuristic results are labeled low confidence, and optional reverse DNS through DoH is listed. Cost estimates use measured bytes and CDN pricing presets, with copyable rules for Cloudflare, Fastly, and AWS. Other modules cover crawl traps, rendering gaps, bot-versus-human performance, traffic bursts, 404 clusters, a GSC overlay, and security anomalies. Inputs include JSON, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended, AWS ALB, Cloudflare Logpush, and gzip files. The project describes processing as client-side, with logs remaining on the user's machine; its only stated network request is for a demo file the user chooses to load. Browser analysis under 500 MB is sampled triage, CLI analysis from 500 MB to 50 GB is described as exact, and BigQuery is for 50 GB and above. It is free, MIT-licensed, and supports CSV and PDF downloads plus generated policy files.