Log File Bot Traffic Cost Analyzer

github.com

Free plan WebLinuxSelf-hosted

Spec score 7.7 #13 of 27 in SEO Log File Analysis Tools

What it is

Log File Bot Traffic Cost Analyzer examines server logs to summarize AI bot traffic, estimate costs, measure crawl budget, and prepare edge rules. It lists 127 bot signatures and 18 browser patterns, with bandwidth, IPs, and time to first byte by category. Verification uses vendor IP data, TLS, cloud ASN heuristics, and behavior; heuristic results are labeled low confidence, and optional reverse DNS through DoH is listed. Cost estimates use measured bytes and CDN pricing presets, with copyable rules for Cloudflare, Fastly, and AWS. Other modules cover crawl traps, rendering gaps, bot-versus-human performance, traffic bursts, 404 clusters, a GSC overlay, and security anomalies. Inputs include JSON, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended, AWS ALB, Cloudflare Logpush, and gzip files. The project describes processing as client-side, with logs remaining on the user's machine; its only stated network request is for a demo file the user chooses to load. Browser analysis under 500 MB is sampled triage, CLI analysis from 500 MB to 50 GB is described as exact, and BigQuery is for 50 GB and above. It is free, MIT-licensed, and supports CSV and PDF downloads plus generated policy files.

Spec sheet

Purpose: Analyzes server logs to break down AI bot traffic and estimated costs, quantify crawl budget, and generate edge rules.Read 4 Oct 2026 · github.com
Intended users: The README positions it for quick private log triage, including cases where DPDP/RBI logs cannot leave the machine.Read 4 Oct 2026 · github.com
Deployment: The project identifies Cloudflare Pages as its primary deployment and GitHub Pages as a backup, and documents a Docker run command.Read 4 Oct 2026 · github.com
Bot classification: It lists 127 bot signatures and 18 browser patterns, with bandwidth, IPs, and TTFB by category.Read 4 Oct 2026 · github.com
Bot verification: Verification uses vendor IP data, TLS, cloud ASN heuristics, and behavior, with heuristics labeled low confidence; optional DoH reverse DNS is listed.Read 4 Oct 2026 · github.com
Cost and edge rules: It estimates costs from measured bytes and CDN pricing presets and offers copyable Cloudflare, Fastly, and AWS rules.Read 4 Oct 2026 · github.com
Exports: The README lists CSV and PDF downloads plus generated robots.txt, llms.txt, and crawlers.json policies.Read 4 Oct 2026 · github.com
License and support: The repository is MIT-licensed, welcomes pull requests, and labels some issues as good first issues.Read 4 Oct 2026 · github.com
Log formats: Supported inputs include JSON, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended, AWS ALB, Cloudflare Logpush, and gzip files.Read 4 Oct 2026 · github.com
Network behavior: The README says it uses no XHR, WebSocket, or sendBeacon, and that the only network request is for a demo file the user clicks to load.Read 4 Oct 2026 · github.com
Other features: Listed modules cover crawl traps, render gaps, bot versus human performance, traffic bursts, 404 clusters, a GSC overlay, and security anomalies.Read 4 Oct 2026 · github.com
Private processing: The project describes itself as 100% client-side and says logs do not leave the user's machine.Read 4 Oct 2026 · github.com
Scale limits: The README describes browser analysis under 500 MB as sampled triage, CLI analysis from 500 MB to 50 GB as exact, and BigQuery for 50 GB and above.Read 4 Oct 2026 · github.com

Plans and prices

Free Free Browser: under 500 MB triage · CLI: 500 MB–50 GB exact · BigQuery: 50 GB+ Read 4 Oct 2026 · github.com

Every Log File Bot Traffic Cost Analyzer plan and price →

Strong points and limits

Strong points

Client-side processing keeps logs on the user's machine
Includes 127 bot signatures and 18 browser patterns
Generates copyable Cloudflare, Fastly, and AWS rules
Accepts several web server and cloud log formats
Free and MIT-licensed

Limits

Browser analysis under 500 MB is sampled triage
CLI exact analysis is documented for 500 MB–50 GB
50 GB and above uses BigQuery

Who it suits

It suits people who want private log triage and bot-traffic analysis, including cases where logs cannot leave the machine. The documented options cover browser triage, CLI analysis, and BigQuery according to log size.

Verdict

The analyzer combines bot categorization, cost estimates, and generated edge and crawler policies. Its analysis path depends on log size, with browser triage, CLI, or BigQuery specified for different ranges.