SEO Log Auditor vs Spider-Lens

SEO Log Auditor analyzes server access logs to show how search bots crawl a site. It accepts nginx and Apache logs and provides seven views: crawl budget, orphan URLs, status waste, stale pages, performance, bot verification, and parameter traps. Reports can break down requests by user agent and highlight bot-to-human ratios, compare bot-accessed URLs with sitemap and internal-link signals, and rank frequently crawled non-200 responses. Other views flag URLs not visited by a major crawler in 30 or more days and surface slow pages by server response time. Bot verification compares declared user agents with known reverse-DNS signatures to help identify Googlebot or Bingbot impersonation. Parameter analysis surfaces URL patterns with high variation, such as session IDs, filters, and sorting. The app runs locally on a laptop without a database or cloud dependency; its maker describes it as having no telemetry or log upload. A repository workflow also describes Grafana/Loki exports in JSON, CSV, or plain text, plus a sitemap URL; direct Loki API streaming is a roadmap item. It uses Python, Streamlit, and Pandas, and is MIT-licensed.

Spider-Lens is a free, open-source application for analyzing Apache and Nginx server logs. It monitors traffic, HTTP errors, crawler bots and server performance through a dashboard covering requests, human visits, error rates, HTTP status trends, bot distribution and 12-week patterns. Crawler monitoring detects 16 or more bots and shows crawl budget and crawler volume. Performance reports include Time to First Byte by URL, configurable slow thresholds and CSV or Excel export. A sitemap crawler follows internal links and records titles, H1 headings, word counts, canonicals, robots directives and crawl depth. SMTP alerts and webhooks can report 404 spikes, server errors, missing Googlebot and weekly summaries. The project documents Discord, Slack and an MCP server integration. It runs as a Node.js application or native WordPress plugin, with no Node.js required for the plugin. Node deployment requires Node.js 18+, npm 9+ and log access. The interface lists 16 languages and automatic browser-language detection. Nova uses Gemini when configured.