Screaming Frog SEO Log File Analyser vs Spider-Lens

Screaming Frog SEO Log File Analyser is a desktop application for examining access logs and understanding search engine and AI bot activity. It installs locally and keeps log event data in a local database. It accepts Apache, W3C Extended, and Amazon Elastic Load Balancing logs, and can verify supported bots to identify spoofed requests. Reports cover crawled URLs, crawl frequency, response errors, redirects, bytes downloaded, and response times. Users can match imported data containing a URLs column against log activity, including crawls, directives, or external link data. Log files can be added by drag and drop, with analysis exported to CSV or Excel. The software can process millions of events, though capacity depends on hard drive space. It runs on Linux, macOS, and Windows. The free plan allows 1,000 log event lines and one project. The paid licence costs $139 per year and includes unlimited events and projects, subject to storage capacity. Free technical support is listed for both versions.

Spider-Lens is a free, open-source application for analyzing Apache and Nginx server logs. It monitors traffic, HTTP errors, crawler bots and server performance through a dashboard covering requests, human visits, error rates, HTTP status trends, bot distribution and 12-week patterns. Crawler monitoring detects 16 or more bots and shows crawl budget and crawler volume. Performance reports include Time to First Byte by URL, configurable slow thresholds and CSV or Excel export. A sitemap crawler follows internal links and records titles, H1 headings, word counts, canonicals, robots directives and crawl depth. SMTP alerts and webhooks can report 404 spikes, server errors, missing Googlebot and weekly summaries. The project documents Discord, Slack and an MCP server integration. It runs as a Node.js application or native WordPress plugin, with no Node.js required for the plugin. Node deployment requires Node.js 18+, npm 9+ and log access. The interface lists 16 languages and automatic browser-language detection. Nova uses Gemini when configured.