Log File Analysis for SEOs (Without an Engineer)

Server logs are the only ground-truth source of what Googlebot actually does. Most SEOs ignore them. Here’s how to read them without an engineer.

What to pull

Ask your host or DevOps for 14 days of access logs. Apache, Nginx, or your CDN’s equivalent. Common locations: /var/log/nginx/access.log or Cloudflare’s analytics export.

How to filter for Googlebot

Filter the user-agent for Googlebot. Forward-slash reverse DNS the IP — real Googlebot IPs resolve to *.googlebot.com or *.google.com. Anything else is fake.

What to look for

• Crawl frequency. How often does Googlebot hit your site?
• Crawl budget. Which URLs consume the most requests?
• Status codes. 4xx and 5xx on Google’s crawls = real problems.
• Response times. Slow responses = poor CWV.
• User-agents. GPTBot, ClaudeBot, CCBot — track AI crawlers separately.

The workflow

1. Export 14 days of logs to a CSV.
2. Filter for Googlebot user-agent.
3. Group by URL. Sort by request count.
4. Find the top 1% of URLs consuming the most requests.
5. Decide: are those URLs worth the crawl? If not, noindex or block.

Tools

• Screaming Frog Log Analyzer: Free, parses logs automatically.
• Botify: Premium, full feature set.
• ELK Stack: If you already run it.

The cadence

Quarterly. Pull 14 days, analyze, decide. Six months of trend data tells you more than any single audit.