Most technical SEO audits stop at a crawl simulator — Screaming Frog, Sitebulb, whatever tool renders the site the way a bot theoretically would. That's useful, but it's a simulation. It tells you what could be crawled, not what Googlebot actually spent its time crawling last Tuesday. Log file analysis is the only source of truth for that, and most SEOs skip it because raw server logs are unglamorous, occasionally enormous, and require actually asking a dev or hosting provider for access instead of running a tool from your own laptop.
That gap between simulated and actual crawl behavior is where a lot of ranking problems hide in plain sight. A site can have a technically clean architecture and still be bleeding crawl budget on parameter URLs, redirect chains, or an infinite-scroll pagination bug that a crawler simulator never triggers the same way a persistent bot does over weeks of repeated visits.
What Log Files Actually Show You
Server logs record every request hitting your server, including every Googlebot visit — timestamp, URL requested, status code returned, user agent, and response time. Filtered down to just Googlebot's user agent strings (and verified by reverse DNS lookup, since user agent alone can be spoofed), this becomes a complete, unfiltered record of exactly what Google crawled, how often, and what it got back.
This is the difference between guessing what Google sees and knowing.
Getting the Data
Ask your hosting provider or dev team for raw access logs — Apache, Nginx, or CDN logs (Cloudflare, Fastly) depending on the stack — covering at minimum 30 days, ideally 90 for a site with any real crawl volume. Tools like Screaming Frog's Log File Analyser, JetOctopus, or Botify will parse and filter this for you; for smaller sites, a spreadsheet and some grep work gets you most of the way there.
Verify Googlebot requests by reverse DNS — a legitimate Googlebot hit resolves back to a googlebot.com or google.com domain.
Skip this step and you risk drawing conclusions from bot traffic pretending to be Googlebot, which is more common than most people assume.
The Five Things to Look For
1. Crawl frequency by page type
Group crawled URLs by template or section and count hits per page type. It's common to find category pages crawled daily while the product or service pages that actually drive revenue get touched once a month. That imbalance is a direct, fixable signal about where your internal linking and sitemap priority are misallocated.
2. Status codes Googlebot is actually hitting
Filter for 4xx and 5xx responses specifically returned to Googlebot's requests, not just what a crawler simulator finds. Redirect chains, especially, show up here in ways they don't always show up in a standard crawl — a chain that resolves fine in three hops for a browser can be abandoned by Googlebot after fewer hops, wasting the crawl entirely.
3. Crawled but non-indexable pages
Cross-reference crawled URLs against your canonical, noindex, and robots.txt rules. Pages Google is spending budget crawling that can never be indexed anyway are pure waste — parameter URLs, filtered/faceted navigation variants, staging or dev paths that never got properly blocked.
4. Orphaned pages Googlebot still visits
Some URLs get crawled purely from historical backlinks or old sitemap entries with no current internal links pointing to them. These often show declining crawl frequency over time — a slow signal that Google's deprioritizing a page you may not have realized was drifting.
5. The gap between crawled and indexed
Compare your log-confirmed crawled URL list against what's actually indexed via Search Console's Index Coverage report. A large gap — pages crawled repeatedly but never indexed — usually points to a content quality or duplication problem, not a crawlability problem, since crawlability obviously isn't the barrier if Googlebot's already there.
Turning This Into Fixes
- **Block or noindex the confirmed waste** — parameter variants, filtered nav, staging paths — via robots.txt or noindex tags, prioritizing by crawl frequency so you fix the biggest budget sinks first.
- **Fix redirect chains found in the logs**, not just the ones a simulator flags; log-confirmed chains are the ones actually costing you crawl budget in practice.
- **Reallocate internal links and sitemap priority** toward the revenue pages showing low crawl frequency relative to their business importance.
- **Re-pull logs 60 days after changes** to confirm crawl behavior actually shifted — this is a measurable, closed-loop fix, not a change you make once and hope worked.
Log file analysis isn't glamorous work, and it won't produce a screenshot worth posting. It's the closest thing in SEO to reading Google's mind, and most sites never bother to look.