Technical SEO: fixing what stops a site from ranking

Crawlability: can search engines reach every page that matters

Crawlability is whether a search engine can find and follow the paths to your content. The three things that quietly block it are robots.txt disallow rules, orphaned pages with no internal links, and navigation that depends on JavaScript a crawler never executes.

Start with robots.txt. A single overbroad disallow rule can wall off an entire directory of valuable pages, and it happens constantly during staging-to-production migrations when someone forgets to remove the blanket block. Read the file line by line and confirm nothing important is disallowed.

Next, internal links. A page with no internal links pointing at it is an orphan. Crawlers reach it only through the sitemap, if at all, and it accrues almost no internal authority. Strong internal linking is the cheapest ranking lever most sites ignore. We go deep on this in internal linking at scale, but the principle is simple: every page that matters should be reachable in a few clicks from the homepage and linked from related content.

Finally, crawl paths. If your primary navigation or pagination only works after JavaScript runs, test what a crawler actually sees in the raw HTML. If a bot cannot render the menu, it cannot follow those links reliably, and pages that live only behind script-dependent navigation can go undiscovered.

Indexation: getting the right pages in, keeping the wrong ones out

Being crawled is not the same as being indexed. Google crawls far more than it keeps. Indexation problems show up in Search Console under Pages, and the two patterns worth learning are crawled, currently not indexed and discovered, currently not indexed.

Crawled, currently not indexed usually means Google looked at the page and judged it not worth keeping. That is a quality signal, not a bug. Thin, near-duplicate, or low-demand pages sit here. The fix is rarely technical. It is consolidating or improving the content, or accepting that the page should not exist.

Discovered, currently not indexed means Google knows the URL exists but has not prioritized crawling it, often a crawl-budget or site-quality signal on larger sites. Programmatic and template-heavy sites hit this wall hard. If you run pages at scale, read programmatic SEO in 2026 before you generate another ten thousand URLs.

Control indexation deliberately. Use the noindex meta tag to keep low-value pages out, canonical tags to consolidate duplicates, and a clean XML sitemap that lists only canonical, indexable URLs. A sitemap full of redirects, 404s, and noindexed pages teaches Google to trust it less.

Duplicate content and canonicalization

Duplicate content on a single domain is almost always a URL problem, not a plagiarism problem. The same page served at http and https, with and without www, with and without a trailing slash, and with tracking parameters appended, is four or more URLs for one piece of content. Search engines have to pick which one to rank, and they do not always pick the one you want.

The fix is canonicalization. Every page should declare a single canonical URL, and every non-canonical variant should either redirect to it with a 301 or carry a canonical tag pointing at it. Faceted navigation on ecommerce and directory sites is the worst offender: filter and sort combinations can multiply one category into thousands of near-identical URLs that eat crawl budget and dilute ranking signals.

Decide your canonical rules once, enforce them in code, and verify with a crawl. Consistency beats cleverness here. Pick https, pick a www convention, pick a trailing-slash convention, and make everything else redirect.

Site speed and Core Web Vitals

Page experience is a real ranking input, and Core Web Vitals are how Google measures it: Largest Contentful Paint for loading, Interaction to Next Paint for responsiveness, and Cumulative Layout Shift for visual stability. These are field metrics, measured on real users, not a lab score you can game.

The usual culprits are predictable. LCP suffers from oversized hero images and slow server response. INP suffers from heavy JavaScript blocking the main thread. CLS comes from images and ads without reserved dimensions and from fonts that swap late. None of these require a rebuild. They require discipline: compress and correctly size images, defer non-critical scripts, and reserve space for anything that loads late.

We cover the current thresholds and the fixes that actually move them in the Core Web Vitals and rankings guide for 2026. One caution worth stating plainly: speed is a tiebreaker, not a magic wand. A fast thin page still loses to a slower page that better answers the query. Fix speed because it helps conversion and it is a genuine signal, not because you expect it to leapfrog a stronger competitor.

Rendering and JavaScript SEO

Modern sites lean on JavaScript, and that creates a gap between what a browser shows a human and what a crawler indexes. Google renders JavaScript, but rendering is a second pass that happens on its own schedule and its own budget. Content that only appears after client-side rendering can be indexed late or incompletely.

The reliable pattern is to serve the important content in the initial HTML. Server-side rendering or static generation puts your title, main copy, structured data, and internal links in the source that arrives on the first request, so no crawler has to wait for a render pass to see what the page is about. If you are on a framework that supports it, use it.

Diagnose rendering by comparing the raw HTML response with the rendered DOM. If your primary content, canonical tag, or internal navigation only exists in the rendered version, that is your risk. This matters most for content-critical elements: nobody should have to execute your JavaScript to learn what your page is for.

Structured data that actually helps

Structured data, or schema markup, tells search engines what your content means in a machine-readable way. It does not directly raise rankings, and any tool or agency that promises otherwise is guessing. What it does do is make you eligible for rich results and give AI answer engines clean, parseable facts to cite.

Focus on the schema types tied to your content: Article for editorial, Product and Offer for ecommerce, LocalBusiness for physical locations, FAQPage for genuine question-and-answer content, and Breadcrumb for navigation context. Mark up what is actually on the page, never invent content in the markup that a user cannot see, and validate every template.

Which types earn visible results, and which are just noise, is a moving target. We keep schema markup that moves rankings current on what is worth implementing. As AI answer engines become a bigger share of discovery, clean structured data is increasingly how you get quoted rather than skipped.

Log file analysis and crawl budget

For most small sites, crawl budget is not a real constraint and you should not spend time on it. For large sites, ecommerce catalogs, publishers, and anything programmatic, it becomes the ceiling. Crawl budget is the number of URLs Google will crawl in a given window, and if it is being spent on parameter junk and redirect chains, your important pages get crawled less often and updated slower in the index.

Server log files are the only source of truth for what search engines actually crawl, as opposed to what a simulator thinks they crawl. Logs show you which URLs Googlebot hits, how often, and what status codes it gets back. That is how you find crawl waste: bots hammering faceted URLs, chasing redirect chains, or repeatedly requesting pages that return errors.

The workflow is covered in log file analysis for crawl budget. The payoff is targeted: cut the waste, funnel crawl activity toward pages that earn revenue, and your fresh and important content gets discovered and refreshed faster.

Frequently Asked Questions

Keep Reading

• The technical SEO checklist for 2026
• Core Web Vitals and rankings in 2026
• Schema markup that moves rankings
• Log file analysis for crawl budget
• Content strategy pillar
• Resources and guides