Three sequential gates stand between every URL and a ranking: crawl, render, index. On popular sites, 16% of valuable pages never make it through the last one. This module is the diagnostic discipline for all three — log files, the two-wave rendering model, every “why pages aren’t indexed” status decoded, and the instruments we use to find out exactly where your pages are dying.
Search visibility is gated by three sequential processes: crawl (Google's bots discover and fetch your pages), render (Google's renderer processes the fetched page including JS), index (Google decides whether to store the rendered page in its searchable index). If any gate fails, the page can't rank no matter how good the content is or how many links it has.
This module is for the diagnostic and remediation work behind those three gates. The Technical SEO Foundations module covers the artifacts (robots.txt, sitemaps, canonicals); this module covers the systems and the diagnostic discipline.
By the numbers The attrition is real — and measurable
What actually survives the pipeline
16%
of valuable, indexable pages on popular sites never get indexed at all (Onely / Tomek Rudzki).
96.55%
of pages get zero Google traffic — Ahrefs, across 14 billion pages. Indexed ≠ visited.
2 MB
of HTML is all Googlebot processes for indexing per page (15 MB infra default; 64 MB PDFs).
9×
longer for Google to crawl JS-dependent content than plain HTML; render median ~5s, queue lag up to weeks.
Crawl budget = the number of URLs Googlebot can and wants to crawl on your site in a given period. It has two components:
Crawl rate limit: What Googlebot can fetch without overloading your server. Set automatically based on response time and errors.
Crawl demand: What Googlebot wants to fetch based on perceived URL value (freshness, popularity, link equity).
For sites under ~10k URLs, crawl budget is rarely a concern — Google can crawl everything quickly. For sites with 100k+ URLs, crawl budget becomes a real constraint. For sites with millions of URLs (ecommerce, news, classifieds), crawl budget management is one of the highest-leverage SEO disciplines.
Symptoms of crawl budget problems
New pages take days/weeks to be indexed.
Updated pages show stale content in SERPs.
Important pages buried deep are never crawled.
Search Console "Crawled - currently not indexed" counts in the tens of thousands.
Google's crawl rate (in Search Console settings) is at or below capacity for the site size.
Common crawl budget wasters
Faceted navigation parameter combinations generating millions of low-value URLs.
Internal search result pages indexable.
Calendar pages with infinite future dates.
Session IDs in URLs.
Tracking parameters duplicating canonical URLs.
Soft 404s (200 status on empty pages).
Long redirect chains.
Slow server response times (TTFB > 600ms).
Pages with mostly duplicate content (boilerplate-heavy templates).
Anatomy of waste Where crawl budget actually goes on an unaudited large site
Typical fetch distribution before cleanup — RGM analysis from client log audits
Faceted params
38%
Redirect chains
14%
Soft 404s / thin
11%
Internal search
9%
Real, canonical pages
28%
Directional pattern, labeled RGM analysis — your logs will draw your own version. The point survives every audit we run: most unaudited large sites spend under a third of Googlebot’s attention on pages that matter.
RGM EXPERT TRICK
Prune with 410, not 404 — and bank the reclaimed crawl
When we prune dead sections, we return 410 Gone, not 404. Google treats 410 as a firmer signal: pages drop faster and Googlebot stops re-fetching sooner.
Then we read the dividend in the logs. Within two or three weeks the fetches that used to burn on the graveyard shift to live templates — we screenshot the before/after and put it in the client deck.
noindex for pruning is the rookie move: it requires Google to keep crawling the page to keep seeing the tag. You pay crawl budget forever for a page you killed.
WHY IT’S RARE · Most platforms only emit 404s, so nobody asks for the 410 — and nobody measures pruning in reclaimed fetches.
URL discovery and prioritization
Google discovers URLs through:
Following links from already-crawled pages (the primary mechanism).
XML sitemaps you submit.
The IndexNow protocol (Bing-led; Google doesn't use yet).
External signals: links from other sites, mentions on social.
Direct URL submission (Search Console, but limited per-day).
Google prioritizes which URLs to crawl based on perceived importance: link equity, freshness signals, sitemap inclusion, recent updates. URLs that have been crawled and found low-value get re-crawled less often.
Log file analysis: the diagnostic that exposes everything
Server access logs record every request Googlebot makes. They show: which URLs were crawled, when, how often, what status code Google saw, what the response time was. No other data source gives you this level of ground truth.
What to look for in log analysis
Wasted crawl budget. URLs Google crawls that you don't care about (parameters, paginated archives, junk).
Under-crawled important pages. Pages you do care about that Googlebot rarely fetches.
Crawl spikes after deploys. Sudden surges when you ship changes — can flag mass URL changes or new internal links.
4xx and 5xx errors visible to Googlebot. Often different from what you see in QA because they happen at scale.
Slow response times to Googlebot. Different distribution than human users; bot-specific server problems.
Verification of Googlebot vs spoofed user agents. Reverse DNS lookup confirms real Google IPs.
Tools for log analysis
Screaming Frog Log File Analyser: Cheap, accessible, good for sites under a few million log lines.
Botify: Enterprise-grade. Strong integrations with Search Console and crawler data.
IP — verify it is really Google (reverse DNS)URL — is this a page you care about?Status — what Google saw, at scaleResponse time — slow to bots = crawled less
Line 1 and line 3 are the same product fetched twice via different parameters — that is crawl waste with a timestamp on it. Line 2 is a redirect Googlebot keeps paying for. Three lines in, you already have two remediation tickets.
Rendering deep dive
Google's rendering happens in two passes:
First pass: Googlebot fetches the HTML response. Initial indexing happens based on this HTML alone.
Second pass: The Web Rendering Service (WRS) executes JavaScript and renders the full DOM. Index is updated with rendered content.
The second pass can lag the first by hours to weeks depending on render queue length. For JS-rendered content, this means a page can be partially indexed (HTML-only) for an extended period before the rendered version is incorporated.
What the WRS supports and doesn't
Modern Chromium engine (always up-to-date with stable Chrome).
Supports modern JavaScript features.
Does NOT execute on user interactions (clicks, hovers, scrolls beyond what auto-fires).
Has time and resource budgets per page; complex pages can time out.
Does not maintain session state across pages.
Caches resources aggressively; cache-busted resources may not be re-fetched on each render.
What breaks rendering
Content that requires user interaction to load (infinite scroll without scroll-trigger, click-to-expand).
JavaScript errors that prevent execution.
Blocking third-party scripts that fail or take too long.
API calls that depend on cookies, session state, or authenticated users.
Lazy-loaded content that depends on viewport interactions Google can't replicate.
Heavy client-side routing that doesn't fall through to server-rendered HTML.
Even though Googlebot can render JavaScript, we don’t want to rely on that.
Cautionary case · Hulu.com · the JS bet that went wrong
-56%search visibility404served to Googlebot200served to users
Hulu moved to client-side JavaScript rendering and its search visibility dropped by more than half. The widely cited diagnosis: rendering errors meant Googlebot was effectively seeing 404s on pages that returned 200 to humans — the site looked alive to users and dead to the crawler. Nobody noticed for weeks because every dashboard the team watched measured humans, not bots. The lesson this module keeps repeating: monitor what Googlebot experiences, not what your browser shows you. (LinkResearchTools / Elephate-Onely case write-up)
RGM EXPERT TRICK
The JS-off diff crawl: find rendering debt in ten minutes
We crawl every new client twice in Screaming Frog: once with JavaScript rendering on, once off. Then we diff word count and internal link count per template.
Any template losing more than ~20% of its words or links with JS off goes straight onto the SSR candidate list. No waiting for Google, no arguing with feelings — a spreadsheet decides.
The same diff catches the quiet killers: client-side canonical rewrites and meta-robots tags injected by JS, which Wave 1 never sees.
WHY IT’S RARE · Teams test rendering by eyeballing one page in URL Inspection. The per-template diff turns an anecdote into an inventory.
JS framework patterns
Framework
SEO-recommended pattern
Next.js
SSG (getStaticProps) for marketing content; ISR for refreshable pages; SSR (getServerSideProps) for dynamic data. Avoid pure CSR for indexable pages.
Nuxt
Universal mode (SSR) or static generation. Avoid SPA mode for indexable content.
Remix
Always server-rendered by design; SEO-friendly out of the box.
SvelteKit
SSR or prerendering. Avoid client-only routes for SEO content.
Astro
Static-first with islands of interactivity. SEO-friendly.
React (CRA)
Add prerendering (Prerender.io) or migrate to Next.js for SEO content.
Vue (no framework)
Add server-side rendering or static generation; consider Nuxt.
Angular
Use Angular Universal for SSR.
Interactive Pick your stack — get the SEO-safe pattern
Eight stacks, eight verdicts
Next.js · verdict: excellent — choose the right mode per template
SSG for marketing pages, ISR for refreshable catalogs, SSR for truly dynamic data. The trap is shipping indexable templates in pure client mode because the default felt fine in dev.
Nuxt · verdict: excellent in universal mode
Universal (SSR) mode or full static generation. SPA mode for anything indexable is the classic Nuxt mistake — it quietly turns your content site into a rendering-queue gamble.
Remix · verdict: SEO-friendly by design
Server-rendered by architecture; there is no client-only escape hatch to misuse. Your remaining risks are ordinary ones: canonicals, status codes, and redirects.
SvelteKit · verdict: good — prerender what you can
SSR by default with per-route prerendering. Watch for client-only load functions on indexable routes and verify with the JS-off diff crawl.
Astro · verdict: excellent for content
Static-first islands architecture: HTML ships complete, interactivity hydrates per component. For content-heavy sites this is the lowest-risk modern stack on the board.
React (CRA / bare SPA) · verdict: liability for indexable content
Pure client rendering: empty shell, render-queue dependency, invisible to AI crawlers. Migrate indexable routes to Next.js or add prerendering as a bridge — and treat the bridge as debt.
Vue (bare SPA) · verdict: same liability, Vue flavor
Identical failure mode to bare React. Nuxt universal mode is the standard exit; prerendering services are the temporary patch.
Angular · verdict: fine with Universal, risky without
Angular Universal provides SSR; without it you are betting your revenue pages on the second wave. Hybrid rendering per route landed in recent versions — use it.
Indexation controls
Mechanism
Effect
When to use
robots.txt Disallow
Blocks crawling. Page may still be indexed without content if linked externally.
Block crawl budget waste, never sole indexation control.
meta robots noindex
Allows crawling, blocks indexing. Most reliable indexation control.
Thin content, duplicates, internal pages.
X-Robots-Tag noindex
HTTP-header version of noindex. Works for non-HTML resources.
PDFs, images, downloadable files you don't want indexed.
Canonical tag
Hints to Google which version is preferred among duplicates.
Speed of removal How fast each mechanism gets a page out of the index
Relative de-indexing speed — RGM analysis from remediation projects
Removals tool
hours (temporary)
410 Gone
days
404
days-weeks
noindex
next recrawl+
robots.txt alone
may never
The Removals tool hides for ~6 months while you fix the real signal — it is an emergency brake, not a removal. robots.txt alone can leave a ghost listing indexed forever. Pair the brake with 410/noindex so the temporary becomes permanent.
Monitoring indexation health
Indexation rate: URLs submitted in sitemap vs URLs indexed (Search Console Coverage report). Sub-80% indicates a problem.
Index Coverage exclusions: Review "Excluded" reasons. "Crawled - currently not indexed" is the most common warning sign.
Page Indexing — Why pages aren't indexed: Search Console's grouped reason list. Use for diagnosis.
Search Console API: Automate monitoring at scale. Pull crawl stats, indexation status into your data warehouse.
Crawler vs index comparison: Run Screaming Frog crawl; compare list to Search Console's indexed pages. Gaps are diagnostic.
Decoder “Why pages aren’t indexed” — every status, what it really means, what to do
The eight statuses you will actually meet — tap each one
Google knows the URL, never fetched itCRAWL DEMAND
Google found the URL (links, sitemap) but decided fetching it was not worth the trip yet. At scale this is a crawl-demand verdict: weak internal links, weak perceived value, or a crawl budget spent elsewhere.
THE FIX · Strengthen internal links from high-crawl pages, confirm the template is in a segmented sitemap, and reduce waste so demand concentrates. Mass “request indexing” does not move this.
Google fetched it and declined to keep itQUALITY VERDICT
The most misread status in Search Console. Google spent the fetch, evaluated the page, and chose not to index it. This is feedback about the page, not a bug in the pipeline.
THE FIX · Sample 20 affected URLs and bucket them: thin, duplicative, or orphaned. Prune, consolidate, or differentiate — then watch the bucket shrink over 4-8 weeks.
You forgot to declare a preferenceCANONICAL GAP
Google found near-identical pages and no canonical to arbitrate, so it picked one to fold the rest into. Sometimes its pick is fine; sometimes your money URL just got consolidated into a parameter variant.
THE FIX · Add self-referencing canonicals on the preferred version and canonical-to-parent on variants. Verify with URL Inspection: declared and Google-selected should match.
You declared, Google disagreedSIGNAL CONFLICT
You set a canonical and Google overrode it — which means your other signals (internal links, sitemaps, redirects, content similarity) point somewhere else. Google believes your site more than your tag.
THE FIX · Audit the contradiction: which URL do internal links use? Which is in the sitemap? Align every signal on one URL and Google usually concedes within a few crawls.
Working exactly as designedNO ACTION
The page canonicals to another URL and Google agrees. Faceted variants, print pages, UTM-tagged duplicates — this is the system doing its job.
THE FIX · Nothing. Spend your attention on statuses 02 and 04. Treating this bucket as a problem is how teams burn weeks on non-issues.
You asked Google not to index itVERIFY INTENT
Indexing is blocked by your own meta robots or X-Robots-Tag. Fine if intentional — catastrophic if a staging flag shipped to production, which we have seen take out entire template families.
THE FIX · Diff the noindexed list against your intended exclusions every month. Anything unexpected is a sev-1: find the tag source (template, CMS plugin, JS injection) and ship the fix same-day.
Returns 200 but Google read “empty”PHANTOM PAGE
The server says success; the content says nothing. Empty category pages, JS shells that never painted, “0 results” pages — Google classifies them as soft 404s and drops them.
THE FIX · Make empties honest: return a real 404/410, or fill the page (related items, content). For JS shells, this status is your rendering failure alarm — run the JS-off diff crawl.
Blocked from crawling — not from indexingWRONG TOOL?
robots.txt stops the fetch, not the listing. If the block is intentional waste control, good. If you meant to deindex, the page can linger as a ghost listing — Google can never see your noindex.
THE FIX · For waste control: leave it. For removal: unblock, serve noindex or 410, wait for the drop, then re-block if needed. Sequence matters.
Calculator Your indexation rate — the one KPI this module reports to
Two numbers from the Pages report → a verdict
80%indexed
AT THE HEALTHY LINE
Thresholds are RGM analysis: ≥90% strong · 80–89% watch · 60–79% structural problem · <60% systemic. Context from Google: indexing 100% never happens — see the quote below. Segment sitemaps by template to see WHERE the gap lives.
It’s completely normal that we don’t index everything off of the website… it’s never going to be the case that we index 100% of everything that’s on a website.
Google does not need your sitemap to find pages on a well-linked site. So we stopped treating sitemaps as discovery and started treating them as measurement.
We split them by template and age cohort: products-2026.xml, products-legacy.xml, articles-2026.xml. The Pages report then shows indexation rate per segment — a per-template health dashboard Google maintains for free.
When articles-2026 indexes at 96% and products-legacy at 41%, the roadmap writes itself. One number per sitemap tells us which template Google dislikes this month.
WHY IT’S RARE · Most sites ship one monolithic sitemap, so their indexation rate is a site-wide average that diagnoses nothing.
The deprecation graveyard: crutches Google took away
Half of the crawl-control advice still circulating on the internet references tools that no longer exist. Google has spent seven years removing every manual override and replacing them with one answer: fix your architecture. If a playbook you inherit mentions any of these, you are reading archaeology — tap each headstone for what replaced it.
Interactive Five retired controls — and what replaced each
RIP: the manual-override era, 2019–2024
rel=prev/next † — pagination hints
Google admitted it had ignored the markup “for years” before announcing the deprecation. Replaced by: self-referencing canonicals per paginated page and real internal links from page one. The lesson that aged best: Google drops signals quietly and tells you later.
URL Parameters tool † — Search Console
Retired April 26, 2022; Google said only ~1% of stored configurations were even useful for crawling. Replaced by: canonicalization, robots rules, and not generating junk parameter URLs in the first place.
Sitemaps ping endpoint †
The “ping Google when the sitemap changes” endpoint went away. Replaced by: accurate lastmod values — which Google now actually reads, making sitemap honesty a feature.
Crawl-rate limiter † — Search Console
Removed January 8, 2024, after a decade. Crawl rate is now fully automatic, set by how fast your server responds. Replaced by: server performance — the only crawl-rate lever left is being fast.
cache: operator † — Google Search
The cache view died in 2024; Google now points to the Wayback Machine instead. Replaced by: URL Inspection’s rendered HTML — the only remaining way to see a page through Google’s eyes.
Crawl budget reclamation project. For large sites: identify top 10 crawl-waste patterns from log analysis, build remediation plan (block, canonical, consolidate), measure improvement in Search Console crawl stats and indexation rate over 90 days.
Render-mode audit. Test each page template (View Source vs DevTools rendered DOM vs Search Console URL Inspection rendered HTML). Identify template-level rendering issues.
JavaScript framework migration risk plan. If migrating to or from a JS framework, do a phased rollout with rigorous before/after comparison of organic traffic, indexation, and rendering.
Faceted navigation policy document. Catalog every filter parameter; classify each as indexable / canonical-to-parent / robots-blocked / noindex. Implement consistently across templates.
Sitemap segmentation. Split sitemaps by content type (products, articles, categories) and monitor indexation rate per segment. Different decay patterns indicate different problems.
Stale-while-revalidate caching for JS-rendered pages. ISR (incremental static regeneration) in Next.js, or stale-while-revalidate at the CDN level, lets you have static-fast SEO without losing dynamic data.
Prerendering services as bridge. Prerender.io, Rendertron, or custom puppeteer services can serve pre-rendered HTML to bots while users get client-side rendering. Useful as a migration bridge.
Pagination strategy. Modern best practice: self-referencing canonicals on each paginated page, internal links from page 1 to other pages, no rel=prev/next (deprecated). Audit for compliance.
Bot verification. Implement reverse-DNS verification on suspicious user-agents. Block fake Googlebots that waste crawl resources without contributing to indexing.
Search Console URL Inspection bulk via API. For high-stakes pages, automated daily inspection catches indexing issues fast.
Step by step The 10-minute “why isn’t this page indexed?” decision path
Run the gates in order — stop at the first failure
URL Inspection first. Always.One lookup answers five questions: known? crawled? indexed? which canonical did Google choose? what did the rendered HTML contain?
Gate 1 — can it be fetched?Check robots.txt for the path, then the status code Googlebot sees (curl with a Googlebot UA). A 200 for you can be a 404 for the bot — ask Hulu.
Gate 2 — does the content survive rendering?Compare raw HTML to the Inspection tool’s rendered HTML. Headline, body, links present in both? If only in rendered, you are betting on Wave 2.
Gate 3 — is indexing blocked or redirected?noindex (meta or header)? Canonical pointing away — and does Google’s selected canonical match yours? Misalignment = your other signals disagree.
Check crawl demand.How many internal links point here, from where, how deep is it? An orphan at depth 6 is not unindexed — it is unloved.
Check the quality verdict.If it was crawled and declined, compare it honestly against what already ranks. Thin, duplicative, or templated pages do not need a fix — they need a reason to exist.
Fix the failed gate, then re-request once.One re-index request after a real fix is signal. Ten requests without a fix is noise Google learned to ignore years ago.
Common mistakes
Pure client-side React/Vue site with no SSR/SSG for content pages — rendering delays cripple visibility.
Blocking JS or CSS in robots.txt — rendered HTML is broken from Google's perspective.
Using robots.txt to noindex pages (wrong tool — blocks crawl, doesn't prevent indexing).
Auto-generated faceted navigation indexable with no canonical strategy.
Sitemap including 404s, redirects, noindex URLs — signals quality issues.
Pagination with infinite scroll and no fallback URLs for paginated pages.
Internal links pointing to 4xx or 3xx URLs — wasted crawl, lost equity.
No log analysis discipline; flying blind on crawl behavior.
Calendar widgets with infinite future date URLs indexable.
Trusting only Search Console's reported "Indexed" count without sampling for actual searchability.
Migrating to JS framework without an SEO plan; permanent traffic loss.
Ten questions, CASE method (Context · Analysis · Strategy · Execution). Pass at 90% to unlock this module’s completion passcode — retake as many times as you like.