--- title: Screaming Frog SEO Spider | RGM® url: https://realgrowthmatters.com/learn/tools/screaming-frog/ updated: 2026-06-10 source_html: https://realgrowthmatters.com/learn/tools/screaming-frog/ --- # Screaming Frog SEO Spider: the operator's guide [Screaming Frog SEO Spider](/learn/tools/screaming-frog/) is the desktop application most serious SEO operators reach for first when they need to crawl, audit, or analyze a website at scale. Founded in 2010 in the UK, Screaming Frog (the company) has stayed deliberately small and the product has stayed deliberately deep — the SEO Spider is the canonical technical SEO tool. By **David Schaefer** · [LinkedIn](https://www.linkedin.com/in/daschaefer/) · Updated May 2026 ## What Screaming Frog actually does It crawls a website like Googlebot would, then surfaces every technical SEO issue. The data it collects per URL: - Status code (200, 301, 302, 404, 500, etc.) and redirect chains. - Page title, meta description, H1, H2 — length, presence, duplication. - Word count, hash for duplicate detection. - Internal links — count, anchor text, link targets. - External links — same. - Image references — alt text, size, missing alt detection. - Canonicals, robots directives, hreflang, structured data. - Response time, content type, server headers. - Custom extractions via XPath, CSS selectors, or regex. ## The seven workflows every SEO runs in Screaming Frog 1. **Technical site audit.** Crawl the full site, surface 4xx/5xx errors, redirect chains, missing titles, duplicate content, broken images. The audit takes hours to interpret but produces a remediation list that compounds. 2. **Pre-launch QA.** Crawl a staging or pre-production site to catch issues before they hit production. 3. **Post-migration validation.** After a site migration or redesign, crawl to verify nothing broke. Compare to a pre-migration crawl. 4. **Internal linking audit.** Find pages with few internal links, find pages over-linked to, find broken anchor text. See [SEO](/learn/seo/local-seo/). 5. **Content audit.** Pull title, meta, H1, word count, and content hash for every page. Identify thin content, duplicate content, missing meta. 6. **Structured data validation.** Extract every Schema.org block; verify markup is present, valid, and consistent. 7. **Custom extraction.** Pull specific page elements via XPath — author bylines, publication dates, product prices, anything in the DOM. ## How to crawl effectively 1. Set crawl depth and URL limit based on site size. 2. Configure user-agent (default is Screaming Frog SEO Spider; for some auth or robots scenarios use Googlebot). 3. Respect robots.txt by default; override only for authorized sites you own. 4. Use the JavaScript rendering option for SPAs (React, Vue, Angular sites). 5. Connect to [GA4](/learn/ga4/ga4-ultimate-guide/) and Google Search Console via API to enrich crawl data with traffic and performance. 6. Export to CSV or sync to a warehouse for downstream analysis. ## Screaming Frog vs Ahrefs Site Audit vs Sitebulb | | Screaming Frog | Ahrefs Site Audit | Sitebulb | | --- | --- | --- | --- | | Type | Desktop application | Cloud-based | Desktop application | | Best for | Pre-launch QA, deep crawls, custom extraction | Scheduled monitoring, integrated with broader SEO tools | Visual reporting, audit storytelling | | Pricing | £199/year (paid), free up to 500 URLs | Included with [Ahrefs](/learn/tools/ahrefs/) subscription | £35-£90/month | | Cloud crawl | No (desktop only) | Yes | No (desktop) | Most mature SEO teams run multiple. Screaming Frog for pre-launch and deep crawls; Ahrefs Site Audit for scheduled monitoring; sometimes Sitebulb for client-facing visual reports. ## How Screaming Frog fits the broader SEO stack - Paired with [Ahrefs](/learn/tools/ahrefs/) for backlink and keyword data. - Paired with Google Search Console for first-party performance data. - Paired with [GA4](/learn/ga4/ga4-ultimate-guide/) for behavioral data. - Paired with [Moz](/learn/tools/moz/) Local for citation management. - Crawl output flows into the data warehouse via CSV export or API for trend analysis. Is Screaming Frog free? Free up to 500 URLs. £199/year for unlimited crawls. The free tier is enough for evaluation; the paid tier is required for any real site work. Does Screaming Frog work on Mac? Yes — Mac, Windows, and Linux desktop applications. Same functionality across platforms. Can Screaming Frog crawl JavaScript-rendered sites? Yes. Enable JavaScript rendering in configuration. Crawls take longer but capture content that requires JS execution. Required for React, Vue, Angular SPAs. How long does a full site crawl take? 1,000 URLs: 5-15 minutes. 10,000 URLs: 1-2 hours. 100,000 URLs: 8-24 hours depending on site speed and crawl rate. JavaScript rendering doubles or triples crawl time. Should I use Screaming Frog or Ahrefs Site Audit? Both. Screaming Frog for pre-launch QA, deep crawls, custom extraction, and any case where you need full operator control. Ahrefs Site Audit for scheduled cloud monitoring and integration with broader SEO data. Can I run Screaming Frog in the cloud? Officially desktop only. Some teams run it on a cloud VM for large crawls; others use Screaming Frog's API to programmatically trigger crawls from automated pipelines. ## Operating checklist 1. Define the use case before installing or signing up. 2. Confirm the right tier and access for your scale. 3. Connect the tool to your source-of-truth data systems. 4. Build the first three workflows everyone on the team uses. 5. Set ownership and access controls. 6. Audit usage quarterly; prune integrations you've stopped using. 7. Document the workflows so the next operator can pick them up. ### In this section - [Tools](/learn/tools/) - [All learn topics](/learn/) - [Concepts](/learn/concepts/) - [Measurement](/learn/measurement/) ### Foundational reading - [What is growth marketing?](/learn/concepts/what-is-growth-marketing/) - [What is performance marketing?](/learn/concepts/what-is-performance-marketing/) - [Server-side vs client-side](/learn/measurement/server-side-vs-client-side-tracking/) - [Vickrey auction](/learn/concepts/vickrey-auction-google-ads/) ### Hubs - [Services](/services/) - [Platforms](/platforms/) - [Areas served](/areas-served/) - [Performance marketing](/services/performance-marketing/)