Screaming Frog SEO Spider: the operator's guide

What Screaming Frog actually does

It crawls a website like Googlebot would, then surfaces every technical SEO issue. The data it collects per URL:

• Status code (200, 301, 302, 404, 500, etc.) and redirect chains.
• Page title, meta description, H1, H2 — length, presence, duplication.
• Word count, hash for duplicate detection.
• Internal links — count, anchor text, link targets.
• External links — same.
• Image references — alt text, size, missing alt detection.
• Canonicals, robots directives, hreflang, structured data.
• Response time, content type, server headers.
• Custom extractions via XPath, CSS selectors, or regex.

The seven workflows every SEO runs in Screaming Frog

1. Technical site audit. Crawl the full site, surface 4xx/5xx errors, redirect chains, missing titles, duplicate content, broken images. The audit takes hours to interpret but produces a remediation list that compounds.
2. Pre-launch QA. Crawl a staging or pre-production site to catch issues before they hit production.
3. Post-migration validation. After a site migration or redesign, crawl to verify nothing broke. Compare to a pre-migration crawl.
4. Internal linking audit. Find pages with few internal links, find pages over-linked to, find broken anchor text. See SEO.
5. Content audit. Pull title, meta, H1, word count, and content hash for every page. Identify thin content, duplicate content, missing meta.
6. Structured data validation. Extract every Schema.org block; verify markup is present, valid, and consistent.
7. Custom extraction. Pull specific page elements via XPath — author bylines, publication dates, product prices, anything in the DOM.

How to crawl effectively

1. Set crawl depth and URL limit based on site size.
2. Configure user-agent (default is Screaming Frog SEO Spider; for some auth or robots scenarios use Googlebot).
3. Respect robots.txt by default; override only for authorized sites you own.
4. Use the JavaScript rendering option for SPAs (React, Vue, Angular sites).
5. Connect to GA4 and Google Search Console via API to enrich crawl data with traffic and performance.
6. Export to CSV or sync to a warehouse for downstream analysis.

Screaming Frog vs Ahrefs Site Audit vs Sitebulb

	Screaming Frog	Ahrefs Site Audit	Sitebulb
Type	Desktop application	Cloud-based	Desktop application
Best for	Pre-launch QA, deep crawls, custom extraction	Scheduled monitoring, integrated with broader SEO tools	Visual reporting, audit storytelling
Pricing	£199/year (paid), free up to 500 URLs	Included with Ahrefs subscription	£35-£90/month
Cloud crawl	No (desktop only)	Yes	No (desktop)

Most mature SEO teams run multiple. Screaming Frog for pre-launch and deep crawls; Ahrefs Site Audit for scheduled monitoring; sometimes Sitebulb for client-facing visual reports.

How Screaming Frog fits the broader SEO stack

• Paired with Ahrefs for backlink and keyword data.
• Paired with Google Search Console for first-party performance data.
• Paired with GA4 for behavioral data.
• Paired with Moz Local for citation management.
• Crawl output flows into the data warehouse via CSV export or API for trend analysis.

Is Screaming Frog free?

Free up to 500 URLs. £199/year for unlimited crawls. The free tier is enough for evaluation; the paid tier is required for any real site work.

Does Screaming Frog work on Mac?

Yes — Mac, Windows, and Linux desktop applications. Same functionality across platforms.

Can Screaming Frog crawl JavaScript-rendered sites?

Yes. Enable JavaScript rendering in configuration. Crawls take longer but capture content that requires JS execution. Required for React, Vue, Angular SPAs.

How long does a full site crawl take?

1,000 URLs: 5-15 minutes. 10,000 URLs: 1-2 hours. 100,000 URLs: 8-24 hours depending on site speed and crawl rate. JavaScript rendering doubles or triples crawl time.

Should I use Screaming Frog or Ahrefs Site Audit?

Both. Screaming Frog for pre-launch QA, deep crawls, custom extraction, and any case where you need full operator control. Ahrefs Site Audit for scheduled cloud monitoring and integration with broader SEO data.

Can I run Screaming Frog in the cloud?

Officially desktop only. Some teams run it on a cloud VM for large crawls; others use Screaming Frog's API to programmatically trigger crawls from automated pipelines.

Operating checklist

1. Define the use case before installing or signing up.
2. Confirm the right tier and access for your scale.
3. Connect the tool to your source-of-truth data systems.
4. Build the first three workflows everyone on the team uses.
5. Set ownership and access controls.
6. Audit usage quarterly; prune integrations you've stopped using.
7. Document the workflows so the next operator can pick them up.