Which technical SEO crawler is worth your budget?
Screaming Frog, Sitebulb, and the rest compared on crawl speed, JavaScript rendering, log file analysis, and price, so you can pick the one that matches how you actually work. If a crawl throws up robots.txt issues you didn't expect, our robots.txt generator builds a clean file for you to check against, or use our robots.txt tester to paste the file and check whether a specific path is allowed or blocked for a given crawler, with the exact rule that decided it quoted back to you. If a crawl flags canonical tag problems you want to check on a single page without waiting for a full re-crawl, our canonical URL checker guide walks through what the declared canonical is telling you and the common issues it flags, such as a missing canonical or one pointing to a page that redirects. If the same crawl turns up a redirect chain worth untangling before the next full run, paste the hops into our redirect chain checker and it will report the hop count, the final destination, whether the chain loops, and the single-hop redirect it recommends in place of a 302 sitting where a 301 belongs, working from what you paste in since it can't watch another site's redirects happen the way a crawler can. When the same crawl turns up a title tag, meta description or hreflang link worth checking on one page before the next full crawl runs, paste that page's HTML source into our Meta Tag Checker and it will flag missing, empty, too-long or duplicate tags and build a Google-style snippet preview from them, though it can only read what you paste in and won't fetch the page itself. And where a crawl turns up Core Web Vitals warnings across a site, our Core Web Vitals Checker reads the same field data Google uses for LCP, INP and CLS at the 75th percentile, showing a pass, needs-improvement or fail verdict for a single URL against Google's thresholds, though not every URL carries enough real-user data for that and the checker falls back to the origin-level result when it doesn't.
Read the crawler reviews and guides on the blog Check the free Keyword Difficulty EstimatorWhat a technical SEO crawler actually does
A technical SEO crawler works through a site the way Googlebot does, following links from page to page and recording what it finds along the way: status codes, redirect chains, missing meta tags, duplicate title tags, thin content, and pages that a robots.txt file or a noindex tag is quietly keeping out of the search results. The output is usually a spreadsheet or dashboard listing every issue by URL, which is normally far quicker than checking pages by hand, especially on a site running into the thousands of pages.
Desktop crawler or cloud crawler
Desktop crawlers install on your machine and crawl using your own connection and processing power, which suits one-off audits and smaller sites where you want full control over crawl speed and don't need to share results with a team. Cloud crawlers run on the vendor's infrastructure, schedule recurring crawls automatically, and store historical data so you can see what changed between one crawl and the next. Agencies managing several client sites tend to prefer the cloud option because the scheduling and reporting saves the repeated manual setup.
Why crawl budget matters here
Crawl budget (the number of pages a search engine is willing to crawl on your site in a given period) matters most on large sites, where a crawler can show you where that budget is being wasted, such as on faceted navigation URLs, parameter-based duplicates, or pagination that spirals into thousands of near-identical pages. On a small brochure site this rarely matters. On an ecommerce site with tens of thousands of product and filter combinations, it's often the difference between new stock pages getting indexed quickly and sitting unseen for weeks.
JavaScript rendering is where crawlers differ most
Some crawlers only read the raw HTML a server sends, which misses content that JavaScript builds after the page loads. Others render the page in a headless browser first, closer to what Googlebot actually sees, which matters a great deal on sites built with modern frameworks. If a crawler's report shows a page as empty or missing content that a browser clearly displays, that's usually a rendering gap in the tool, so it's worth checking a sample of flagged pages manually before acting on the report.
Log files add what a crawler can't see
A crawler tells you what your site's structure looks like. Log file analysis (reading the server's own record of which bots visited which URLs, and when) tells you what search engines actually did with that structure, which pages they visited repeatedly and which they ignored. The two are complementary.
What to check before picking one
- Whether it renders JavaScript, and how accurately, if your site relies on it
- Whether pricing is based on pages crawled, projects, or a flat monthly fee
- Whether it stores crawl history so you can track changes over time
- Whether it integrates with Google Search Console or analytics data
- Whether the export format suits how you actually report findings
For quick, specific checks, a single-purpose tool is often faster than firing up a full crawl. Redirect chains are a good example: a crawler will flag them in bulk across a whole site, but confirming exactly what's happening on one URL, and why, is usually quicker with a dedicated checker. The Redirect Chain Checker works from what you paste in, because a browser isn't able to see another site's redirect hops the way a server-side crawler can. Paste the chain one hop per line, status code and URL (so 301 http://example.com/old, then 302 https://example.com/new, then 200 https://www.example.com/final), or paste the raw headers from a curl -I run, and it reports the hop count, the final destination, whether the chain loops back on itself, and http-to-https or www hops that could be collapsed into one. It also flags a temporary 302 or 307 sitting where a permanent 301 or 308 belongs, and a 302 that quietly changes the request method, then gives a plain verdict with the single-hop redirect it recommends instead. This matters for crawl budget because every extra hop is another URL a crawler has to fetch before it reaches the content actually worth indexing, and it matters for link equity because some of the value a link passes on is lost at each hop along the way. There's a URL field that attempts a direct browser fetch, but most sites block that kind of cross-origin read, so it says as much and points you to the paste mode instead, along with a curl -I -L command ready to copy. It runs entirely in the browser, with nothing sent to a server, and it can only tell you what's in the chain you give it, not confirm that chain still matches the live site. The Robots.txt Tester can confirm a similar kind of single issue in seconds without setting up a crawl at all. The Meta Tag Checker works the same way for the on-page tags a full crawl usually flags in bulk: paste a page's HTML source (View Source in the browser) and it parses the title tag, meta description, canonical URL, robots directives, viewport, Open Graph and Twitter card tags, hreflang links and the first H1, each marked pass, warn or fail, with a Google-style snippet preview built from the title and description. There's a URL field that tries a direct fetch instead, but most sites block that kind of cross-origin read, so pasting the source in remains the reliable route, and the tool says what it can't see.
Core Web Vitals sit in a similar spot: many full crawlers bundle them in as an add-on, firing a separate API call for every URL in the crawl, which can slow a large run down considerably. If you only need a read on a handful of pages, the Core Web Vitals Checker does the same job on demand. Paste a URL, choose mobile or desktop, and it calls Google's own public PageSpeed Insights endpoint to pull the LCP, INP and CLS scores at the 75th percentile, showing each one against Google's own thresholds (LCP under 2.5 seconds, INP under 200ms, CLS under 0.1 to pass) with a pass, needs-improvement, or fail verdict and the underlying distribution. Where no field data exists for that specific URL, it offers the origin-level result, and it says what it can't see. It runs entirely client-side, with no server, API key, or third-party proxy involved, so there's nothing to set up and no crawl licence needed for a spot check.
The robots.txt tester takes a pasted copy of the file, a URL path, and a user-agent (Googlebot, Bingbot, or the generic wildcard), then works through the rules in order and tells you whether that path is allowed or disallowed, quoting the exact line responsible so you're not left guessing which rule actually applies. It also flags syntax slips such as an unknown directive or a missing user-agent group, and lists any Sitemap lines it finds. It runs entirely in the browser, so there's a fetch option that tries to pull the file straight from a domain, but most sites block that kind of cross-origin read, so pasting the file in directly is the more reliable route. Any automated check, including a full crawler's report, can flag something incorrectly, so it's worth verifying an unexpected result before treating it as fact. Canonical tags are another area a full crawl will flag in bulk but that's often faster to check one URL at a time: a missing canonical, a canonical that doesn't point back to itself, or one pointing at a URL that redirects elsewhere can all sit unnoticed in a crawl export until you look for them specifically. Our Canonical URL Checker Guide covers how to spot each of these on a single page before you decide whether it's worth rerunning the full crawl. If a crawl turns up pages you'd rather search engines didn't index, the Robots.txt Generator is a useful next step: it takes you through which bots to block, which directories to disallow, and where your sitemap sits, then produces a file ready to upload, so you can build the rule and check it with the tester before it goes live.
Technical SEO crawlers compared by use case
These crawlers all find broken links, duplicate content and indexability problems, but they differ in where they run, how they present findings, and what else they're tied to. Pick based on the size of the sites you work on and whether you need the output to stand alone or feed into other tools. Before you crawl, it's worth confirming what you're actually letting these tools see: our Robots.txt Generator walks you through which bots to allow, which directories to block, and where your sitemap sits, so the crawl results reflect what you intended. If you've already got a robots.txt file in place and just want to check how a specific bot and path fare under the existing rules, the Robots.txt Tester parses the file, quotes the exact line that decides Allow or Disallow for that path, and flags syntax slips like conflicting rules or a missing user-agent group, all in your browser without sending the file anywhere. If a crawl comes back flagging canonical problems on a specific page, the Canonical URL Checker guide lets you check what canonical tag that single URL is actually declaring, whether it's missing, self-referential, or pointing somewhere that redirects, without waiting on a full site crawl to confirm it. Crawls also turn up redirect chains, where a URL bounces through two or three hops before it lands, and each of those hops costs crawl budget and dilutes the link equity passed through it; if you want to check one chain without re-running the whole crawl, our Redirect Chain Checker takes the hops you paste in, or the headers from a curl -I run, and reports the hop count, the final destination, and whether a temporary 302 is sitting where a permanent 301 belongs. The same applies to title tags and meta descriptions: if a crawl flags one as missing, duplicated or too long, our Meta Tag Checker lets you paste that page's HTML source and see the title, description, canonical, robots directives and Open Graph tags each marked pass, warn or fail, alongside a snippet preview of how the page would appear in Google. It can't fetch most live URLs directly, since browsers block that kind of cross-site read, so pasting the source is the route that actually works, and it says so rather than pretending otherwise. Most of these crawlers will also flag page speed problems across a site, but for a quick check on one URL before or after a fix, our Core Web Vitals Checker pulls the real-user LCP, INP and CLS figures Google holds for that page, weighs each against Google's own pass and fail thresholds, and says when there isn't enough field data for that URL to score at all.
| Crawler | Best for | Watch out for | Runs on |
|---|---|---|---|
| Screaming Frog SEO Spider | Freelancers and consultants who want full control over a deep crawl of one site at a time | Desktop-only, and large sites can eat through your machine's memory if you don't tune the crawl settings | Desktop app on Windows, Mac or Linux, running on your own hardware |
| Sitebulb | Agencies who need to hand a client a report that explains issues | Raw data export and scripting are less flexible than Screaming Frog if you're building your own workflows | Desktop app, with an option to run crawls in the cloud |
| Ahrefs Site Audit | Teams already paying for Ahrefs who want crawl data sitting next to keyword and backlink data | How often you can crawl and how deep depends on the Ahrefs plan you're on | Cloud-based, scheduled from inside your Ahrefs account |
| Lumar (formerly DeepCrawl) | Enterprise sites that need scheduled crawls across thousands of URLs, tied into a wider development workflow | More platform than a small site needs, both in cost and in the setup time it asks for | Cloud platform with API access for integrating into other systems |
| OnCrawl | Sites that want crawl data checked against log files (records of what search engine bots actually requested from your server) to see where crawl budget is going | Getting value out of it depends on already collecting and exporting server logs cleanly | Cloud platform built around log file ingestion |
This comparison is drawn from how each tool is built, priced and marketed, not from crawls run side by side on the same site. Check current plan limits and crawl allowances directly with each vendor before committing, as these change often.
What a technical SEO crawler actually does
A technical SEO crawler works through a site the way Googlebot does, following links from page to page and recording what it finds along the way: status codes, redirect chains, missing meta tags, duplicate title tags, thin content, and pages that a robots.txt file or a noindex tag is quietly keeping out of the search results. The output is usually a spreadsheet or dashboard listing every issue by URL, which is normally far quicker than checking pages by hand, especially on a site running into the thousands of pages.
Desktop crawler or cloud crawler
Desktop crawlers install on your machine and crawl using your own connection and processing power, which suits one-off audits and smaller sites where you want full control over crawl speed and don't need to share results with a team. Cloud crawlers run on the vendor's infrastructure, schedule recurring crawls automatically, and store historical data so you can see what changed between one crawl and the next. Agencies managing several client sites tend to prefer the cloud option because the scheduling and reporting saves the repeated manual setup.
Why crawl budget matters here
Crawl budget (the number of pages a search engine is willing to crawl on your site in a given period) matters most on large sites, where a crawler can show you where that budget is being wasted, such as on faceted navigation URLs, parameter-based duplicates, or pagination that spirals into thousands of near-identical pages. On a small brochure site this rarely matters. On an ecommerce site with tens of thousands of product and filter combinations, it's often the difference between new stock pages getting indexed quickly and sitting unseen for weeks.
JavaScript rendering is where crawlers differ most
Some crawlers only read the raw HTML a server sends, which misses content that JavaScript builds after the page loads. Others render the page in a headless browser first, closer to what Googlebot actually sees, which matters a great deal on sites built with modern frameworks. If a crawler's report shows a page as empty or missing content that a browser clearly displays, that's usually a rendering gap in the tool, so it's worth checking a sample of flagged pages manually before acting on the report.
Log files add what a crawler can't see
A crawler tells you what your site's structure looks like. Log file analysis (reading the server's own record of which bots visited which URLs, and when) tells you what search engines actually did with that structure, which pages they visited repeatedly and which they ignored. The two are complementary.
What to check before picking one
- Whether it renders JavaScript, and how accurately, if your site relies on it
- Whether pricing is based on pages crawled, projects, or a flat monthly fee
- Whether it stores crawl history so you can track changes over time
- Whether it integrates with Google Search Console or analytics data
- Whether the export format suits how you actually report findings
For quick, specific checks, a single-purpose tool is often faster: the Robots.txt Tester can confirm one issue in seconds without setting up a crawl at all. The Meta Tag Checker works the same way for the on-page tags a full crawl usually flags in bulk: paste a page's HTML source (View Source in the browser) and it parses the title tag, meta description, canonical URL, robots directives, viewport, Open Graph and Twitter card tags, hreflang links and the first H1, each marked pass, warn or fail, with a Google-style snippet preview built from the title and description. There's a URL field that tries a direct fetch instead, but most sites block that kind of cross-origin read, so pasting the source in remains the reliable route, and the tool says what it can't see.
Redirect chains are another thing a full crawl will flag by URL but that's often quicker to untangle one chain at a time, since a crawler's export tells you a chain exists without always making the hops themselves easy to read. The Redirect Chain Checker takes a pasted list of hops, one per line in the form status and URL, for example 301 http://example.com/old then 302 https://example.com/new then 200 https://www.example.com/final, or the raw output of a curl -I run, and works out the hop count, the final destination, whether the chain loops, and whether an http-to-https or www hop could be collapsed into the single redirect it recommends instead. It also flags a temporary (302/307) hop sitting where a permanent (301/308) one belongs, and a method-changing 302, both of which matter for crawl budget and for passing link equity through to the final URL, and it shows the chain as a timeline with a status badge on each hop. Because a browser isn't able to observe another site's redirect hops the way a server-side crawler can, there's also a URL field that runs a best-effort fetch and reports only whether the final response was redirected, with a curl -I -L command printed for when the target blocks that read, and the tool says what it can't see. Chains longer than two hops are flagged as too long for crawlers to reliably follow, which is worth checking on any URL a full crawl report shows taking three or four hops to reach a page that should redirect in one.
Core Web Vitals sit in a similar spot: many full crawlers bundle them in as an add-on, firing a separate API call for every URL in the crawl, which can slow a large run down considerably. If you only need a read on a handful of pages, the Core Web Vitals Checker does the same job on demand. Paste a URL, choose mobile or desktop, and it calls Google's own public PageSpeed Insights endpoint to pull the LCP, INP and CLS scores at the 75th percentile, showing each one against Google's own thresholds (LCP under 2.5 seconds, INP under 200ms, CLS under 0.1 to pass) with a pass, needs-improvement, or fail verdict and the underlying distribution. Where no field data exists for that specific URL, it offers the origin-level result, and it says what it can't see. It runs entirely client-side, with no server, API key, or third-party proxy involved, so there's nothing to set up and no crawl licence needed for a spot check.
The robots.txt tester takes a pasted copy of the file, a URL path, and a user-agent (Googlebot, Bingbot, or the generic wildcard), then works through the rules in order and tells you whether that path is allowed or disallowed, quoting the exact line responsible so you're not left guessing which rule actually applies. It also flags syntax slips such as an unknown directive or a missing user-agent group, and lists any Sitemap lines it finds. It runs entirely in the browser, so there's a fetch option that tries to pull the file straight from a domain, but most sites block that kind of cross-origin read, so pasting the file in directly is the more reliable route. Any automated check, including a full crawler's report, can flag something incorrectly, so it's worth verifying an unexpected result before treating it as fact. Canonical tags are another area a full crawl will flag in bulk but that's often faster to check one URL at a time: a missing canonical, a canonical that doesn't point back to itself, or one pointing at a URL that redirects elsewhere can all sit unnoticed in a crawl export until you look for them specifically. Our Canonical URL Checker Guide covers how to spot each of these on a single page before you decide whether it's worth rerunning the full crawl. If a crawl turns up pages you'd rather search engines didn't index, the Robots.txt Generator is a useful next step: it takes you through which bots to block, which directories to disallow, and where your sitemap sits, then produces a file ready to upload, so you can build the rule and check it with the tester before it goes live.
Not sure which crawler fits your site?
The right crawler depends on site size, budget and how deep you need to go into crawl budget (the number of pages a search engine will bother crawling on your site) and log file analysis. Compare the options above before you commit to a subscription. If part of what you're checking is which bots you're letting in, the Robots.txt Generator will draft a file for you to upload, so you can test the result with whichever crawler you pick. Once you have a file in place, or if you inherited one and aren't sure what it actually permits, the Robots.txt Tester lets you paste the contents alongside a URL path and a user-agent (Googlebot, Bingbot, or generic), then tells you whether that path is allowed or disallowed and quotes the exact line that decided it, which is worth doing before a paid crawl burns budget on pages a search engine was never going to reach anyway. If canonical tags are part of the audit, the Canonical URL Checker Guide covers how to spot a missing canonical, a self-referential one, or one pointing at a URL that redirects, before a full site crawl turns up the same problem at scale. A canonical that points into a redirect is one way budget gets wasted on a hop the crawler didn't need to make, and the Redirect Chain Checker lets you paste the hops (either as status codes and URLs, one per line, or the raw headers from a curl -I run) to see the hop count, the final destination, and whether a temporary redirect is sitting where a permanent one belongs, before a paid crawl turns up the same chains across the whole site. Before you point a crawler at the whole site, the Meta Tag Checker will parse a single page's source, the same HTML you'd get from View Source, and flag a missing or duplicate title, an empty meta description, a canonical pointing somewhere unexpected, or a stray noindex, entirely in your browser with nothing sent to a server. It won't explain why a crawl budget is going to waste, but it's a fast way to rule out the obvious before a paid crawl goes looking for it. If page experience is also on the list, the Core Web Vitals Checker pulls Google's own field data for a URL and shows LCP, INP and CLS against Google's pass thresholds, so you can see whether real visitors are having a slow experience before a crawler flags which pages might be causing it. It only reports what Google has recorded, so a low-traffic page may show nothing at all, and the origin-level result is offered instead when that happens.