Algorithms & Concepts

Crawl budget

Crawl budget is the number of pages Googlebot will crawl on your site within a given time, shaped by crawl capacity (how much your server can handle) and crawl demand (how much Google wants your content). It mainly matters for large sites.

For a small site, Google can easily crawl everything, so crawl budget is rarely a concern. It becomes important on large sites — big e-commerce catalogues, sites with millions of URLs, or those generating many parameter and faceted URLs — where Googlebot may not reach or refresh every page.

Budget is wasted by things a crawler shouldn't be spending time on: infinite filter combinations, duplicate URLs, redirect chains, soft 404s, and low-value pages. Efficient 304 responses, clean internal linking and a good sitemap help Google prioritise what matters.

The goal is to steer crawling toward pages you want indexed: consolidate duplicates with canonicals, block low-value paths, fix errors, and keep the site fast. This directly supports indexing — crawling is the step before a page can be indexed at all.

This is a technical SEO topic I handle for clients: Technical SEO →

FAQ

Frequently asked questions

No. For most small and medium sites Google crawls everything without issue. Crawl budget becomes a real concern on large sites — hundreds of thousands to millions of URLs, or heavy parameter/faceted navigation — where crawling efficiency affects how quickly pages are found and refreshed.


Work together

Need help with this?

Tell me about your site and where you're stuck. I'll give you an honest read on what would actually move the needle.

Start a conversation →

Open for projects