Crawl budget is the set of URLs Google can and wants to crawl on your site. It's determined by two things: how fast your server can handle requests, and how badly Google wants what you're serving. Most sites never bump up against either limit. The ones that do share a specific set of technical patterns worth understanding.
Google's own guidance puts explicit thresholds on when you should care: sites with 1M+ pages updating weekly, or 10K+ pages updating daily. If your pages get indexed the same day you publish them and your Search Console coverage reports look clean, you can stop reading here. Seriously. Go optimize something else.
The reason most SEOs have heard of crawl budget is that it sounds like something you should optimize. It has "budget" in the name. But treating it like a scarce resource you need to micromanage is, for the vast majority of sites, a waste of time that could go toward content or links.
Two Levers, One You Barely Control
Google splits crawl budget into crawl capacity limit and crawl demand. Capacity is the mechanical ceiling: how many simultaneous connections Googlebot will open to your server without degrading performance. If your server responds fast, capacity goes up. If it starts throwing errors or slowing down, Google backs off. You control this indirectly through server performance and infrastructure.
Crawl demand is the interesting half. It's Google's assessment of how much of your site is worth crawling. Four things drive it:
Perceived inventory. If Google sees thousands of URLs that resolve to near-identical content, it scales back. Duplicate pages, parameter variations, and session IDs all inflate perceived inventory without adding value.
Popularity. Pages that get links and traffic get crawled more often. Pages nobody visits get crawled less. This is self-reinforcing and largely outside your control.
Staleness. Content that changes frequently gets recrawled more aggressively. Static pages that haven't changed in years get deprioritized.
Crawl health. Server response times, error rates, overall site stability. Google explicitly notes that capacity increases when a site responds quickly and decreases when slowdowns or server errors occur.
For most sites, the only lever worth pulling is crawl health. Make your server fast and reliable. Everything else is either out of your hands or only relevant at serious scale.
Faceted Navigation: Where Crawl Budget Actually Breaks
If there's a single villain in the crawl budget story, it's faceted navigation. According to Gary Illyes at Google, faceted navigation accounts for 50% of all crawling challenges Google deals with. Action parameters (like ?add_to_cart=true) make up another 25%, and irrelevant parameters like session IDs and UTM codes account for 10%.
The math explains why. A retailer with 1,000 products, five filters, and 10 criteria per filter could theoretically generate millions of unique URLs. Every color-size-price-brand-rating combination creates a crawlable page. Most of those pages show near-identical product listings with minor sort differences. Googlebot doesn't know that until it crawls them.
This is the key insight from Illyes: Googlebot "cannot make a decision about whether that URL space is good or not unless it crawled a large chunk of that URL space." The crawler has to taste enough of your parameter soup to decide most of it isn't worth indexing. On a site generating millions of filter combinations, that's a lot of soup.
The damage isn't just theoretical. These crawling loops can consume enough server resources to degrade performance for actual users. You end up with Googlebot hammering your server to crawl pages nobody wants indexed, while your important product pages and category pages wait in line.
What Doesn't Work (and What Does)
A few common "fixes" that don't actually help:
Noindex tags on junk URLs. Google still has to crawl the page to discover the noindex directive. You've saved indexing resources but not crawl resources. If the goal is keeping Googlebot away from worthless URLs entirely, robots.txt is the right tool. Block the parameter patterns at the door.
Temporarily blocking pages via robots.txt to "reallocate" budget. Google's documentation is clear that this doesn't redistribute crawl demand unless your serving capacity is genuinely maxed out.
What actually works is straightforward, if not always easy to implement:
Block junk URL patterns in robots.txt. Disallow: /*?sort= and Disallow: /*?sessionid= keep Googlebot from chasing parameter tails. This is the single highest-leverage fix for faceted navigation crawl waste.
Canonical tags on filter pages pointing back to the parent category. This doesn't stop crawling, but it consolidates indexing signals and tells Google which version matters.
Clean sitemaps with accurate <lastmod> dates. Don't list URLs you've blocked in robots.txt. Don't include pages returning soft 404s. Your sitemap should be a curated list of what you actually want indexed, not a dump of every URL your CMS generates.
Fix redirect chains and soft 404s. Both waste crawl capacity on pages that lead nowhere useful.
The 2025 Efficiency Shift
Here's why crawl efficiency matters more now than it did two years ago, even for sites that previously didn't think about it.
Cloudflare's 2025 data shows Googlebot volume up 96% year-over-year, generating 4.5% of all HTML request traffic. At the same time, AI crawler activity surged more than 15x. Your server isn't just handling Googlebot anymore; it's handling GPTBot, ClaudeBot, PerplexityBot, and dozens of others competing for the same resources.
Google's crawl-to-traffic ratio ranges from 3:1 to 30:1, meaning for every page Google crawls, it sends between one-thirtieth and one-third of a visit back. That's efficient compared to some AI crawlers, where Cloudflare measured ratios as high as 500,000:1. But the aggregate load has grown substantially.
This changes the calculus on server performance. A slow site with 100K pages now faces worse crawl problems than a fast site with 1M+ pages, because Googlebot throttles itself based on server responsiveness. Speed isn't just a ranking signal; it's a crawl budget multiplier.
Google's updated guidance also emphasizes mobile-desktop link parity for large sites using separate HTML versions. Limiting links on mobile pages slows Google's discovery of new content, since Google primarily crawls and indexes from mobile. If your mobile templates strip navigation or internal links, you're effectively shrinking your crawlable site from Google's perspective.
Who Should Actually Care
If you run a site under a few hundred thousand pages, your content gets indexed promptly, and you don't see a growing "Discovered, currently not indexed" count in Search Console: crawl budget isn't your problem. Spend your time on content quality, internal linking, and page speed instead.
If you run an ecommerce site, marketplace, or any platform with faceted navigation generating hundreds of thousands (or millions) of parameter URLs: this is your problem. Audit your server logs. See how much of Googlebot's activity goes to filter combinations and parameter variations. If a third or more of crawl activity hits low-value URLs, you have work to do.
The line between "ignore this" and "fix this now" is clearer than the SEO industry usually admits. Check your Search Console coverage. Look at your server logs. The data will tell you which side you're on.