What is crawl budget?
The crawl budget concepts takes into consideration that search engines have limited crawling resources and cannot crawl every page on the www instantaneously. The more popular a site through the lense of a search engine the more often they will visit this site and the more time/ressources they will dedicate to crawl it. Google’s crawl budget describes how much time and resources it devotes to crawling a site. Not all pages crawled on your site will be indexed; each page must be evaluated, consolidated, and assessed to determine whether it will be indexed. There are two main factors that determine crawl budgets: crawl capacity limit and crawl demand.
- Crawl Capacity Limit: It is Googlebot’s goal to crawl your site without overburdening your servers. A crawl capacity limit is calculated by Googlebot, and it determines the maximum number of parallel connections that Googlebot can use to crawl a site and the time delay between fetches.
- Crawl Demand: The amount of time Google spends crawling a site depends on its size, frequency of updates, quality, and relevancy