Crawl budget
What is crawl budget?
The crawl budget concepts takes into consideration that search engines have limited crawling resources and cannot crawl every page on the www instantaneously. The more popular a site through the lense of a search engine the more often they will visit this site and the more time/ressources they will dedicate to crawl it. Google’s crawl budget describes how much time and resources it devotes to crawling a site. Not all pages crawled on your site will be indexed; each page must be evaluated, consolidated, and assessed to determine whether it will be indexed. There are two main factors that determine crawl budgets: crawl capacity limit and crawl demand.
- Crawl Capacity Limit: It is Googlebot’s goal to crawl your site without overburdening your servers. A crawl capacity limit is calculated by Googlebot, and it determines the maximum number of parallel connections that Googlebot can use to crawl a site and the time delay between fetches.
- Crawl Demand: The amount of time Google spends crawling a site depends on its size, frequency of updates, quality, and relevancy