{"id":4940,"date":"2022-01-21T01:54:04","date_gmt":"2022-01-21T01:54:04","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2022\/01\/21\/google-considers-reducing-webpage-crawl-rate\/"},"modified":"2022-01-21T01:54:04","modified_gmt":"2022-01-21T01:54:04","slug":"google-considers-reducing-webpage-crawl-rate","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2022\/01\/21\/google-considers-reducing-webpage-crawl-rate\/","title":{"rendered":"Google Considers Reducing Webpage Crawl Rate"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n

Google may reduce the frequency of crawling webpages as it grows more conscious of the sustainability of crawling and indexing.<\/p>\n

This topic is discussed by Google\u2019s Search Relations team, which is made up of John Mueller, Martin Splitt, and Gary Illyes.<\/p>\n

Together, in the latest episode of the Search Off the Record podcast, they discuss what to expect from Google in 2022 and beyond.<\/p>\n

Among the topics they address is crawling and indexing, which SEO professionals and website owners say they\u2019ve seen less of over the past year.<\/p>\n

That\u2019s going to be a key focus for Google this year as it aims to make crawling more sustainable by conserving computing resources.<\/p>\n

Here\u2019s what that will mean for your website and its performance in search results.<\/p>\n

Sustainability Of Crawling & Indexing<\/h2>\n

Since Googlebot crawling and indexing happens virtually, it\u2019s not something you may think has an impact on the environment.<\/p>\n

Illyes brings this issue to attention when he says computing isn\u2019t sustainable in general:<\/p>\n

\n

\u201c\u2026 what I mean is that computing, in general, is not really sustainable. And if you think of Bitcoin, for example, Bitcoin mining has real impact on the environment that you can actually measure, especially if the electricity is coming from coal plants or other less sustainable plants.<\/p>\n

We are carbon-free, since I don\u2019t even know, 2007 or something, 2009, but it doesn\u2019t mean that we can\u2019t reduce even more our footprint on the environment. And crawling is one of those things that early on, we could chop off some low-hanging fruits.\u201d<\/p>\n<\/blockquote>\n

The low-hanging fruits, in this instance, refers to unnecessary web crawling. Such as crawling webpages that haven\u2019t had any recent updates.<\/p>\n

How Will Google Make Crawling More Sustainable?<\/h2>\n

Illyes goes on to explain that web crawling can be made more sustainable by cutting down on refresh<\/strong> crawls.<\/p>\n

There are two types of Googlebot crawling: crawling to discover new content and crawling to refresh existing content.<\/p>\n

Google is considering scaling back on crawling to refresh content.<\/p>\n

Illyes continues:<\/p>\n

\n

\u201c\u2026 one thing that we do, and we might not need to do that much, is refresh crawls. Which means that once we discovered a document, a URL, then we go, we crawl it, and then, eventually, we are going to go back and revisit that URL. That is a refresh crawl.<\/p>\n

And then every single time we go back to that one URL, that will always be a refresh crawl. Now, how often do we need to go back to that URL?\u201d<\/p>\n<\/blockquote>\n

He goes on to give an example of certain websites that warrant a significant number of refresh crawls for some parts of the site but not others.<\/p>\n

A website like Wall Street Journal is constantly updating its homepage with new content, so it deserves a lot of refresh crawls.<\/p>\n

However, WSJ is not likely updating its About page as frequently, so Google doesn\u2019t need to keep doing refresh crawls on those types of pages.<\/p>\n

\n

\u201cSo you don\u2019t have to go back there that much. And often, we can\u2019t estimate this well, and we definitely have room for improvement there on refresh crawls. Because sometimes it just seems wasteful that we are hitting the same URL over and over again.<\/p>\n

Sometimes we are hitting 404 pages, for example, for no good reason or no apparent reason. And all these things are basically stuff that we could improve on and then reduce our footprint even more.\u201d<\/p>\n<\/blockquote>\n

If Google were to cut down on refresh crawls, which is not 100% confirmed, here\u2019s the impact that could have on your website.<\/p>\n

What Does A Reduction In Crawl Rate Mean For Your Website?<\/h2>\n

There\u2019s a belief out there that a high crawl rate is a positive SEO signal, even if you\u2019re not updating your content as often as Google is crawling it.<\/p>\n

That\u2019s a misconception, Illyes says, as content will not necessarily rank better because it gets crawled more.<\/p>\n

Mueller:<\/strong><\/p>\n

\n

\u201cSo I guess that\u2019s kind of also a misconception that people have inthat they think if a page gets crawled more, it\u2019ll get ranked more. Is that correct that that\u2019s a misconception, or is that actually true?\u201d<\/p>\n<\/blockquote>\n

Illyes:<\/strong><\/p>\n

\n

\u201cIt\u2019s a misconception.\u201d<\/p>\n<\/blockquote>\n

Mueller:<\/strong><\/p>\n

\n

\u201cOK, so no need to try to force something to be re-crawled if it doesn\u2019t actually change. It\u2019s not going to rank better.\u201d<\/p>\n<\/blockquote>\n

Again, it\u2019s not confirmed that Google will reduce refresh crawls, but it\u2019s an idea the team is actively considering.<\/p>\n

If Google follows through on this idea it won\u2019t be a bad thing for your website. More crawling does not mean better rankings.<\/p>\n

Moreover, the idea is to learn which pages need<\/strong> refresh crawls and which pages do not. That means the pages you change more often will continue to be refreshed and updated in search results.<\/p>\n

For more details on how Google plans to pull this off, listen to the full discussion in the podcast below (starting at the 2:40 mark):<\/p>\n