Google warns against using 403 or 404 status codes for Googlebot crawl rate limiting

Google is warning towards utilizing 404 and different 4xx consumer server standing errors, resembling 403s, for the aim of making an attempt to set a crawl charge restrict for Googlebot. “Please don’t try this,” Gary Illyes from the Google Search Relations crew wrote.

Why the discover. There was a current improve within the variety of websites and CDNs utilizing these strategies to attempt to restrict Googlebot crawling. “Over the previous few months we seen an uptick in web site house owners and a few content material supply networks (CDNs) making an attempt to make use of 404 and different 4xx consumer errors (however not 429) to aim to scale back Googlebot’s crawl charge,” Gary Illyes wrote.

What to do as an alternative. Google has a detailed help document simply on the subject of decreasing Googlebot crawling in your website. The beneficial method is to make use of the Google Search Console crawl charge settings to regulate your crawl charge.

Google defined, “To shortly scale back the crawl charge, you possibly can change the Googlebot crawl rate in Search Console. Adjustments made to this setting are usually mirrored inside days. To make use of this setting, first verify your site ownership. Just remember to keep away from setting the crawl charge to a worth that’s too low on your website’s wants. Study extra about what crawl budget means for Googlebot. If the Crawl Rate Settings is unavailable on your website, file a special request to scale back the crawl charge. You can not request a rise in crawl charge.”

Should you can’t try this, Google then says “scale back the crawl charge for brief time frame (for instance, a few hours, or 1-2 days), then return an informational error web page with a 500, 503, or 429 HTTP response standing code.”

Why we care. Should you seen crawling points, possibly your internet hosting supplier or CDN just lately deployed these strategies. Chances are you’ll need to submit a help request with them to indicate them Google’s weblog submit on this matter to make sure they aren’t utilizing 404s or 403s to scale back crawl charges.