{"id":4109,"date":"2022-01-19T03:11:04","date_gmt":"2022-01-19T03:11:04","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2022\/01\/19\/does-google-have-a-problem-with-big-robots-txt-files\/"},"modified":"2022-01-19T03:11:04","modified_gmt":"2022-01-19T03:11:04","slug":"does-google-have-a-problem-with-big-robots-txt-files","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2022\/01\/19\/does-google-have-a-problem-with-big-robots-txt-files\/","title":{"rendered":"Does Google Have A Problem With Big Robots.txt Files?"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n

Google addresses the subject of robots.txt files and whether it\u2019s a good SEO practice to keep them within a reasonable size.<\/p>\n

This topic is discussed by Google\u2019s Search Advocate John Mueller during the Google Search Central SEO office-hours hangout recorded on January 14.<\/p>\n

David Zieger, an SEO manager for a large news publisher in Germany, joins the livestream with concerns about a \u201chuge\u201d and \u201ccomplex\u201d robots.txt file.<\/p>\n

How huge are we talking here?<\/p>\n

Zieger says there\u2019s over 1,500 lines with a \u201cmultitude\u201d of disallows that keeps growing over the years.<\/p>\n

The disallows prevent Google from indexing HTML fragments and URLs where AJAX calls are used.<\/p>\n

Zieger says it\u2019s not possible to set a noindex, which is another way to keep the fragments and URLs out of Google\u2019s index, so he\u2019s resorted to filling the site\u2019s robots.txt with disallows.<\/p>\n

Are there any negative SEO effects that can result from a huge robots.txt file?<\/p>\n

Here\u2019s what Mueller says.<\/p>\n

SEO Considerations For Large Robots.txt Files<\/h2>\n

A large robots.txt file will not directly<\/strong> cause any negative impact to a site\u2019s SEO.<\/p>\n

However, a large file is harder to maintain, which may lead to accidental issues down the road.<\/p>\n

Mueller explains:<\/p>\n

\n

\u201cNo direct negative SEO issues with that, but it makes it a lot harder to maintain. And it makes it a lot easier to accidentally push something that does cause issues.<\/p>\n

So just because it\u2019s a large file doesn\u2019t mean it\u2019s a problem, but it makes it easier for you to create problems.\u201d<\/p>\n<\/blockquote>\n

Zieger follows up by asking if there are any issues with not including a sitemap in the robots.txt file.<\/p>\n

Mueller says that\u2019s not a problem:<\/p>\n

\n

\u201cNo. Those different ways of submitting a sitemap are all equivalent for us.\u201d<\/p>\n<\/blockquote>\n

Zieger then launches into a several more follow-up questions that we\u2019ll take a look at in the next section.<\/p>\n

Does Google Recognize HTML Fragments?<\/h2>\n

Zieger asks Mueller what would be the SEO impact of radically shortening the robots.txt file. Such as removing all the disallows, for example.<\/p>\n

The following questions are asked:<\/p>\n