{"id":13912,"date":"2022-02-16T15:01:06","date_gmt":"2022-02-16T15:01:06","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2022\/02\/16\/6-common-robots-txt-issues-and-how-to-fix-them\/"},"modified":"2022-02-16T15:01:06","modified_gmt":"2022-02-16T15:01:06","slug":"6-common-robots-txt-issues-and-how-to-fix-them","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2022\/02\/16\/6-common-robots-txt-issues-and-how-to-fix-them\/","title":{"rendered":"6 Common Robots.txt Issues & And How To Fix Them"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n

Robots.txt is a useful and relatively powerful tool to instruct search engine crawlers on how you want them to crawl your website.<\/p>\n

It is not all-powerful (in Google\u2019s own words<\/a>, \u201cit is not a mechanism for keeping a web page out of Google\u201d) but it can help to prevent your site or server from being overloaded by crawler requests.<\/p>\n

If you have this crawl block in place on your site, you need to be certain it\u2019s being used properly.<\/p>\n

This is particularly important if you use dynamic URLs or other methods that generate a theoretically infinite number of pages.<\/p>\n

In this guide, we will look at some of the most common issues with the robots.txt file, the impact they can have on your website and your search presence, and how to fix these issues if you think they have occurred.<\/p>\n

But first, let\u2019s take a quick look at robots.txt and its alternatives.\n<\/p>\n

What Is Robots.txt?<\/h2>\n

Robots.txt uses a plain text file format and is placed in the root directory of your website.<\/p>\n

It must be in the topmost directory of your site; if you place it in a subdirectory, search engines will simply ignore it.<\/p>\n

Despite its great power, robots.txt is often a relatively simple document, and a basic robots.txt file can be created in a matter of seconds using an editor like Notepad<\/a>.<\/p>\n

There are other ways to achieve some of the same goals that robots.txt is usually used for.<\/p>\n

Individual pages can include a robots meta tag within the page code itself.<\/p>\n

You can also use the X-Robots-Tag HTTP header<\/a> to influence how (and whether) content is shown in search results.<\/p>\n

What Can Robots.txt do?<\/h2>\n

Robots.txt can achieve a variety of results across a range of different content types:<\/p>\n

Web pages can be blocked from being crawled<\/strong>.<\/p>\n

They may still appear in search results, but will not have a text description. Non-HTML content on the page will not be crawled either.<\/p>\n

Media files can be blocked from appearing in Google search results.<\/strong><\/p>\n

This includes images, video, and audio files.<\/p>\n

If the file is public, it will still \u2018exist\u2019 online and can be viewed and linked to, but this private content will not show in Google searches.<\/p>\n

Resource files like unimportant external scripts can be blocked<\/strong>.<\/p>\n

But this means if Google crawls a page that requires that resource to load, the Googlebot robot will \u2018see\u2019 a version of the page as if that resource did not exist, which may affect indexing.<\/p>\n

You cannot use robots.txt to completely block a web page from appearing in Google\u2019s search results.<\/p>\n

To achieve that, you must use an alternative method such as adding a noindex meta tag<\/a> to the head of the page.<\/p>\n

How Dangerous Are Robots.txt Mistakes?<\/h2>\n

A mistake in robots.txt can have unintended consequences, but it\u2019s often not the end of the world.<\/p>\n

The good news is that by fixing your robots.txt file, you can recover from any errors quickly and (usually) in full.<\/p>\n

Google\u2019s guidance to web developers<\/a> says this on the subject of robots.txt mistakes:<\/p>\n

\n

\u201cWeb crawlers are generally very flexible and typically will not be swayed by minor mistakes in the robots.txt file. In general, the worst that can happen is that incorrect [or] unsupported directives will be ignored.<\/p>\n

Bear in mind though that Google can\u2019t read minds when interpreting a robots.txt file; we have to interpret the robots.txt file we fetched. That said, if you are aware of problems in your robots.txt file, they\u2019re usually easy to fix.\u201d<\/p>\n<\/blockquote>\n

6 Common Robots.txt Mistakes<\/h2>\n
    \n
  1. Robots.txt Not In The Root Directory.<\/li>\n
  2. Poor Use Of Wildcards.<\/li>\n
  3. Noindex In Robots.txt.<\/li>\n
  4. Blocked Scripts And Stylesheets.<\/li>\n
  5. No Sitemap URL.<\/li>\n
  6. Access To Development Sites.<\/li>\n<\/ol>\n

    If your website is behaving strangely in the search results, your robots.txt file is a good place to look for any mistakes, syntax errors, and overreaching rules.<\/p>\n

    Let\u2019s take a look at each of the above mistakes in more detail and see how to ensure you have a valid robots.txt file.<\/p>\n

    1. Robots.txt Not In The Root Directory<\/h3>\n

    Search robots can only discover the file if it\u2019s in your root folder.<\/p>\n

    That\u2019s why there should be only a forward slash between the .com (or equivalent domain) of your website, and the \u2018robots.txt\u2019 filename, in the URL of your robots.txt file.<\/p>\n

    If there\u2019s a subfolder in there, your robots.txt file is probably not visible to the search robots, and your website is probably behaving as if there was no robots.txt file at all.<\/p>\n

    To fix this issue, move your robots.txt file to your root directory.<\/p>\n

    It\u2019s worth noting that this will need you to have root access to your server.<\/p>\n

    Some content management systems will upload files to a \u2018media\u2019 subdirectory (or something similar) by default, so you might need to circumvent this to get your robots.txt file in the right place.<\/p>\n

    2. Poor Use Of Wildcards<\/h3>\n

    Robots.txt supports two wildcard characters:<\/p>\n