{"id":123972,"date":"2026-04-23T17:09:13","date_gmt":"2026-04-23T17:09:13","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2026\/04\/23\/google-may-expand-unsupported-robots-txt-rules-list\/"},"modified":"2026-04-23T17:10:37","modified_gmt":"2026-04-23T17:10:37","slug":"google-may-expand-unsupported-robots-txt-rules-list","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2026\/04\/23\/google-may-expand-unsupported-robots-txt-rules-list\/","title":{"rendered":"Google May Expand Unsupported Robots.txt Rules List"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n

Google could increase the record of unsupported robots.txt guidelines in its documentation based mostly on evaluation of real-world robots.txt information collected by HTTP Archive.<\/p>\n

Gary Illyes and Martin Splitt described the undertaking on the newest episode of Search Off the Record<\/a>. The work began after a neighborhood member submitted a pull request to Google\u2019s robots.txt repository<\/a> proposing two new tags be added to the unsupported record.<\/p>\n

Illyes defined why the crew broadened the scope past the 2 tags within the PR:<\/p>\n

\n

\u201cWe tried to not do issues arbitrarily, however reasonably acquire information.\u201d<\/p>\n<\/blockquote>\n

Somewhat than add solely the 2 tags proposed, the crew determined to have a look at the highest 10 or 15 most-used unsupported guidelines. Illyes mentioned the aim was \u201can honest place to begin, an honest baseline\u201d for documenting the commonest unsupported tags within the wild.<\/p>\n

How The Analysis Labored<\/h2>\n

The crew used HTTP Archive<\/a> to check what guidelines web sites use of their robots.txt information. HTTP Archive runs month-to-month crawls throughout tens of millions of URLs utilizing WebPageTest and shops the ends in Google BigQuery.<\/p>\n

The primary try hit a wall. The crew \u201crapidly found out that nobody is definitely requesting robots.txt information\u201d in the course of the default crawl, which means the HTTP Archive datasets don\u2019t usually embody robots.txt content material.<\/p>\n

After consulting with Barry Pollard and the HTTP Archive neighborhood, the crew wrote a customized JavaScript parser that extracts robots.txt guidelines line by line. The custom metric<\/a> was merged earlier than the February crawl, and the ensuing information is now out there within the custom_metrics dataset in BigQuery.<\/p>\n

What The Knowledge Reveals<\/h2>\n

The parser extracted each line that matched a field-colon-value sample. Illyes described the ensuing distribution:<\/p>\n

\n

\u201cAfter enable and disallow and person agent, the drop is extraordinarily drastic.\u201d<\/p>\n<\/blockquote>\n

Past these three fields, rule utilization falls into an extended tail of much less widespread directives, plus junk information from damaged information that return HTML as an alternative of plain textual content.<\/p>\n

Google at the moment supports four fields<\/a> in robots.txt. These fields are user-agent, enable, disallow, and sitemap. The documentation says different fields \u201caren\u2019t supported\u201d with out itemizing which unsupported fields are commonest within the wild.<\/p>\n

Google has clarified<\/a> that unsupported fields are ignored. The present undertaking extends that work by figuring out particular guidelines Google plans to doc.<\/p>\n

The highest 10 to fifteen most-used guidelines past the 4 supported fields are anticipated to be added to Google\u2019s unsupported guidelines record. Illyes didn’t identify particular guidelines that may be included.<\/p>\n

Typo Tolerance Might Broaden<\/h2>\n

Illyes mentioned the evaluation additionally surfaced widespread misspellings of the disallow rule:<\/p>\n

\n

\u201cI\u2019m most likely going to increase the typos that we settle for.\u201d<\/p>\n<\/blockquote>\n

His phrasing implies the parser already accepts some misspellings. Illyes didn\u2019t decide to a timeline or identify particular typos.<\/p>\n

Why This Issues<\/h2>\n

Search Console already surfaces some unrecognized robots.txt tags. If Google paperwork extra unsupported directives, that would make its public documentation extra intently mirror the unrecognized tags individuals already see surfaced in Search Console.<\/p>\n

Trying Forward<\/h2>\n

The deliberate replace would have an effect on Google\u2019s public documentation and the way disallow typos are dealt with. Anybody sustaining a robots.txt file with guidelines past user-agent, enable, disallow, and sitemap ought to audit for directives which have by no means labored for Google.<\/p>\n

The HTTP Archive information is publicly queryable on BigQuery<\/a> for anybody who desires to look at the distribution straight.<\/p>\n\n

Featured Picture: Screenshot from: YouTube.com\/GoogleSearchCentral, April 2026.\u00a0<\/em><\/p>\n<\/div>\n