{"id":105055,"date":"2025-12-05T13:09:37","date_gmt":"2025-12-05T13:09:37","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2025\/12\/05\/complete-crawler-list-for-ai-user-agents-dec-2025\/"},"modified":"2025-12-05T13:11:37","modified_gmt":"2025-12-05T13:11:37","slug":"complete-crawler-list-for-ai-user-agents-dec-2025","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2025\/12\/05\/complete-crawler-list-for-ai-user-agents-dec-2025\/","title":{"rendered":"Complete Crawler List For AI User-Agents [Dec 2025]"},"content":{"rendered":"

<\/a>\n<\/p>\n

\n

AI visibility performs a vital function for SEOs, and this begins with controlling AI crawlers. If AI crawlers can\u2019t entry your pages, you\u2019re invisible to AI discovery engines.<\/p>\n

On the flip facet, unmonitored AI crawlers<\/a> can overwhelm servers with extreme requests, inflicting crashes and sudden internet hosting payments.<\/p>\n

Consumer-agent strings are important for controlling which AI crawlers can entry your web site, however official documentation is commonly outdated, incomplete, or lacking completely. So, we curated a verified checklist of AI crawlers from our precise server logs as a helpful reference.<\/p>\n

Each user-agent is validated in opposition to official IP lists when out there, making certain accuracy. We’ll keep and replace this checklist to catch new crawlers and modifications to current ones.<\/p>\n

The Full Verified AI Crawler Listing (December 2025)<\/h2>\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Title<\/th>\nFunction<\/th>\nCrawl Fee of SEJ (pages\/hour)<\/th>\nVerified IP Listing<\/th>\nRobots.txt disallow<\/th>\nFull Consumer Agent<\/th>\n<\/tr>\n<\/thead>\n
GPTBot<\/a><\/td>\nAI coaching knowledge assortment for GPT fashions (ChatGPT, GPT-4o)<\/td>\n100<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: GPTBot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; GPTBot\/1.3; +https:\/\/openai.com\/gptbot)<\/td>\n<\/tr>\n
ChatGPT-User<\/a><\/td>\nAI agent for real-time net searching when customers work together with ChatGPT<\/td>\n2400<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: ChatGPT-Consumer
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko); appropriate; ChatGPT-Consumer\/1.0; +https:\/\/openai.com\/bot<\/td>\n<\/tr>\n
OAI-SearchBot<\/a><\/td>\nAI search indexing for ChatGPT search options (not for coaching)<\/td>\n150<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: OAI-SearchBot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/131.0.0.0 Safari\/537.36; appropriate; OAI-SearchBot\/1.3; +https:\/\/openai.com\/searchbot<\/td>\n<\/tr>\n
ClaudeBot<\/a><\/td>\nAI coaching knowledge assortment for Claude fashions<\/td>\n500<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: ClaudeBot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; ClaudeBot\/1.0; +claudebot@anthropic.com)<\/td>\n<\/tr>\n
Claude-User<\/a><\/td>\nAI agent for real-time net entry when Claude customers browse<\/td>\n\n<\/td>\nNot out there<\/td>\nConsumer-agent: Claude-Consumer
Disallow: \/sample-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; Claude-Consumer\/1.0; +Claude-Consumer@anthropic.com)<\/td>\n<\/tr>\n
Claude-SearchBot<\/a><\/td>\nAI search indexing for Claude search capabilities<\/td>\n\n<\/td>\nNot out there<\/td>\nConsumer-agent: Claude-SearchBot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; Claude-SearchBot\/1.0; +https:\/\/www.anthropic.com)<\/td>\n<\/tr>\n
Google-CloudVertexBot<\/a><\/td>\nAI agent for Vertex AI Agent Builder (website homeowners\u2019 request solely)<\/td>\n\n<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: Google-CloudVertexBot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Construct\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/141.0.7390.122 Cell Safari\/537.36 (appropriate; Google-CloudVertexBot; +https:\/\/cloud.google.com\/enterprise-search)<\/td>\n<\/tr>\n
Google-Extended<\/a><\/td>\nToken controlling AI coaching utilization of Googlebot-crawled content material.<\/td>\n\n\nConsumer-agent: Google-Prolonged
Permit: \/
Disallow: \/private-folder<\/td>\n		\n<\/tr>\n
Gemini-Deep-Research<\/a><\/td>\nAI analysis agent for Google Gemini\u2019s Deep Analysis characteristic<\/td>\n\n<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: Gemini-Deep-Analysis
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; Gemini-Deep-Analysis; +https:\/\/gemini.google\/overview\/deep-research\/) Chrome\/135.0.0.0 Safari\/537.36<\/td>\n<\/tr>\n
Google<\/a>\u00a0<\/span><\/td>\nGemini\u2019s chat when a person asks to open a webpage<\/td>\n\n<\/td>\n\n\nGoogle<\/span><\/td>\n<\/tr>\n
Bingbot<\/a><\/td>\nPowers Bing Search and Bing Chat (Copilot) AI solutions<\/td>\n1300<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: BingBot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; bingbot\/2.0; +http:\/\/www.bing.com\/bingbot.htm) Chrome\/116.0.1938.76 Safari\/537.36<\/td>\n<\/tr>\n
Applebot-Extended<\/a><\/td>\nDoesn\u2019t crawl<\/a> however controls how Apple makes use of Applebot knowledge.<\/td>\n\n<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: Applebot-Prolonged
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit\/605.1.15 (KHTML, like Gecko) Model\/17.4 Safari\/605.1.15 (Applebot\/0.1; +http:\/\/www.apple.com\/go\/applebot)<\/td>\n<\/tr>\n
PerplexityBot<\/a><\/td>\nAI search indexing for Perplexity\u2019s reply engine<\/td>\n150<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: PerplexityBot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; PerplexityBot\/1.0; +https:\/\/perplexity.ai\/perplexitybot)<\/td>\n<\/tr>\n
Perplexity-User<\/a><\/td>\nAI agent for real-time searching when Perplexity customers request info<\/td>\n\n<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: Perplexity-Consumer
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; Perplexity-Consumer\/1.0; +https:\/\/perplexity.ai\/perplexity-user)<\/td>\n<\/tr>\n
Meta-ExternalAgent<\/a><\/td>\nAI coaching knowledge assortment for Meta\u2019s LLMs (Llama, and so forth.)<\/td>\n1100<\/td>\nNot out there<\/td>\nConsumer-agent: meta-externalagent
Permit: \/
Disallow: \/private-folder<\/td>\n		meta-externalagent\/1.1 (+https:\/\/builders.fb.com\/docs\/sharing\/site owners\/crawler)<\/td>\n<\/tr>\n
Meta-WebIndexer<\/a><\/td>\nUsed to enhance Meta AI search.<\/span><\/td>\n\n<\/td>\nNot out there<\/td>\nConsumer-agent: Meta-WebIndexer
Permit: \/
Disallow: \/private-folder<\/td>\n		meta-webindexer\/1.1 (+https:\/\/builders.fb.com\/docs\/sharing\/site owners\/crawler)<\/td>\n<\/tr>\n
Bytespider<\/td>\nAI coaching knowledge for ByteDance\u2019s LLMs for merchandise like TikTok<\/td>\n\n<\/td>\nNot out there<\/td>\nConsumer-agent: Bytespider
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 (Linux; Android 5.0) AppleWebKit\/537.36 (KHTML, like Gecko) Cell Safari\/537.36 (appropriate; Bytespider; https:\/\/zhanzhang.toutiao.com\/)<\/td>\n<\/tr>\n
Amazonbot<\/a><\/td>\nAI coaching for Alexa and different Amazon AI providers<\/td>\n1050<\/td>\nNot out there<\/td>\nConsumer-agent: Amazonbot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; Amazonbot\/0.1; +https:\/\/developer.amazon.com\/help\/amazonbot) Chrome\/119.0.6045.214 Safari\/537.36<\/td>\n<\/tr>\n
DuckAssistBot<\/a><\/td>\nAI search indexing for DuckDuckGo search engine<\/td>\n20<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: DuckAssistBot
Permit: \/
Disallow: \/private-folder<\/td>\n		DuckAssistBot\/1.2; (+http:\/\/duckduckgo.com\/duckassistbot.html)<\/td>\n<\/tr>\n
MistralAI-Consumer<\/td>\nMistral\u2019s real-time quotation fetcher for \u201cLe Chat\u201d assistant<\/td>\n\n<\/td>\nNot out there<\/td>\nConsumer-agent: MistralAI-Consumer
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 AppleWebKit\/537.36 (KHTML, like Gecko; appropriate; MistralAI-Consumer\/1.0; +https:\/\/docs.mistral.ai\/robots)<\/td>\n<\/tr>\n
Webz.io<\/a><\/td>\nInformation extraction and net scraping utilized by different AI coaching firms. Previously referred to as Omgili.<\/td>\n\n<\/td>\nNot out there<\/td>\nConsumer-agent: webzio
Permit: \/
Disallow: \/private-folder<\/td>\n		webzio (+https:\/\/webz.io\/bot.html)<\/td>\n<\/tr>\n
Diffbot<\/a><\/td>\nInformation extraction and net scraping utilized by firms everywhere in the world.<\/td>\n\n<\/td>\nNot out there<\/td>\nConsumer-agent: Diffbot
Permit: \/
Disallow: \/private-folder<\/td>\n		Mozilla\/5.0 (Home windows; U; Home windows NT 5.1; en-US; rv:1.9.1.2) Gecko\/20090729 Firefox\/3.5.2 (.NET CLR 3.5.30729; Diffbot\/0.1; +http:\/\/www.diffbot.com)<\/td>\n<\/tr>\n
ICC-Crawler<\/a><\/td>\nAI and machine studying knowledge assortment<\/td>\n\n<\/td>\nNot out there<\/td>\nConsumer-agent: ICC-Crawler
Permit: \/
Disallow: \/private-folder<\/td>\n		ICC-Crawler\/3.0 (Mozilla-compatible; ; https:\/\/ucri.nict.go.jp\/en\/icccrawler.html)<\/td>\n<\/tr>\n
CCBot<\/a><\/td>\nOpen-source net archive used as coaching knowledge by a number of AI firms<\/td>\n\n<\/td>\nOfficial IP List<\/a><\/td>\nConsumer-agent: CCBot
Permit: \/
Disallow: \/private-folder<\/td>\n		CCBot\/2.0 (https:\/\/commoncrawl.org\/faq\/)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

The user-agent strings above have all been verified in opposition to Search Engine Journal server logs.<\/p>\n

Fashionable AI Agent Crawlers With Unidentifiable Consumer Agent<\/h2>\n

We\u2019ve discovered that the next didn\u2019t determine themselves:<\/p>\n