OpenAI tripled its web crawl after GPT-5

A new analysis of greater than 7 billion server log entries has discovered that OpenAI’s automated internet crawlers tripled in exercise following the launch of GPT-5 in August 2025, whilst knowledge from the identical interval suggests the variety of individuals straight utilizing ChatGPT could also be falling. The analysis, printed right now by Chris Lengthy, co-founder of search engine marketing and AI-engine optimization consultancy Nectiv, was carried out in partnership with Botify, the enterprise search engine marketing and crawl intelligence platform. It’s among the many most detailed examinations up to now of how OpenAI’s three distinct crawlers work together with the open internet.

The dataset spans November 2024 by March 2026 and attracts on Botify’s log file infrastructure, which the corporate processes on behalf of huge enterprise shoppers throughout retail and e-commerce, media and publishing, healthcare, software program, journey, and marketplaces. In accordance with the evaluation, the overall dataset exceeds 250 billion log recordsdata, of which roughly 7 billion had been remoted for this examine to focus particularly on occasions linked to OpenAI’s crawlers.

Three crawlers, three functions

Understanding the findings requires readability on how OpenAI’s crawling infrastructure is organized. In accordance with the analysis, three bots are in common operation. ChatGPT-Consumer represents a user-initiated motion – when somebody instructs ChatGPT to fetch or work together with a web page straight, that is the agent that executes the request. GPTBot is OpenAI’s general-purpose coaching crawler, designed to gather knowledge that can be utilized to enhance the foundational information of its language fashions. OAI-SearchBot is the net search crawler: it operates when ChatGPT performs a search that’s not straight triggered by a consumer, sourcing outcomes for responses that contain real-time internet retrieval.

This distinction between crawlers issues as a result of their respective developments have moved in totally completely different instructions since August 2025 – a divergence that carries completely different implications for publishers, entrepreneurs, and anybody making an attempt to grasp how ChatGPT truly reads the net.

PPC Land has coated the evolution of those crawlers intimately. In December 2025, OpenAI revised its crawler documentation, eradicating coaching language from the OAI-SearchBot description and increasing the described scope of ChatGPT-Consumer to incorporate Customized GPT requests and GPT Actions. That replace clarified that OAI-SearchBot and GPTBot might coordinate – sharing crawl outcomes with one another to keep away from duplicate requests when a web site has allowed each bots.

The GPT-5 set off

The central discovering of the Botify and Nectiv evaluation is that the launch of GPT-5 in August 2025 functioned as a structural inflection level for OpenAI’s crawling exercise. In accordance with the analysis, virtually in a single day, all three of OpenAI’s main crawlers registered speedy will increase in log occasions. When the evaluation isolates solely the automated crawlers – OAI-SearchBot and GPTBot – the distinction earlier than and after GPT-5 is described as a “huge” shift. In complete, the researchers estimate that OpenAI’s crawl of the net has tripled since August 2025.

OAI-SearchBot noticed the sharpest acceleration. Evaluating exercise earlier than and after the GPT-5 launch, the bot registered a 3.5x enhance in occasions. Inside Botify’s dataset alone, OAI-SearchBot jumped by 2.2 billion occasions. This development was not confined to a single business. In accordance with the evaluation, no vertical in Botify’s dataset registered unfavorable development from OAI-SearchBot. The sectors seeing the most important relative will increase had been healthcare at 740.94% and media and publishing at 701.91%. Marketplaces grew by 215.56%, web and software program by 204.76%, retail and e-commerce by 194.96%, and journey by 29.81%.

GPTBot – the coaching crawler – additionally expanded considerably, registering a 2.9x enhance for the reason that GPT-5 launch. That interprets to a delta of 1.8 billion occasions when evaluating the pre- and post-GPT-5 intervals inside the similar dataset.

The shift aligns with a thesis that circulated within the search engine marketing group on the time of GPT-5’s launch. In accordance with the Botify weblog publish, search engine marketing analyst Dan Petrovic had written that GPT-5 signaled a transfer towards AI methods which might be designed to be clever fairly than educated – methods that use the net as their reside information supply fairly than counting on static coaching knowledge. In accordance with the evaluation, that interpretation turned out to be correct.

Search now outpaces coaching

One of many extra exact findings within the examine issues the ratio between looking and coaching exercise. The researchers expressed this as OAI-SearchBot divided by GPTBot and in contrast the consequence earlier than and after GPT-5.

Earlier than GPT-5, the ratio stood at 0.95 – that means OpenAI was spending marginally extra time coaching than looking. After GPT-5, the ratio moved to 1.14, that means OpenAI now formally spends extra time crawling for search functions than for coaching. It’s a small however directionally important shift, suggesting the structure of how ChatGPT retrieves and generates responses has modified.

Nevertheless, this combination image masks significant variation by business. For media and publishing websites, OAI-SearchBot exercise outpaces GPTBot by a 256% relative crawl distinction – the very best of any vertical measured. Software program and websites additionally lean towards looking. Healthcare and retail and e-commerce transfer in the wrong way: healthcare exhibits a -50% relative distinction, and retail and e-commerce exhibits -33%, indicating GPTBot is comparatively extra energetic on these websites. This implies a healthcare writer and a media outlet face materially completely different optimization challenges when excited about how OpenAI’s methods are interacting with their content material.

The ChatGPT-Consumer anomaly

Whereas automated crawling has surged, the story for ChatGPT-Consumer runs in the wrong way. In accordance with the evaluation, evaluating December 1, 2025, by March 14, 2026, in opposition to the equal prior interval, ChatGPT-Consumer occasions dropped by 28%. The decline has been constant since December 2025 and is described within the analysis as “dramatic.”

Two explanations are provided. The primary is simple: fewer individuals might merely be utilizing ChatGPT. The analysis cites SimilarWeb knowledge displaying ChatGPT’s site visitors share fell from 86.7% inside the AI platform class in January 2025 to 64.5% by January 2026. Sistrix knowledge can be cited as displaying ChatGPT utilization plateauing round late 2025 after which declining.

The second rationalization is extra structurally attention-grabbing. In accordance with the Botify staff, a drop in ChatGPT-Consumer occasions might not point out fewer conversations in any respect – it could point out that OAI-SearchBot is crawling extra aggressively and constructing a extra complete index. If OpenAI has assembled a sufficiently contemporary HTML internet index, the system doesn’t must fetch pages in actual time as usually when a consumer interacts with a web page. The analogy used within the analysis is Gemini, which depends on Google’s pre-built index fairly than crawling pages on demand when grounding responses. Beneath this studying, the ChatGPT-Consumer decline may very well be an indication of improved indexing infrastructure fairly than consumer loss.

The 2 explanations usually are not mutually unique, and the researchers don’t declare to settle the query definitively. What the log file knowledge offers is a measurable proxy: when user-initiated actions lower and automatic crawling will increase concurrently, deciphering the mixed sign requires understanding which mechanism is driving every development.

This ambiguity issues for the advertising and marketing group. Publishers and types that observe ChatGPT-Consumer occasions as a measure of platform engagement could also be measuring one thing that correlates solely loosely with the precise quantity of conversations that contain their content material. PPC Land has noted similar concerns within the context of AI quotation analysis, the place blocking crawlers confirmed restricted correlation with precise quotation charges.

OpenAI versus Google: nonetheless a big hole

The evaluation situates OpenAI’s crawl growth inside the broader aggressive panorama. Even with the tripling of crawl exercise, OpenAI’s bots symbolize a fraction of what Google generates. In accordance with the analysis, within the final month coated by the dataset, Googlebot – each desktop and smartphone – registered 18.2 billion occasions, whereas OpenAI’s mixed crawlers generated 887 million. That locations OpenAI at roughly 4% of Google’s crawl quantity. Bing, for context, registered 5.49 billion occasions in the identical interval, that means OpenAI represents about 14% of Bing’s crawl.

The year-over-year comparability, nevertheless, exhibits how shortly the hole is narrowing. Within the equal 30-day window in 2025, Google crawlers registered 15 billion occasions whereas OpenAI registered 207 million, inserting ChatGPT at simply 1.38% of Google’s complete crawl. In a single yr, that share moved from 1.38% to 4%.

PPC Land has tracked this aggressive dynamic in associated protection. In January 2026, Cloudflare CEO Matthew Prince disclosed that Googlebot accesses 3.2 instances extra distinctive URLs than OpenAI’s crawlers and 4.8 instances greater than Microsoft’s, a disparity Prince framed as a structural aggressive benefit for Google in AI improvement. The Botify knowledge provides a time dimension to that image: the hole exists, however it’s measurably smaller than it was twelve months in the past.

Earlier knowledge from Cloudflare, reported by PPC Land in December 2025, confirmed that AI crawlers as an entire accounted for 4.2% of all HTML requests throughout Cloudflare’s international community in 2025, with GPTBot’s share of AI crawling site visitors rising from 4.7% in July 2024 to 11.7% in July 2025. The Botify examine, protecting a later interval and a unique dataset, extends and corroborates that trajectory.

Methodology and scope

The Botify and Nectiv examine examined log recordsdata generated by Botify’s enterprise shoppers throughout a spread of industries. The general dataset exceeds 250 billion log entries. The 7 billion-plus entries analyzed for this analysis had been remoted by filtering for occasions linked to OpenAI’s three crawlers. The time window runs from November 2024 by March 2026, with the GPT-5 inflection level recognized in August 2025 used as the first before-and-after comparability level.

Business segmentation was doable as a result of Botify tags its consumer knowledge by vertical. This allowed the researchers to match OAI-SearchBot and GPTBot exercise not simply in combination however throughout healthcare, media and publishing, marketplaces, software program and web, retail and e-commerce, and journey individually.

The log file methodology itself is price understanding. A server log information each request made to a web site, whether or not from a human consumer or an automatic bot. Every request generates an entry that features the consumer agent string – the identifier that tells the server what sort of software program made the request. OpenAI’s bots establish themselves utilizing their respective consumer agent strings: ChatGPT-Consumer, GPTBot, and OAI-SearchBot. Log file evaluation at this scale presents one thing that third-party site visitors estimates or crawl simulation instruments can’t: a direct remark of what bots truly requested, at what frequency, and throughout which sorts of websites.

For the search engine marketing and advertising and marketing group, the examine reinforces a degree that has been constructing throughout a number of analysis publications: server log evaluation is essentially the most dependable technique for understanding how AI methods work together with a given web site, since combination statistics and business benchmarks can obscure important vertical-level variation.

Implications for publishers and entrepreneurs

The findings carry sensible weight for various components of the advertising and marketing business, even when the examine stops in need of prescriptive suggestions.

For media and publishing websites, the information exhibits each the very best OAI-SearchBot development fee amongst all verticals and the clearest lean towards search over coaching within the OAI-SearchBot to GPTBot ratio. Publishers on this class are being searched aggressively by OpenAI’s methods. That has implications for the way continuously their content material must be contemporary and accessible to crawlers. PPC Land has reported that information publishers blocking AI crawlers skilled a post-blocking decline of 23.1% in log-monthly visits in response to SimilarWeb knowledge, suggesting the connection between crawl entry and referral site visitors isn’t simple however is actual.

For healthcare and retail websites, the information factors in a unique route. GPTBot is comparatively extra energetic on these verticals, indicating that OpenAI is extra prone to classify content material from these sectors as coaching materials fairly than a reside search supply. Whether or not that framing modifications with future mannequin iterations is unclear from the present knowledge.

The broader image – that OpenAI is utilizing internet search greater than it ever has, that its automated bots are at all-time-high exercise ranges, and that the hole with Google is closing even when it stays massive – offers the search engine marketing and GEO (generative engine optimization) communities extra concrete knowledge to work with than most prior analysis. The examine doesn’t change the basics of what web site homeowners can do – log evaluation, crawl accessibility, contemporary content material – but it surely places measurable numbers behind the developments these communities have been monitoring.

Timeline

• August 7, 2023 – OpenAI broadcasts GPTBot; inside two weeks, main websites together with Amazon, Quora, The New York Occasions, and CNN implement blocks. Coverage on PPC Land
• July 3, 2024 – Cloudflare studies AI bots accessed roughly 39% of the highest a million web properties utilizing its providers. Coverage on PPC Land
• August 3, 2024 – Originality.AI knowledge exhibits GPTBot blocked by 35.7% of prime 1,000 web sites, a seven-fold enhance from its 5% blocking fee at launch. Coverage on PPC Land
• August 2025 – GPT-5 launches; OpenAI’s automated crawlers – OAI-SearchBot and GPTBot – start a sustained, speedy enhance in exercise that the Botify and Nectiv examine identifies as the important thing inflection level.
• August 29, 2025 – Cloudflare releases knowledge displaying Anthropic crawled 38,000 pages per referred go to in July 2025; OpenAI maintained a ratio of 1,091 crawls per referral. GPTBot’s share of AI crawling site visitors rose from 4.7% to 11.7% year-on-year. Coverage on PPC Land
• September 2025 – OpenAI commerce capabilities increase; visits from OpenAI to retail web sites enhance 200% month on month following the rollout, per Botify evaluation. Coverage on PPC Land
• December 2025 – ChatGPT-Consumer log occasions start a sustained decline; by March 14, 2026, the drop measures -28% in comparison with the identical interval a yr earlier.
• December 9, 2025 – OpenAI revises ChatGPT crawler documentation, eradicating coaching language from OAI-SearchBot description and clarifying bot coordination conduct. Coverage on PPC Land
• December 20, 2025 – Cloudflare’s 2025 Yr in Assessment exhibits AI crawlers accounted for 4.2% of all HTML requests as international web site visitors grew 19%. Coverage on PPC Land
• January 31, 2026 – Cloudflare CEO Matthew Prince discloses that Googlebot accesses 3.2 instances extra distinctive URLs than OpenAI crawlers, framing the disparity as a aggressive benefit in AI improvement. Coverage on PPC Land
• February 25, 2026 – Anthropic updates crawler documentation, detailing ClaudeBot, Claude-Consumer, and Claude-SearchBot in a construction mirroring OpenAI’s tiered crawler mannequin. Coverage on PPC Land
• March 7, 2026 – Retail Economics, AWS, Botify, and DataDome publish a report displaying AI bot site visitors at retail websites grew 5.4 instances in 2025, with OpenAI producing 1 go to per 198 crawls in comparison with Google’s 1 go to per 6. Coverage on PPC Land
• March 8, 2026 – Google publishes a brand new internet crawling overview explaining how Googlebot discovers and renders pages. Coverage on PPC Land
• March 19, 2026 – BuzzStream examine of 4 million AI citations finds blocking AI crawlers has little measurable impact on whether or not a writer seems in AI-generated responses. Coverage on PPC Land
• April 23, 2026 – Botify and Nectiv publish evaluation of seven billion-plus OpenAI log file occasions, discovering OpenAI’s internet crawl has tripled since August 2025 whereas ChatGPT-Consumer occasions have dropped 28%.

Abstract

Who: Chris Lengthy, co-founder of Nectiv, authored the evaluation in collaboration with Botify, an enterprise search engine marketing and crawl intelligence platform. The examine attracts on log file knowledge from Botify’s enterprise consumer base spanning a number of industries.

What: An evaluation of greater than 7 billion server log occasions from OpenAI’s three crawlers – ChatGPT-Consumer, GPTBot, and OAI-SearchBot – protecting November 2024 by March 2026. Key findings embrace a tripling of OpenAI’s complete internet crawl since August 2025, a 3.5x enhance in OAI-SearchBot, a 2.9x enhance in GPTBot, and a -28% decline in ChatGPT-Consumer occasions since December 2025.

When: The examine was printed on April 23, 2026, protecting a knowledge interval from November 2024 by mid-March 2026. The GPT-5 launch in August 2025 serves because the central inflection level within the evaluation.

The place: The findings apply globally throughout all websites accessible to OpenAI’s crawlers. Business-level knowledge covers healthcare, media and publishing, marketplaces, software program and web, retail and e-commerce, and journey verticals.

Why: The analysis issues as a result of it offers log file-level proof – fairly than estimates or projections – for the way OpenAI’s infrastructure has modified in response to GPT-5. For the advertising and marketing and publishing group, it clarifies which verticals are being crawled for search versus coaching functions, quantifies the closing hole between OpenAI and Google in internet crawl quantity, and introduces a measurable sign that ChatGPT’s direct consumer engagement could also be altering whilst its automated infrastructure scales.