AI bots crawl retail sites 198x more than Google, new report warns

A brand new industry report revealed this week by Retail Economics, Amazon Internet Providers, Botify, and DataDome has put a exact quantity on how dramatically synthetic intelligence has disrupted the underlying mechanics of retail discovery – and it’s a quantity that ought to focus minds throughout search, e-commerce, and promoting alike. For each single go to OpenAI’s methods ship to a retail web site, those self same methods carry out 198 crawls. Google, by comparability, generates one go to for each six crawls. The disparity, drawn from evaluation of roughly 200 retail and e-commerce web sites, illustrates how AI platforms work together with the net in a essentially totally different manner from the major search engines which have formed digital advertising and marketing for the previous twenty years.

The report, titled “The Way forward for Search and Discovery: A strategic playbook to know agentic commerce,” relies on a shopper survey of 6,000 nationally consultant respondents throughout the UK, US, and France, carried out in November 2025. Its conclusions vary from quantitative measurements of bot visitors development to qualitative assessments of shopper belief, and it arrives at a second when agentic commerce infrastructure is being built at pace across every major platform.

Bot visitors has multiplied – and skewed analytics

The headline infrastructure discovering is stark. In keeping with Botify’s evaluation, AI-driven bot visitors throughout the roughly 200 retail and e-commerce web sites examined elevated 5.4 instances throughout 2025, with the index shifting from a baseline of 100 within the first quarter to roughly 640 by the fourth quarter. The expansion was not linear. A very sharp acceleration occurred within the weeks previous September 2025, when crawl depth rose sharply as AI methods refreshed and ingested product knowledge. Shortly after, OpenAI expanded its commerce-related capabilities, together with agent-led buying and in-chat buying options. Visits from OpenAI to retail web sites then elevated 200% month on month in September 2025 following that rollout – a direct illustration of the connection between platform-level functionality updates and referral visitors patterns. PPC Land reported on OpenAI’s instant checkout launch on September 29, 2025, protecting how the Stripe-backed Agentic Commerce Protocol enabled direct purchases inside ChatGPT conversations.

The category-level image is much more granular. In keeping with the report, meals and grocery skilled a 29-times improve in AI-driven bot visitors over the course of 2025, pushed by the excessive volatility of costs and inventory ranges that make the class precious for AI methods to watch repeatedly. House and DIY noticed an 11-times improve. Electronics and home equipment additionally crawled considerably. The divergence displays a structural perception: AI methods deal with retail classes in a different way primarily based on how often knowledge modifications, not purely on industrial significance or retailer efficiency.

The size and velocity of this automated visitors introduces a measurement downside that retailers have solely begun to grapple with. In keeping with the report, AI bot methods generate high-volume, concurrent requests that aren’t all the time distinguishable from human shopping in conventional analytics. The results are concrete. When Google eliminated a technical shortcut – the &num=100 parameter – that many monitoring instruments relied on in early September 2025, Botify’s enterprise retail purchasers reported search impressions fell by roughly 67%, whereas clicks stayed largely flat and common place appeared to enhance. Click on-through fee development then elevated by roughly 150%, not as a result of efficiency had genuinely modified, however as a result of the info was now not contaminated by artificial bot impressions. A lot of the obvious development in impressions had been pushed by AI bots able to making 100 or extra simultaneous requests, not by actual customers.

Almost 80% of internet sites uncovered to agent spoofing

Maybe probably the most operationally pressing discovering within the report considerations safety. DataDome, the bot and agent belief administration firm that co-produced the analysis, analysed 698,214 stay web sites utilizing a spoofed “ChatGPT AI assistant” user-agent. The outcome: 79.7% didn’t block or problem the impersonation try. Of these, 79.2% returned a “200 OK” response code, which means the spoofed agent was admitted with out problem. Solely 17.2% returned a “403 Forbidden” response.

This isn’t an summary vulnerability. In keeping with the report, spoofable consumer brokers and incomplete IP lists make it tough for retailers to tell apart official AI brokers from stealth or human-driven automation utilizing shared infrastructure. The sensible impact is that malicious actors can clone weakly declared AI brokers to use pricing, stock, or checkout flows. The report notes that DataDome’s risk analysis crew, Galileo, lately recognized that 80% of AI brokers don’t declare themselves correctly when visiting web sites. That determine underpins a broader argument that retailers face “skewed efficiency metrics that undermine industrial choices and expose them to fraud.”

The danger is uneven. Blocking all AI visitors to guard in opposition to spoofing carries a distinct price: if manufacturers don’t enable AI bots to seek out and use content material on their web sites, in keeping with the report, these methods could discover knowledge elsewhere – from third-party overview websites, boards, or rivals. PPC Land has tracked how Amazon chose the restrictive path, blocking AI bots from OpenAI, Anthropic, Meta, Google, and Huawei in August 2025, a method that runs in parallel with Amazon’s improvement of its personal proprietary AI buying instruments.

The 1-in-198 ratio and what it means for discovery

The visit-to-crawl ratio is price dwelling on. It indicators that for OpenAI’s methods, the first function of participating with retail web sites just isn’t delivering guests however slightly ingesting, validating, and evaluating info inside their very own interfaces. Discovery and analysis more and more occur inside AI interfaces earlier than a shopper ever reaches a retailer’s web site. This challenges the foundational assumption of search engine optimization: that being crawled interprets, over time, into being visited.

The report frames this as a shift in the place affect operates. In keeping with Botify’s knowledge, Google drives one go to per six crawls, in contrast with one per 198 for OpenAI. In sensible phrases, a product that ranks extremely in Google search nonetheless generates visitors straight. A product evaluated by an OpenAI agent could form a advice with out ever producing a referral go to. Conversion attribution, session metrics, and bounce fee develop into much less significant because of this. Brainlabs reported earlier in 2025 that AI search visitors can be worth 4.4 times more than traditional organic traffic, however that premium relies upon fully on the customer arriving at an internet site within the first place – an end result the 1-in-198 ratio suggests is way from assured.

The report introduces a taxonomy of AI-led visitors that distinguishes between coaching crawlers (reminiscent of GPTBot from OpenAI and ClaudeBot from Anthropic), stay retrieval crawlers (reminiscent of ChatGPT-user and Perplexity-user, which fetch recent content material in actual time), index-building crawlers (reminiscent of OAI-SearchBot and PerplexityBot), AI assistants and buying brokers (reminiscent of ChatGPT, Microsoft Copilot, Gemini, and Amazon Rufus), agentic browsers (reminiscent of Perplexity Comet, ChatGPT Atlas, and Gemini built-in into Chrome), and malicious or exploitative bots(unauthorised scrapers, aggressive intelligence bots, and automatic fraud visitors). Every class carries totally different implications for governance and entry coverage. PPC Land reported on OpenAI’s revised ChatGPT crawler documentation in December 2025, which created totally different compliance requirements for various crawler sorts.

JavaScript invisibility and the structured knowledge crucial

A separate technical discovering deserves consideration amongst search and e-commerce professionals. In keeping with the report, most AI bots can not learn content material rendered in JavaScript. If a model’s product knowledge – pricing, availability, specs, evaluations – sits behind JavaScript, AI methods will see solely a stripped-down model of the web page. The report illustrates this with a comparability: a shoe product web page seen by a shopper exhibits dimension, color choices, supplies, value, and promotional particulars; the identical web page seen by most AI bots exhibits solely a handful of seen textual content labels and a stripped visible shell.

The consequence is direct. If AI methods can not entry or interpret a retailer’s knowledge, that retailer could by no means seem in AI-mediated discovery. The report locations structured, authenticated, and accessible knowledge on the centre of its 5 recognized forces of disruption, alongside discovery shifts, infrastructure necessities, LLM evolution, and measurement change. Poor metadata or inconsistent taxonomies could make merchandise invisible to AI crawlers fully. PPC Land reported in December 2025 on Google’s documentation clarifications around JavaScript rendering for error pages, reinforcing the identical underlying technical vulnerability.

The report identifies Reply Engine Optimisation (AEO) as the expansion layer constructed on prime of conventional search engine optimization. Conventional key phrase rankings, natural impressions, click-through fee, area authority, and bounce fee – the usual dashboard of digital advertising and marketing efficiency – have been constructed for a world of hyperlinks and human clicks. They don’t present how AI brokers see, interpret, and act on content material. The report proposes a brand new era of efficiency metrics: agent inclusion fee (what quantity of merchandise or pages are recognised and surfaced by AI brokers), discovery visibility (presence fee throughout multimodal environments), engagement confidence index (how usually customers act on AI-surfaced outcomes), structured-data protection, belief sign energy, visibility-to-sale ratio, and discovery ROI index. These are rising requirements, not but extensively deployed, however the report argues they’re obligatory to know industrial affect in an AI-mediated surroundings. An SEO expert released a related AI search content optimisation checklist in June 2025 that addressed related necessities round server-side rendering and structured knowledge protection.

Shopper adoption: 73%, however belief lags

The buyer survey element of the report attracts from 6,000 respondents throughout the UK, US, and France surveyed in November 2025, with 2,000 per nation. In keeping with Retail Economics, 73% of customers throughout the three markets have consciously used AI in some type over the previous twelve months. Of these, 38% have used AI assistants particularly for buying duties together with product concepts, recommendations, or comparisons. An extra 34% have used AI options on retailer web sites or apps. Twenty-one % have used AI instruments to make choices or assist purchases.

The US data the best adoption fee at 73%, with France at 69% and the UK at 68% – nearer to one another than is likely to be anticipated given variations in digital tradition. Amongst 18-to-24-year-olds, roughly one in 4 use AI assistants commonly and one in 5 use them day-to-day. Amongst these aged 55 and older, fewer than one in ten report day-to-day use. The hole widens additional when analyzing AI use relative to different discovery channels: amongst 18-to-24-year-olds, AI assistants and social discovery channels exert affect that matches or exceeds conventional engines like google within the discovery part.

Belief, nonetheless, tells a distinct story. Thirty-two % of customers throughout the surveyed areas say they don’t belief AI-enabled search and discovery. Whereas 38% really feel comfy with suggestions from instruments like ChatGPT, Microsoft Copilot, and Gemini, far fewer are prepared to let these methods act on their behalf. Almost half – 49% – say discovery is one thing they need to do themselves, not one thing to outsource. The report describes this as a “key pressure within the shift in direction of agentic commerce: individuals worth the advantages afforded by AI, however do not but really feel totally assured to delegate choices.”

The belief hole is structured by age and revenue. Larger-income customers exhibit better confidence in AI methods, probably reflecting better familiarity from work settings. Center-aged, high-affluence customers emerge as probably the most AI-trusting section. Least prosperous customers present the bottom belief, the place considerations about threat, accuracy, and management are most acute.

Which classes and missions face the earliest publicity

The report maps retail classes by shopper belief in AI-led discovery and willingness to make use of AI, weighted by typical spend. Electronics and home equipment constantly lead throughout all three markets. Purchases on this class contain technical specs, fast product cycles, and significant value variations – precisely the situations the place AI help compared and shortlisting is most valued. Journey and leisure sits shut behind. Clothes and footwear exhibits rising publicity, with massive on-line ranges and frequent shopping creating fertile floor for AI-led personalisation.

Classes sitting decrease on each axes embrace jewelry, magnificence, and homewares – purchases that contain emotional, tactile, and private judgements the place customers nonetheless search human reassurance. Meals and grocery exhibits sturdy regional variation: the US exhibits greater openness to AI help in grocery discovery, whereas France displays a stronger meals tradition centred on freshness and bodily inspection.

Buying missions comply with a parallel gradient. In keeping with the survey, customers present the best willingness to delegate to AI for “thought-about or technical purchases” and for “shopping for items for others” – each missions involving uncertainty, excessive info load, and profit from structured comparability. Routine replenishment sits on the backside of the willingness scale throughout all three markets. The sample is constant: AI help is welcomed the place choices really feel cognitively demanding, and resisted the place recurring or emotional judgement dominates.

Amazon Rufus supplies a industrial datapoint that anchors these projections. In keeping with the report, greater than 250 million prospects used Rufus throughout 2025, with interactions up 210% 12 months on 12 months. Clients who use Rufus whereas buying are over 60% extra more likely to make a purchase order throughout that session. Amazon’s full-year financial results subsequently confirmed that Rufus generated nearly $12 billion in incremental annualized sales during 2025, with greater than 300 million prospects utilizing the device all year long.

4 shopper personas and three readiness workstreams

The report identifies 4 distinct shopper personas in relation to AI-assisted discovery. AI-first optimisers (10% of the full, skewing youthful at a mean age of 38) use AI assistants as their main discovery device and present 47% full belief in AI for analysis and comparability. Assisted explorers (55%, common age 42) welcome AI as a sensible co-pilot for shortlisting and comparability however need to stay within the approval loop. Guarded adopters (16%, common age 53) use AI in managed, low-risk methods however scrutinise outcomes and hesitate earlier than delegating significant choices. Human loyalists(19%, common age 62) hardly ever use AI for buying and require concrete proof of profit earlier than adopting extra meaningfully.

The strategic part of the report organises its suggestions into three readiness workstreams. The primary considerations visitors coverage for AI bots and brokers – establishing which methods must be allowed, blocked, restricted, or monetised, with steady belief evaluation and dynamic behaviour-based safety. The second considerations knowledge readiness and product info administration – standardising product attributes, metadata, and taxonomy to create a single machine-readable supply of product reality, and testing how AI crawlers really extract and interpret that knowledge. The third considerations on-site AI experiences – constructing conversational, voice, and embedded-agent consumer experiences that full the discovery-to-purchase loop with out shedding the client to a competitor’s AI interface.

Cloudflare’s launch of pay-per-crawl in July 2025 and its subsequent Markdown for Agents service in early 2026characterize infrastructure-level responses to precisely these workstreams, creating mechanisms for retailers to regulate and monetise AI entry to their content material whereas decreasing the token price of that entry by roughly 80%.

The report concludes that early-mover benefits are rising, however manufacturers that delay motion threat changing into tougher to seek out, tougher to belief, and simpler to exchange. The age of agentic search and discovery, it argues, will arrive progressively – however the transition is already underway, and hastening.

Timeline

• 2010-2014 – Key phrase search dominates retail discovery; customers kind actual phrases into engines like google with outcomes ranked on key phrases and web page relevance.
• 2014-2017 – Behavioural and personalised search takes maintain; retailers use cookies and shopping historical past to introduce advice engines.
• 2017-2019 – Cellular, social, and voice discovery expands search past textual content by means of smartphones, Alexa, Siri, Fb, YouTube, Instagram, and TikTok.
• 2019-2021 – Visible and contextual discovery arrives with Amazon Lens, Pinterest Lens, and Google Lens enabling image-based buying.
• 2022-2024 – Generative discovery begins; ChatGPT and Google AI Overviews rework search into dialogue, summarising and evaluating merchandise.
• August 7, 2023 – OpenAI publicizes GPTBot; main web sites start implementing blocks inside two weeks. Coverage on PPC Land
• 2024 – Bot visitors exceeds human web site guests for the primary time, in keeping with Imperva knowledge cited in Brainlabs analysis. Coverage on PPC Land
• Could 2024 – Google launches GoogleOther-Picture and GoogleOther-Video crawlers for analysis and improvement knowledge gathering. Coverage on PPC Land
• Q1 2025 – AI-driven bot visitors baseline established at index 100 throughout roughly 200 retail and e-commerce web sites analysed by Botify.
• April 18, 2025 – Microsoft launches Copilot Service provider Program for retail integration. Coverage on PPC Land
• April 28, 2025 – OpenAI introduces buying options to ChatGPT, reporting over 1 billion weekly searches. Coverage on PPC Land
• July 1, 2025 – Cloudflare launches pay-per-crawl service in non-public beta. Coverage on PPC Land
• July 16, 2025 – search engine optimization knowledgeable warns Google’s AI might remove web site clicks amid deteriorating crawl-to-visit ratios. Coverage on PPC Land
• August 21, 2025 – Amazon blocks AI crawlers from OpenAI, Anthropic, Meta, Google, and Huawei. Coverage on PPC Land
• Early September 2025 – Surge in AI crawl depth at retail web sites precedes OpenAI’s commerce functionality enlargement; Google removes &num=100 monitoring parameter, inflicting obvious 67% drop in search impressions.
• September 29, 2025 – OpenAI launches Immediate Checkout for ChatGPT with Stripe partnership and Agentic Commerce Protocol. Coverage on PPC Land
• September 2025 – OpenAI commerce capabilities broaden; visits from OpenAI to retail web sites improve 200% month on month, per Botify evaluation.
• October 6, 2025 – Unbiased analyst questions industrial viability of agentic commerce regardless of ChatGPT checkout launch. Coverage on PPC Land
• November 2025 – Retail Economics shopper survey of 6,000 respondents carried out throughout UK, US, and France.
• November 13, 2025 – Google launches agentic checkout and AI buying instruments for the vacation season. Coverage on PPC Land
• November 17, 2025 – Google Search Console provides annotations; Google’s AI Mode features agentic options together with desk reservations. Coverage on PPC Land
• November 25, 2025 – UK analysis exhibits 85% of customers planning AI-assisted vacation buying would belief brokers to position orders and pay. Coverage on PPC Land
• This fall 2025 – AI-driven bot visitors reaches 5.4x the Q1 2025 baseline throughout retail web sites analysed by Botify.
• December 9, 2025 – OpenAI revises ChatGPT crawler documentation with vital coverage modifications. Coverage on PPC Land
• December 18, 2025 – Google clarifies JavaScript rendering behaviour for error pages. Coverage on PPC Land
• January 8, 2026 – Microsoft launches Copilot Checkout with PayPal, Shopify, and Stripe integration. Coverage on PPC Land
• Early 2026 – Cloudflare launches Markdown for Brokers, decreasing AI token prices by 80%. Coverage on PPC Land
• February 2026 – Amazon confirms Rufus generated practically $12 billion in incremental annualised gross sales throughout 2025, with over 300 million customers. Coverage on PPC Land
• March 5, 2026 – Greenough Company pitches the Retail Economics/AWS/Botify/DataDome report back to PPC Land.
• March 7, 2026 – “The Way forward for Search and Discovery: A strategic playbook to know agentic commerce” revealed by Retail Economics, AWS, Botify, and DataDome.

Abstract

Who: Retail Economics, Amazon Internet Providers, Botify, and DataDome revealed the report. The buyer analysis covers 6,000 nationally consultant customers within the UK, US, and France. Key knowledge contributors embrace Botify’s evaluation of roughly 200 retail and e-commerce web sites, and DataDome’s safety take a look at of 698,214 stay web sites. AJ Ghergich, World VP of AI at Botify, is obtainable for touch upon the findings.

What: A 35-page strategic report measuring the dimensions and industrial implications of AI-driven crawling and agentic discovery in retail. Core quantitative findings embrace: AI bot visitors grew 5.4 instances throughout 2025; OpenAI generates 1 go to per 198 crawls in comparison with Google’s 1 go to per 6 crawls; 79.7% of internet sites are unprotected in opposition to AI agent spoofing; 73% of customers have used AI in some type; and 38% have used AI particularly for buying duties. The report introduces a brand new taxonomy of AI visitors sorts and a set of next-generation efficiency metrics for the agentic period.

When: The buyer survey was carried out in November 2025. The bot visitors evaluation covers the complete calendar 12 months 2025. The report was revealed at the moment, March 7, 2026.

The place: The report covers retail and e-commerce markets throughout the UK, US, and France for shopper knowledge. The bot visitors evaluation attracts from roughly 200 retail and e-commerce web sites globally. The safety evaluation of agent spoofing covers 698,214 stay web sites internationally.

Why: AI methods now perform as gatekeepers between manufacturers and customers, shaping consideration units earlier than customers ever go to a retailer’s web site. The mixture of quickly escalating bot visitors, widespread vulnerability to agent spoofing, and the invisibility of JavaScript-rendered content material to most AI crawlers creates materials industrial threat for retailers who haven’t but tailored their knowledge infrastructure, visitors governance, and measurement frameworks to the agentic period. The report argues that early-mover benefits are already rising and that delay will increase the chance of being excluded from AI-mediated discovery fully.