This publish was sponsored by JetOctopus. The opinions expressed on this article are the sponsor’s personal.
How do I optimize my web site for ChatGPT and Perplexity, not simply Google?
How do I do know if AI bots are literally crawling my web site?
How ought to my technical web optimization technique change for AI Search?
A good portion of your web site’s search impressions in 2026 are generated by machines researching on behalf of humans.
These machines don’t care about your key phrase rankings. They care whether or not your:
- HTML masses cleanly in below 200 milliseconds
- Product element web page is reachable in fewer than 4 clicks
- Content material solutions a selected, nine-word query that has by no means appeared in any key phrase analysis software in your profession.
This isn’t hypothesis. It’s what our server log knowledge throughout a whole lot of enterprise web sites is displaying us, constantly, since mid-2025.
What’s Really Occurring On Your Web site
My colleague, Stan, flagged a sample in a Slack message: question lengths have been rising at charges that didn’t correlate with human conduct.
A 161% progress price in 10-word queries year-over-year isn’t pushed by customers who immediately obtained extra verbose. It’s pushed by AI brokers decomposing a single consumer immediate into dozens of parallel sub-queries, a course of researchers now name “fan-out.”
Question Size Progress in 2025
The gradient is the inform. Human search conduct doesn’t scale this cleanly by phrase rely. Machines do. By October 2025, 7-plus-word queries reached practically 1% of complete question quantity, roughly triple their historic share.
Extra revealing than the amount is the CTR. Whereas impression counts for 10-word queries spiked 161%, click-through price collapsed to 2.26%, down from 8–11% in 2023.
The AI reads your web page, extracts the reply, synthesizes it for the consumer. Your web site by no means will get the go to.
We name these “phantom impressions.” They’re actual alerts that your content material is being evaluated inside AI reasoning chains. Should you’re filtering them out of your reporting as a result of they don’t drive visitors, you might be flying blind.
The Three Bots Visiting Your Web site & Their Influence On SERP Visibility
Not all AI crawlers are equal, and treating them as a single class is the primary mistake most technical SEOs make.
Training bots crawl broadly and ignore click on depth. A coaching go to means the AI is aware of your content material exists, not that customers will ever see it.
AI search bots drop off rapidly past two or three clicks from the homepage and usually go to every web page solely as soon as a month.
AI consumer bots are initiated when an actual particular person asks a query in ChatGPT, Perplexity, or Claude, and the AI researches the reply on their behalf. These are the one visits that translate to precise AI visibility.
| Bot Kind | What Triggers It | Crawl Depth | Influence on AI Visibility |
| Coaching bots | Mannequin schooling cycles | Deep — ignores click on distance | None straight. Consciousness solely. |
| AI search bots | New URL discovery & contemporary content material | Shallow — ~1 go to/month past 2–3 clicks | Essential gatekeeper. If it misses a web page, consumer bots gained’t discover it both. |
| AI user bots | Actual consumer question in ChatGPT / Claude / Perplexity | Selective — pushed by pace and construction | Excessive. Closest proxy to an AI impression. |
Your web site can obtain heavy crawling from coaching and search bots and nonetheless be utterly absent from AI-generated solutions. Should you’re not segmenting AI bot visitors by sort in your log analysis, you haven’t any concept which third of the iceberg you’re measuring.
Which web optimization Alerts Do LLMs Respect?
Robots.txt is your main lever.
Most main AI platforms (ChatGPT, Claude, Gemini) observe robots.txt directives. Perplexity is a partial exception: PerplexityBot respects robots.txt, however Perplexity-Person, the user-triggered bot, doesn’t. Cloudflare confirmed this in an investigation. Most websites haven’t audited their robots.txt with AI entry in thoughts. Do it.
Sitemaps are broadly supported.
ChatGPT, Claude, and PerplexityBot all use XML sitemaps for URL discovery. Hold them correct.
Alerts Greatest Saved For web optimization & Rating Efforts
These alerts under don’t seem to impression AI visibility, however are nonetheless key for rating for queries that also set off conventional SERPs.
Canonical tags and noindex directives do nothing for AI bots.
AI crawlers don’t construct a search index, in order that they haven’t any use for these meta-signals. Content material hidden from Google utilizing noindex is totally seen to ChatGPT’s crawler.
LLM.txt does nothing.
Our log knowledge exhibits main AI bots don’t learn this file. Don’t make investments time right here.
JavaScript rendering is a crucial blind spot.
Most AI crawlers (ChatGPT, Claude, Perplexity) don’t render JavaScript. In case your product pages load key content material client-side, these brokers learn an empty shell. Server-side rendering is the one structure that works universally. The exception is Google Gemini, which makes use of the identical Internet Rendering Service as Googlebot.
How To Make Positive ChatGPT, Perplexity & LLMs Can Attain Your Content material
AI search bots go to deep pages roughly as soon as a month and drop off sharply past three clicks from the homepage. The pages with probably the most particular, answerable info are sometimes the toughest for brokers to succeed in.
The repair: Elevate your Most worthy deep pages by means of inner linking, guaranteeing they’re reachable inside 4 clicks.
Pages crawled by coaching bots however by no means reached by consumer bots are your highest-priority targets. Pages AI consumer bots go to often are telling you what to scale: extra content material overlaying the identical matter cluster and depth.
Optimize Content material For Longer, Fan-Out Queries
95% of the queries driving AI citations have zero month-to-month search quantity. They’re artificial sub-queries generated by AI fashions. However they present up in GSC: impressions, no clicks, question lengths you’d by no means goal voluntarily.
How To Discover Fan Out Question Alternatives
To floor fan out queries which can be value chasing, join your GSC API to JetOctopus (to bypass the 1,000-row UI limit) and filter for: question size higher than 7 phrases, impressions below 50, clicks at 0, over the past 3 months. That’s your Fan-Out Opportunity Matrix, the precise questions AI brokers are asking about your content material.
Immediate Varieties That Fan Out Most
In case your content material isn’t structured to reply record and comparability queries, with express rankings, execs/cons, and side-by-side specs, you’re leaving the very best fan-out floor space unoptimized.
“Product evaluate” intent queries surged from 239 in June 2025 to over 40,000 by September 2025. That 16,000% improve was AI brokers systematically harvesting structured opinion knowledge. In case your product pages lack this depth, you’re invisible to that harvest.
The Technical Audit: The place to Begin
Step 1: Establish AI Person Bot Visitors In Logs
Pull uncooked server logs (Apache/Nginx) and export all strains containing these consumer brokers: OAI-SearchBot and ChatGPT-Person, PerplexityBot and Perplexity-Person, Claude-SearchBot and Claude-Person. Then manually group hits by user-agent patterns and endpoints in a spreadsheet. To tell apart coaching bots from consumer bots, you’ll want to keep up your personal classification record — one which adjustments usually and isn’t standardized.
In JetOctopus Log Analyzer, this segmentation is in-built: filter by bot sort (coaching, search, and consumer) in just a few clicks and instantly see which pages AI consumer bots go to (your AI-visible content material, able to scale) versus pages coaching bots hit however consumer bots by no means attain (your highest-priority repair targets).
Step 2: Audit Technical Accessibility Of Deep Pages
Choose a pattern of deep URLs and test HTML payload dimension, verify key content material isn’t injected by way of JavaScript by viewing uncooked HTML, simulate crawl depth by counting clicks from the homepage, and check load time in Chrome DevTools or Lighthouse. Additionally test whether or not vital content material sits behind accordions or “View Extra” components — these require JavaScript execution that AI bots skip solely. For big websites with 1000’s of deep pages, this sampling method misses rather a lot. AI brokers don’t click on. If info solely seems after consumer interplay, it doesn’t exist for these crawlers.
Step 3: Clear Up Your Robots.txt
Open your robots.txt and evaluate all Disallow and Enable directives for each user-agent line by line. AI bots observe Disallow guidelines, so be sure you’re not unintentionally blocking vital URLs. Manually check key URLs to substantiate they aren’t blocked. A 30-minute audit right here can stop you from blocking crawlers you need in, or exposing content material you’d slightly hold out.
Step 4: Map Your Phantom Impressions
Export knowledge from GSC Efficiency studies filtered by impressions with zero clicks. Due to the 1,000-row UI restrict, you’ll want to make use of the GSC API or export in chunks by date and question, then merge datasets in spreadsheets or BigQuery. Additionally consider question frequency: lengthy queries showing each day are doubtless not fan-outs.
Join your GSC API to JetOctopus to bypass the row restrict and construct your Fan-Out Alternative Matrix mechanically — the precise questions AI brokers are asking about your content material, able to act on.
Step 5: Monitor The Adjustments
Arrange a recurring export course of — pull GSC knowledge month-to-month and evaluate impressions over time, re-run log evaluation scripts and diff bot exercise, monitor Core Internet Vitals individually in PageSpeed Insights or CrUX. You’ll find yourself stitching collectively a number of knowledge sources with no unified alerting, making it laborious to catch regressions early.
JetOctopus Alerts covers precisely this: unified notifications for adjustments in AI bot exercise alongside Googlebot conduct, Core Internet Vitals, on-page web optimization points, and SERP efficiency drops, so that you catch regressions earlier than they compound.
The New KPI: Technical Accessibility
web optimization in 2026 is restructuring round one constraint: can an AI agent crawl, attain, and extract a truth out of your 50,000th product web page in below 200 milliseconds?
If the reply is not any, your rankings, backlinks, and content material high quality develop into irrelevant for a rising share of search interactions. The machines are looking. The query is how rapidly you may see what’s truly taking place.
Begin together with your logs. All the things else follows from there.
| Need to see precisely how AI bots are interacting together with your web site: which pages they attain, which they skip, and the place your fan-out alternatives are hiding? Ebook a live walkthrough of the JetOctopus platform. We’ll pull your precise log knowledge and present you what your GSC studies aren’t telling you. |
Picture Credit
Featured Picture: Picture by JetOctopus. Used with permission.
Source link
