Agent-readiness crossed from idea to measurable infrastructure this week. On April 17, as Cloudflare Brokers Week prolonged into its sixth day, the corporate shipped isitagentready.com, a public scanner that scores any website on how prepared it is for AI agents. Paste a URL, get a rating, see which checks handed and which failed, learn AI-generated steerage on the way to enhance. For the primary time, the agent-legibility dialog moved from “is my web site prepared for brokers” as a intestine feeling to “my web site scored X out of 100 in these 5 classes, listed here are the failing indicators.”
The Agent Readiness Rating is an actual shift. It is usually a structurally deceptive instrument if you happen to cease studying after the composite quantity.
I ran the scan on this web site (nohacks.co) and scored 33 out of 100, Degree 2 “Bot-Conscious.” The robots.txt handed. The sitemap handed. The AI bot rules in robots.txt handed. Content material Indicators handed. Then the rating collapsed throughout classes the place a content-only weblog genuinely doesn’t want what the scanner checks for. Extra on that in a minute.
First, the context. Cloudflare has been transport agent-facing infrastructure all week. The Agent Readiness Rating arrived alongside Agent Reminiscence, Shared Dictionaries, Redirects for AI Coaching, an LLM compression method referred to as Unweight, and a feature-flag instrument referred to as Flagship constructed for AI-generated code. 4 days earlier, they shipped Challenge Suppose (a brand new Brokers SDK), and OpenAI matched it inside hours with their very own Brokers SDK. I wrote about that in The Agent Runtime Wars Started This Week. The readiness scanner is the logical subsequent piece: If runtimes are the brand new browser layer, web site house owners want a approach to take a look at whether or not their web site is legible to that layer. Cloudflare shipped the tester.
The query this text solutions is narrower: What does the scanner truly verify, what do you have to do along with your rating, and the place is the scoring structurally deceptive sufficient that the quantity by itself leads you astray?
What Cloudflare Shipped: Scanner, API, And An MCP Endpoint Brokers Can Name On You
The scanner is at isitagentready.com. Paste any URL, choose an internet site kind (All Checks, Content material Website, or API/Software) to scope which indicators get scanned, hit Scan. The scanner fetches the homepage and a handful of well-known paths, runs a set of checks in opposition to every, and returns a scored report with go/fail markers, standing codes, response our bodies, and AI-generated steerage on what to repair.
The scanner can be obtainable in three different methods:
- Built-in into Cloudflare Radar, so the identical checks run alongside Radar’s present URL evaluation.
- Uncovered programmatically through the Cloudflare URL Scanner API for automation.
- Accessible as a stateless MCP server at
/.well-known/mcp.json, so any MCP-compatible agent can name the scan as a instrument and motive over the consequence
That final one is price sitting with for a second. Cloudflare shipped an agent-readiness scanner that brokers themselves can name to audit web sites earlier than deciding the way to work together with them. The scanner checks whether or not your web site is prepared for brokers, and any agent can invoke it to resolve the way to work together with you earlier than arriving. The measurement and the measured are beginning to share the identical floor.
Again to the sensible query. What precisely does it verify?
16 Checks, 5 Classes: What The Scanner Really Checks
The scanner teams its checks into 5 classes. Here’s what every one seems for, grouped by what the verify truly means in apply.
Discoverability (3 Checks)
Whether or not the web site publishes the essential metadata an agent wants to search out what’s the place.
- robots.txt exists. The basic crawl-policy file. An agent that follows robots.txt wants it to exist and parse.
- sitemap.xml exists. Both declared through a Sitemap directive in robots.txt or obtainable at the usual path. An agent that desires to enumerate pages makes use of the sitemap.
- Hyperlink headers (RFC 8288). HTTP Hyperlink headers pointing to canonical, alternate, or associated sources. Helpful for brokers that parse responses fairly than HTML.
Content material (1 Test)
- Markdown for Brokers. Content material negotiation. The scanner sends
Settle for: textual content/markdownand checks whether or not the web site returns Markdown as an alternative of HTML. That is Cloudflare’s personal proposal fairly than an IETF spec, although the mechanism (HTTP content material negotiation through theSettle forheader) is commonplace. Actual agent runtimes desire Markdown as a result of it’s cheaper to tokenize and simpler to parse than HTML. Some early movers (Cloudflare itself, a handful of docs web sites) support Markdown content negotiation; most web sites don’t.
Bot Entry Management (3 Checks)
- AI bot guidelines in robots.txt (RFC 9309). Whether or not robots.txt accommodates directives for AI-specific person brokers (GPTBot, ClaudeBot, PerplexityBot, and many others.).
- Content material Indicators in robots.txt. An rising spec for expressing per-URL entry guidelines inside robots.txt. Parsed as
Person-agent: *adopted byContent material-signal:directives. Adoption is minimal proper now. - Internet Bot Auth request signing. HTTP message signatures at
/.well-known/http-message-signatures-directorythat allow brokers show their identification cryptographically. That is the Agent Title Service facet of issues, Cloudflare shipped with GoDaddy earlier in Brokers Week. Adoption is nearly zero outdoors Cloudflare’s personal properties.
API, Auth, MCP & Talent Discovery (6 Checks)
- API Catalog (RFC 9727). A machine-readable index of an internet site’s API endpoints at
/.well-known/api-catalog. - OAuth / OIDC discovery (RFC 8414). Normal OAuth 2.0 authorization server metadata at
/.well-known/oauth-authorization-serverand/.well-known/openid-configuration. - OAuth Protected Useful resource (RFC 9728). A web site declaring which endpoints are OAuth-protected and the way to authenticate.
- MCP Server Card (SEP-1649). A Mannequin Context Protocol server promoting its capabilities at
/.well-known/mcp/server-card.json. SEP-1649 is a draft proposal contained in the MCP spec course of. - Agent Expertise index. An inventory of agent-callable expertise at
/.well-known/agent-skills/index.json. Additionally rising. - WebMCP (Experimental). An in-page JavaScript API registering agent-callable instruments through
navigator.modelContext. The scanner makes use of headless browser rendering to detect whether or not the web site registers any WebMCP instruments on web page load.
Commerce (3 Non-compulsory Checks, Not Scored On Non-Commerce Web sites)
- x402 fee protocol. HTTP 402 Fee Required infrastructure for agent-native funds.
- UCP profile (Common Commerce Protocol). Google’s merchant-metadata standard at
/.well-known/ucp. - ACP discovery doc (Agentic Commerce Protocol). At
/.well-known/acp.json.
The Commerce class is flagged “elective” on non-commerce web sites. The scanner detects whether or not any ecommerce indicators are current and, if not, shows the commerce checks for informational functions with out counting them within the rating.
That final design element issues. It’s proof Cloudflare anticipated precisely the issue the remainder of this text is about.
Nohacks.co Scored 33/100, Degree 2 Bot-Conscious
I ran the scan on nohacks.co. The consequence was 33 out of 100, Degree 2 “Bot-Conscious.”
A word on that quantity: After the primary scan, I added Content material Indicators directives to robots.txt, which moved Bot Entry Management from 50 to 100 and pulled the composite up eight factors from an preliminary 25. Each different class beneath is unchanged from the primary scan. I’ll come again to the Content material Indicators repair and why I made it on the finish of this part.
Here’s what drove every class rating:
- Discoverability: 67. robots.txt and sitemap.xml handed. Hyperlink headers failed as a result of this web site doesn’t emit
Hyperlink:headers in its responses. - Content material: 0. Markdown content material negotiation will not be configured. The web site returns HTML whatever the
Settle forheader. - Bot Entry Management: 100. Each scored checks handed. AI bot guidelines in robots.txt (I’ve express guidelines for AI person brokers) and Content material Indicators in robots.txt (I added these after the primary scan). Internet Bot Auth request signing is listed on this class as an informational verify, however not counted towards the two/2.
- API, Auth, MCP & Talent Discovery: 0. All six checks failed. No API Catalog. No OAuth discovery. No OAuth Protected Useful resource metadata. No MCP Server Card. No Agent Expertise index. No WebMCP instruments on the web page.
- Commerce: not scored. nohacks.co has no e-commerce. The Commerce checks all failed, however the class is accurately excluded from the composite rating.
That may be a 33 on a scanner constructed by the corporate I most belief to know the place the agent-ready internet goes. I think about this web site fairly well-designed for brokers. The robots.txt is clear and express. The content material is server-rendered, machine-readable HTML with clear semantic construction. The sitemap is present. The URLs are secure. In case you requested me per week in the past whether or not this web site was agent-ready, my reply can be someplace between “largely sure” and “for what it must do, sure.”
And but: 33, Degree 2.
The scanner is measuring what it says it’s measuring. The composite rating, by itself, continues to be the mistaken quantity to optimize for.
One word on the Content material Indicators repair, as a result of it’s related to the Goodhart argument later on this article. Content material Indicators is a Cloudflare proposal with nearly no deployment past Cloudflare-aligned crawlers. I debated including it for precisely the score-chasing motive this text warns about. I made a decision it was defensible for 2 causes. First, the repair is declarative, not ornamental. The directives state actual coverage about what ought to occur with my content material, and the assertion has that means even when the spec fails. That’s totally different from including an empty MCP Server Card to fulfill a scorer. Second, for an internet site that writes about agent-readiness particularly, publicly declaring content material coverage is editorial apply no matter which crawler respects it. The repair was one decide to public/robots.txt and the directives are readable by any human curious sufficient to verify.
Similar Web site Scores 33 Or 67 Relying On The Preset You Choose
On the All Checks preset, nohacks.co scores 33 out of 100, Degree 2 “Bot-Conscious.” On the Content material Website preset, similar web site, similar day, totally different scan configuration, it scores 67, nonetheless Degree 2 “Bot-Conscious.” Almost double the composite quantity. The 34-point hole is the distinction between two scan configurations of the identical scanner, not a distinction between two web sites.
Here’s what the Content material Website preset adjustments within the scan configuration:
Operating that preset on nohacks.co produced this consequence:
4 of six scored checks go. The 2 failures are unambiguous remediation targets: Hyperlink headers through HTTP response configuration, Markdown content material negotiation through origin or CDN response logic. Each ship in opposition to actual agent-runtime conduct immediately. Neither is a proposal-stage format that can solely perhaps turn out to be a normal. That is the sincere studying of nohacks.co’s agent-readiness state: two particular, actionable gaps.
The Right Toggle Is Hidden, And The Default Rating Is Improper
The scanner is doing its job. It is aware of a weblog doesn’t want an MCP Server Card. It is aware of a podcast archive doesn’t publish an API catalog. The Content material Website preset will not be beauty. It removes irrelevant checks and provides a content material web site an correct studying in opposition to requirements that truly apply.
The issue is that the preset is hidden. When a person lands on isitagentready.com and pastes a URL, the default scan is All Checks. The Website Kind toggle that will swap to Content material Website or API/Software lives inside a Customise dropdown that almost all customers won’t ever open. The person clicks Scan, reads the composite quantity, takes a screenshot, shares it. The shareable quantity, the one which travels on social media, the one opponents examine throughout, is the All Checks composite.
For a content material web site that runs the default scan with out studying particular person checks, the composite is structurally too low. The 33 on nohacks.co is mistaken for the form of web site nohacks.co is. The 67 from the Content material Website preset is the correct studying. Two numbers from the identical scanner on the identical web site. The correct quantity is behind a dropdown. The mistaken quantity is on the entrance web page.
Any internet skilled who runs the scanner and plans to share the rating anyplace public must open Customise, choose the preset that matches their web site kind, and re-run earlier than sharing. With out that step, the general public rating will understate the web site’s precise agent-readiness, and the hole between the shared quantity and the correct quantity will probably be bigger for content material web sites than for API web sites (that are nearer to the All Checks baseline). Learn the person checks. Don’t share a composite till you understand which preset produced it.
For the document: the 67 is bothering me. I’m going to go get the 100. I do know precisely what the Goodhart part beneath is about to warn in opposition to, and I’m going to do it anyway. Two fixes stand between me and the 100. Each are five-minute jobs. Each map to actual agent-runtime conduct (Hyperlink headers for discovery, Markdown content material negotiation for environment friendly agent parsing), so not less than the motivation is reliable and never pure score-chasing. That caveat can be precisely what score-chasers say. Public scores are a gravitational discipline. Even the individual writing a protracted article about their unreliability finally ends up orbiting.
Agent Readiness Measures Supply, Not Message
Each class the Agent Readiness scanner checks is about supply: discoverability, content material negotiation, bot entry, API discovery, commerce protocols. None checks the standard of the message itself.
The scanner by no means asks whether or not your headlines are clear, whether or not your product descriptions persuade, whether or not your content material solutions the question effectively, whether or not your writing is any good. These are search engine marketing and CRO questions. They occupy the self-discipline of constructing the message higher. The Agent Readiness Rating occupies a distinct self-discipline solely. It asks whether or not an agent can fetch your content material, parse the format it arrives in, authenticate in opposition to your endpoints, name your features, pay on your outputs.
That’s the distinction that issues. Classical internet optimization (search engine marketing, CRO) is about what you say and the way persuasively you say it. Agent-readiness is about the way you ship what you say to a non-human reader. Two web sites can publish word-for-word an identical content material. One serves it as server-rendered HTML with semantic markup, responds to Settle for: textual content/markdown, exposes structured information, returns predictable response codes. The opposite serves it as a JavaScript-rendered single-page software with no content material negotiation and an inconsistent error floor. The message is an identical. The supply is totally different. The agent-readiness rating will probably be totally different. And it is going to be proper to be totally different, as a result of the supply is what the agent interacts with.
That is additionally why agent-readiness fixes are usually orthogonal to search engine marketing and CRO work. You’ll be able to enhance an agent-readiness rating with out rewriting a single phrase of your content material. You too can have world-class search engine marketing content material that scores a ten on the agent-readiness scanner as a result of none of your supply pipeline was designed for machine shoppers. search engine marketing and CRO work on the content material layer. Agent-readiness works on the transport and protocol layer. They’re adjoining however not the identical craft, and treating them as the identical is the error that turns an agent-readiness challenge right into a content-rewrite challenge and misses the precise repair.
The individuals who will do effectively over the subsequent a number of years are those who cease arguing about which self-discipline issues extra and begin recognizing they occupy totally different layers of the stack.
3 Goodhart Dangers Constructed Into The Agent Readiness Rating
Goodhart’s legislation says that when a measure turns into a goal, it stops being a very good measure. The Agent Readiness Rating is well-designed, however it is usually now a public, shareable, in contrast quantity, which produces three predictable behavioral failures within the wild.
The primary threat is that web site house owners will optimize for the quantity fairly than for actual agent conduct. Add an MCP Server Card that factors nowhere as a result of the scanner desires one. Publish an Agent Expertise index with no precise expertise. Ship a WebMCP instrument that does nothing simply to go the detection verify. The rating goes up, and nothing adjustments for actual agent runtimes visiting the web site.
The second threat is that consultancies will begin promoting “Agent Readiness Rating optimization” as a service, promoting the rating fairly than the underlying structure. The historical past of search engine marketing provides us a century of information on how this performs out. PageRank turned a goal, and a decade of link-spam financial system grew up round it. Core Internet Vitals turned a goal, and a era of performance-theater optimizations adopted. The Agent Readiness Rating is a better-designed metric than both of these had been at launch, however the identical gravity applies.
The third threat is that the scanner’s inclusion of rising requirements as scored indicators will speed up the adoption of these requirements previous the purpose the place they’re prepared to hold actual site visitors. The scanner checks for llms.txt, a proposed format for exposing web site content material to language fashions. Llms.txt is not a ratified standard, has no governing physique, and has competing proposals for the way it needs to be structured. Together with it as a scored sign provides it weight it has not earned within the ecosystem. A web site proprietor seeking to repair a failing verify is the marginal adopter who ideas a proposal right into a de facto commonplace earlier than the spec work is finished.
None of those failure modes are hypothetical. They’re how each public measurement rating within the historical past of the net has performed out. The Agent Readiness Rating is best than most as a result of Cloudflare is sincere about what it’s, as a result of the per-check element is accessible proper alongside the composite quantity, and since the Commerce class accurately excludes itself on non-commerce web sites. That honesty is a characteristic price defending. Web site house owners and the consultancy trade will probably be tempted to deal with the composite quantity because the goal anyway.
Don’t do that.
6 Weekend Fixes That Map To Actual Agent Runtimes
Six actions for an internet skilled working the scanner the weekend of its launch, ordered from highest-leverage to lowest:
- Run the scan in your web site. It takes about 30 seconds. Observe the rating and open the detailed report. The element is the place the sign is.
- Repair the failing checks that ship in opposition to actual agent runtimes immediately. These are those whose absence measurably hurts your web site for brokers visiting it proper now:
- robots.txt. If lacking, add one. If current, be sure that it accommodates particular guidelines for AI person brokers (GPTBot, ClaudeBot, PerplexityBot, Google-Prolonged, and many others.).
- sitemap.xml. If lacking, generate one and hyperlink it from robots.txt. Maintain it present.
- Markdown content material negotiation. Configure your origin or CDN to return
textual content/markdownwhen theSettle forheader requests it. Cloudflare’s personal AI Crawl Management has first-class assist for this. Different suppliers require customized server logic. - Structured information. Ship schema.org JSON-LD for the content material varieties your web site publishes (Article, Product, Group, BreadcrumbList). This isn’t a scored verify, however it’s the highest-leverage fix for citation behavior throughout each agent runtime at present deployed.
- Deal with the proposal-stage codecs as a watch checklist, not a guidelines. llms.txt, Content material Indicators in robots.txt, Internet Bot Auth, API Catalog, MCP Server Card, Agent Expertise, WebMCP, ACP, UCP are all actual working requirements in some sense. They aren’t transport in opposition to actual agent-runtime conduct at scale but. Watch them. Implement them when your stack has a motive to, not as a result of the scanner flags them.
- Ignore the composite quantity in your individual monitoring. Monitor particular person verify outcomes over time. A web site that goes from 3 of 5 real-runtime checks passing to five of 5 has measurably improved, even when the composite rating barely moved as a result of the ten proposal-stage checks nonetheless fail.
- Re-scan after adjustments. The scanner is quick, free, and obtainable through the URL Scanner API if you wish to script regression checks into your deployment pipeline.
- Skip the consultancies promoting Agent Readiness Rating optimization. The work is easy sufficient {that a} half-day audit and a centered remediation dash will beat any packaged service.
The scanner is the instrument. The work continues to be the work.
Vendor-Particular Scanners Are Coming: Monitor What Each Scanner Checks
The Agent Readiness scanner is standards-list-shaped: a set of checks in opposition to a set checklist of protocols and codecs, some ratified (RFC 8288 Hyperlink headers, RFC 9309 robots.txt guidelines, RFC 8414 OAuth discovery, RFC 9727 API Catalog, RFC 9728 OAuth Protected Useful resource), some rising proposals (MCP SEP-1649, WebMCP, Content material Indicators, Internet Bot Auth, x402, UCP, ACP, llms.txt). The subsequent factor that occurs within the ecosystem is predictable: Other vendors will ship their own scanners in opposition to their very own most popular lists. The overlap will probably be important as a result of many of the ratified requirements are uncontroversial. The divergence will probably be wherein proposals every vendor scores for.
That divergence is the place the agent-readiness measurement story will get fascinating. A Cloudflare scanner that checks for Internet Bot Auth and UCP is betting. A Google scanner, if it ships, would verify for among the similar issues and a few totally different ones (Google has UCP, doesn’t have Internet Bot Auth). A Perplexity scanner would verify for yet one more set. Web site house owners would see totally different scores from totally different scanners on the identical web site. The composite quantity, already not reliable, turns into vendor-specific.
The sign price monitoring is which checks present up in each scanner that ships. Those are the de facto standards. The checks that solely present up in Cloudflare’s scanner are Cloudflare’s bets. Some will win. Most is not going to.
That is the sample that made me snug publishing an article a few Cloudflare instrument on the day it shipped. The Agent Readiness Rating is actual. The thesis behind it (agent-readiness is a measurable property) is the suitable thesis. The precise scorecard is model one among one thing that’s going to have dozens of variations, every reflecting its vendor’s bets. Internet professionals ought to have interaction with the version-one scorecard, repair what it accurately flags as actual, watch what it flags as rising, and maintain their very own working checklist of which checks survive throughout each scanner that ships within the subsequent six months.
That working checklist is the actual agent-readiness commonplace. The composite rating is the advertising and marketing layer.
Run the scan. Learn the report. Repair what issues. Watch what may.
Extra Sources:
This put up was initially revealed on No Hacks.
Featured Picture: RobinRmD/Shutterstock
Source link
