When an AI agent reads a web page, it pays for every byte of markup it does not need. Navigation, scripts, styles and wrapper divs all consume context that could have held your actual content.
Worse, when a page is too large the agent truncates it. Your answer may be sitting in the half that never got read.
Content negotiation is a standard part of HTTP. The client states what format it wants, and the server responds in that format.
When a request carries Accept: text/markdown, return Markdown instead of HTML.
GET /blog/my-article HTTP/1.1Host: www.example.comAccept: text/markdown
HTTP/1.1 200 OKContent-Type: text/markdown; charset=utf-8Vary: Accept# My articleThe content, with no navigation, scripts or wrappers.
✅ Figure: Good example - Good example - The agent asks for Markdown and gets Markdown
Cloudflare measured up to 80% fewer tokens compared to the HTML equivalent. Cheaper to read, faster to process, and far more likely to be consumed whole rather than truncated.
Vary: AcceptThis is the trap that bites people. Without Vary: Accept, a CDN or browser cache can store the Markdown response and serve it to the next visitor, who gets a wall of plain text instead of your website.
If one URL can return more than one format, Vary: Accept is mandatory. Skipping it is a caching bug waiting to happen.
Not every agent sends the header. Some coding agents do, most other clients do not.
So expose the same content at a predictable URL as well. The convention is to append .md to the page path.
| Page | Markdown |
/blog/my-article | /blog/my-article.md |
/ | /index.md |
Do both. The header is the correct mechanism, and the URL is the fallback for agents that have not caught up.
If your content is already authored in Markdown, and it is on any Git-based CMS, then the Markdown already exists. You are converting it to HTML at build time and throwing the original away.
Instead of generating Markdown from your HTML, serve the source you started with. This is a build step or an edge function, not a rewrite.
They solve different problems and you want both.
llms.txt is one index file at the root, listing what your site contains and where to find it. It helps an agent decide what to readcurl -sI -H "Accept: text/markdown" https://www.example.com/ | grep -i "content-type\|vary"
If Content-Type comes back as text/html, you are not negotiating. Adoption is still low, as Cloudflare's Agent Readiness score shows, so this is an easy way to stand out.