llms.txt and AI-Readable Files: The robots.txt Moment for AI Crawlers

There is a tiny file most sites do not have yet, and it exists to hand AI crawlers a map of your best content.

TL;DR llms.txt is an emerging convention: a small markdown file at your site root that points AI systems to your most important, cleanest content. Paired with markdown mirrors of your key pages, it makes your site easier for models to read. Support is not universal yet, so treat it as a cheap, forward-looking bet, not a magic switch.

What llms.txt actually is

llms.txt is a plain-text file, written in markdown, that you place at the root of your domain, the same spot where robots.txt and sitemap.xml already live. Its job is simple: tell AI systems what your site is about and point them to the pages worth reading.

Instead of making a language model crawl your entire site and guess which pages matter, llms.txt hands it a short, curated list. Here is our product. Here is our documentation. Here is the page that explains pricing. Here is the article that answers the question people actually ask. It is a guided tour rather than a filing cabinet.

The format is deliberately readable. It is markdown, so it is mostly headings, short descriptions, and links. A human can skim it in ten seconds, and a model can parse it without wading through scripts, styling, or navigation menus. That readability is the entire point.

One thing to be clear about up front: llms.txt is a proposed convention, not an official web standard. Jeremy Howard of Answer.AI proposed it on September 3, 2024, and the spec is still explicitly "open for community input." Some tools have started to support it, but no governing body has ratified it and no AI engine promises to read it. More on that honesty below.

How it relates to robots.txt and sitemaps (and how it differs)

If you already know robots.txt and sitemap.xml, llms.txt slots in next to them without replacing either.

robots.txt is the rulebook. It tells crawlers which parts of your site they are allowed to touch and which to leave alone. It is about permission and access.

sitemap.xml is the inventory. It lists every URL you want search engines to know about, so nothing important gets missed. It is about completeness.

llms.txt is the guided tour. It does not gate access and it does not try to list everything. It curates: of all the pages on this site, these are the ones that best explain who we are, with a one-line summary of each so you know what you are getting.

The three work at different layers. robots.txt controls the door. sitemap.xml maps every room. llms.txt walks a model straight to the rooms that matter. You can and probably should have all three.

An illustrative example

Here is what a simple llms.txt might look like. This is illustrative, not a real file, so treat it as a shape to copy rather than a spec to obey.

# Acme Analytics

> Acme Analytics is a product-analytics tool that helps small teams
> understand how people use their software, without a data team.

## Core pages

- [What Acme does](https://example.com/product.md): The one-page
  overview of the product and who it is for.
- [Pricing](https://example.com/pricing.md): Plans, limits, and what
  each tier includes.
- [Getting started](https://example.com/docs/quickstart.md): Install
  and send your first event in ten minutes.

## Guides

- [Event tracking basics](https://example.com/guides/events.md): How
  to name and structure events cleanly.
- [Funnels explained](https://example.com/guides/funnels.md): Build a
  conversion funnel step by step.

## About

- [Company](https://example.com/about.md): Who we are and why we
  built this.

Notice a few things. There is a short description of the whole site at the top, in a blockquote, so a model gets context before it follows a single link. Each link has a plain-English summary next to it. And several of those links point to .md versions of pages, which brings us to the second half of this story.

Markdown mirrors: clean copies machines can actually read

A modern web page is a heavy thing. A single article can ship with navigation bars, cookie banners, share buttons, analytics scripts, lazy-loaded images, and layers of styling, all wrapped around the few hundred words you actually wrote. A human eye filters that noise automatically. A machine has to work harder.

A markdown mirror is the fix. It is a clean, plain-text version of a page, usually the same content served at the same path with a .md extension, for example /pricing and /pricing.md. Strip out the scripts, the layout, and the chrome, and what remains is your actual content in a format a model reads without effort: headings, paragraphs, lists, links.

Why does this matter for AI? Cleaner input means fewer chances to misread you. Heavy HTML forces a model to guess where the real content ends and the interface begins. A markdown mirror removes the guessing, and your meaning arrives intact.

You do not need markdown copies of every page on your site. Mirror the pages that carry your core message: your product explainer, your pricing, your key guides, the handful of articles you most want quoted correctly. Then point llms.txt at those clean copies.

How to create and host these files

Conceptually, the work is small.

For llms.txt, you write a markdown file, list your most important pages with a one-line summary each, and place it at the root of your domain so it is reachable at /llms.txt. It is a static text file, so hosting it is no harder than any other file on your site.

For markdown mirrors, you generate a plain-text markdown version of each key page and serve it at a predictable path, most commonly the page URL with .md on the end. Some sites produce it from the same source that builds the HTML, others convert the rendered page. Either way, the goal is clean output.

Two habits keep this useful. Keep the files current, because a tour that points to a deleted page is worse than no tour at all. And be selective, because the value of llms.txt is curation. A file that links to all four hundred of your pages has thrown away the one thing that made it worth having.

What to include, and what to leave out

Include the pages that explain what you do and answer real questions: your product overview, pricing, core documentation, your best guides, and an about page that establishes who you are. Write honest one-line summaries. Point to markdown mirrors where you have them.

Leave out the clutter. Login pages, cart pages, thank-you pages, thin tag archives, and anything gated or private do not belong in a file whose whole purpose is to highlight your strongest public content. And never treat llms.txt as a security control. It is a suggestion, not a lock. If content should not be public, protect it properly rather than hoping a crawler skips it because you left it off a list.

An honest note on adoption

Here is the part a lot of writeups skip. llms.txt is early. It is a community proposal that has picked up momentum, some tools look for it, and interest is real. A SE Ranking scan of roughly 300,000 domains found llms.txt on 10.13% of them, so about one site in ten after a year and a half of conversation. Developer-facing companies led the way: Anthropic, Stripe, Cloudflare, Vercel, and Supabase all serve one. But it is not an official standard, adoption is uneven, and no major AI engine promises to read your file or quote you because you have one.

In fact, the biggest engines say plainly that they do not use it. Google's John Mueller called llms.txt "purely speculative for now," noting the file has existed for years yet no AI system fetches it, and compared it to the long-ignored keywords meta tag. The same SE Ranking analysis found no meaningful correlation between having an llms.txt file and being cited by AI; at most, GPTBot fetches one occasionally. So take the enthusiasm with a grain of salt.

So why bother? Because the cost is close to zero and the downside is nothing. Curating your best content and serving clean copies helps machines read you regardless of whether any single convention wins. If llms.txt becomes widely honored, you are ready. If it does not, you still spent an afternoon making your most important pages easier to parse. That is a good trade either way. Just do not oversell it to yourself: this is a low-cost bet on where AI crawling is heading, not a guarantee of citations.

Key takeaways

• llms.txt is a small markdown file at your site root that points AI systems to your most important, cleanest content. It curates; it does not gate or inventory.
• It sits alongside robots.txt (the rules) and sitemap.xml (the full list) without replacing either. Keep all three.
• Markdown mirrors are clean plain-text copies of key pages that machines parse far more reliably than heavy HTML. Mirror your core pages, not every page.
• It is an emerging convention, not an official standard. It sits on about 10% of domains, Google says its Search does not use the file, and no AI engine guarantees it will read yours.
• The cost is near zero and the downside is nothing, which makes it a sensible forward-looking bet even while adoption is early.

Frequently asked questions

What is llms.txt?

llms.txt is a proposed plain-text file, written in markdown, that you place at the root of your site. It points AI systems to your most important, cleanly formatted content so a model can find and quote the right pages. It is an emerging convention, not an official standard, and not every AI engine reads it yet.

How is llms.txt different from robots.txt and sitemap.xml?

robots.txt tells crawlers what they may and may not access. sitemap.xml lists every URL you want indexed. llms.txt does something different: it curates and explains your best content for a language model, in a format the model can read easily. Think of robots.txt as the rules, sitemap.xml as the full inventory, and llms.txt as the guided tour.

Do AI crawlers actually read llms.txt?

Rarely, if at all. Google has said its Search systems do not use the file, and independent log analysis shows the major AI crawlers seldom request it, with GPTBot fetching one only occasionally. Support is far from universal, and no major AI engine guarantees it will honor the file. Treat llms.txt as a low-cost, forward-looking signal that may help, not a switch that forces AI systems to quote you.

What is a markdown mirror?

A markdown mirror is a clean, plain-text version of a web page, usually served at the same URL with a .md extension. It strips out navigation, scripts, ads, and styling so a model gets your actual content without the noise. Heavy HTML is harder for machines to parse, so a markdown copy makes your meaning easier to read.

Will llms.txt help my site get cited by AI?

It might help, but it is not a guarantee, and it is not a substitute for good content. AI systems still weigh quality, clarity, structure, and trust. llms.txt and markdown mirrors make it easier for a model to read what you already have. They raise your odds, they do not rig the outcome.

Sources

• llms.txt specification and proposal - Jeremy Howard, Answer.AI, published September 3, 2024.
• llms.txt adoption study across ~300,000 domains - SE Ranking (10.13% adoption; no meaningful correlation with AI citations).
• Google says llms.txt is "purely speculative for now" - Search Engine Journal, on John Mueller's comments.
• llms.txt implementation guide and company examples - lists Anthropic, Stripe, Cloudflare, Vercel, Supabase.

Curious whether machines can read your site cleanly today? Run a free AI-search scan and see how your structure, markup, and AI-readiness hold up, along with the specific fixes worth making first.