llms.txt: What It Actually Does for Your AI Visibility (Honest 2026 Review)

Here's the short answer most llms.txt guides won't give you: as of mid-2026, the major AI search crawlers largely ignore llms.txt. One monitoring study logged over 500 million AI bot visits across 90 days — only 408 of them requested an llms.txt file. Google confirmed back in July 2025 that it doesn't support the standard and has no plans to. OpenAI, Anthropic, and Perplexity have never committed to reading it.

So should you skip it? Not quite — but you should understand what it actually does before you spend an afternoon on it. This post covers what llms.txt is, what the adoption data really shows, who genuinely benefits from one, how to create one properly if you should, and — most importantly — what to do instead if your goal is AI search visibility.

What is llms.txt?

llms.txt is a proposed standard, introduced by Jeremy Howard of Answer.AI in September 2024: a markdown file at your site root (yoursite.com/llms.txt) that gives AI systems a curated map of your most important content — a table of contents written for machines.

The reasoning goes like this: AI models have limited context windows, and HTML pages are full of navigation, scripts, and clutter. A single clean markdown file lets an AI understand your site without crawling and guessing.

A minimal llms.txt looks like this:

# YourCompany

> One-sentence description of what your company does.

## Docs

- [Getting started](https://yoursite.com/docs/start.md): Setup in 5 minutes
- [API reference](https://yoursite.com/docs/api.md): Full endpoint documentation

## Product

- [Pricing](https://yoursite.com/pricing.md): Plans and what's included

That's the whole format: an H1 with your name, a blockquote summary, then H2 sections containing annotated links. Some sites also publish llms-full.txt, which inlines the complete content of every page into one file instead of linking out.

It's a reasonable idea. Adoption tells a different story.

Does llms.txt actually work? What the data shows

Three findings define the state of llms.txt in 2026:

• Adoption is stuck around 10%. An SE Ranking study of 300,000 domains found a 10.13% adoption rate — after eighteen months of industry hype, roughly one site in ten has the file, and the growth curve is flat.
• AI search crawlers don't fetch it. Server log analysis across 500+ million AI bot visits found GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended requested llms.txt just 408 times in 90 days. They crawl your HTML directly, exactly like traditional search bots.
• Google publicly rejected it. Google's Gary Illyes confirmed in July 2025 that Google doesn't use llms.txt and isn't planning to — he compared it to the old keywords meta tag. No major AI company has committed to the standard since.

If a guide tells you llms.txt is "essential for ranking in ChatGPT," check what they're selling.

Where llms.txt genuinely works: AI coding assistants

Here's the twist: llms.txt found real users — just not the ones it was designed for. AI coding assistants use it routinely. Cursor, Claude Code, GitHub Copilot, Windsurf, Cline, and Aider all look for /llms.txt and /llms-full.txt when developers point them at a documentation site.

That changes the recommendation depending on who you are:

1. You run a developer tool, API, or documentation-heavy product: create an llms.txt. Developers using AI coding tools will hit it daily, and it directly improves how those tools answer questions about your product. This is the one audience where the file has proven, measurable use.
2. You run a store, agency site, or standard marketing site: it's a 30-minute nice-to-have, not a priority. It won't hurt, and if the standard gains crawler support later, you're early. But do it after the fundamentals below, not instead of them.

llms.txt vs robots.txt: don't confuse them

A common mix-up worth clearing:

• robots.txt controls access — it tells crawlers (including AI crawlers like GPTBot and ClaudeBot) which parts of your site they may visit. Every AI company that matters respects it, which also means a careless robots.txt can silently block you from AI training data and search indexes. This file is critical whether or not you care about llms.txt.
• llms.txt offers guidance — a suggested reading list AI systems may use. Nothing enforces it, and as the data above shows, search crawlers mostly don't.

If you only check one file this week, make it robots.txt. Our free robots.txt tester shows exactly which AI crawlers can and can't access your site.

How to create an llms.txt (if you should)

If you're in the "developer tool or docs-heavy" camp — or you've covered the fundamentals and want the option value — here's the right way to do it:

1. Start with your five to ten most important pages. Documentation, pricing, product overviews, key guides. Curation is the point; don't dump your whole sitemap.
2. Follow the format: one H1 (your name), one blockquote (what you do, one sentence), H2 sections with markdown links, each link followed by a one-line description.
3. Link to markdown versions of pages where possible. The standard suggests appending .md to URLs for clean-text versions. If you can't serve markdown, linking to your HTML pages still works.
4. Keep it current. A stale llms.txt that points at dead pages is worse than none — AI coding tools will confidently give users outdated answers.
5. Serve it at the root as plain text (text/plain or text/markdown) at /llms.txt.

Then verify it's reachable: fetch yoursite.com/llms.txt and confirm you get the file, not a 404 page or a redirect to your homepage.

What actually gets you cited by AI

If llms.txt isn't the lever, what is? The citation-pattern studies from the past year are consistent about what moves the needle:

• Structured data (schema markup). Pages with substantially complete schema get cited in roughly 54% of eligible answers versus 44% without — the single most reliable technical lever. Validate yours with our free schema checker.
• Freshness. Content updated within the last 30 days earns about 3x more AI citations than stale content. AI search engines heavily prefer recent sources — a strong argument for updating your best pages quarterly rather than always writing new ones.
• Statistics and specifics. Princeton researchers found adding concrete statistics can raise AI visibility by up to 40%. Vague claims don't get quoted; numbers do.
• Answer-first structure. Put the direct answer to the target question in the first 200 words, then elaborate. AI systems extract passages, and self-contained passages win. (You may have noticed this post opens with its conclusion — that's not an accident.)
• Named authors. A real byline measurably increases citation odds versus anonymous content — expert attribution is part of how AI systems weigh trustworthiness.
• Crawler access. None of the above matters if your robots.txt blocks GPTBot or ClaudeBot. This is the first thing to check and the most common silent failure we see.

Notice what's on that list: fundamentals. The uncomfortable truth about GEO in 2026 is that there's no magic file — being citable is mostly being clear, current, structured, and accessible. If you're new to this whole space, our guide to what AI search visibility is and why it matters covers the foundations.

Check where you stand in 60 seconds

DidYouSEO's free audit checks the things on that list — AI crawler access in your robots.txt, schema markup validity, content structure — across 30+ checks, no signup required. Run a free audit and you'll know whether you're invisible to AI crawlers before you spend another hour writing content for them.

FAQ

Does llms.txt help SEO? No — not traditional SEO, and per current crawler data, not AI search visibility either. Google has stated it doesn't use llms.txt. Its proven benefit is limited to AI coding assistants reading documentation sites.

Do ChatGPT and Claude read llms.txt? Their web crawlers (GPTBot, ClaudeBot) almost never request it. However, the coding tools built on these models (Claude Code, Cursor, Copilot) do fetch llms.txt when working with documentation sites.

Should I make an llms.txt or llms-full.txt? Start with llms.txt (the index). Add llms-full.txt only if you have documentation that AI coding tools consume — it inlines full page content and gets large fast.

Is llms.txt the same as robots.txt? No. robots.txt controls crawler access and is respected by every major AI company. llms.txt is optional guidance that most crawlers currently ignore. robots.txt is the one that can make or break your AI visibility.

Will llms.txt matter in the future? Possibly — standards sometimes win slowly. The cost of having one is 30 minutes, so being early is cheap. Just don't prioritize it over schema markup, content freshness, and crawler access, which demonstrably affect citations today.

The bottom line

llms.txt is a good idea waiting for adoption that hasn't come. Create one if you serve developers, or as cheap option value once your fundamentals are covered — but if you're choosing where to spend your next hour of SEO effort, spend it on schema, freshness, and crawler access. That's where the citations are.