Do You Need an llms.txt File? An Honest Answer

Short answer: probably not for AI search visibility — and if an agency is charging you to create one, ask them to show you the crawler logs first.

llms.txt is a proposed convention: a Markdown file at your domain root that gives AI systems a curated map of your most important pages, with short descriptions. It was proposed in late 2024, spread quickly through documentation platforms, and by 2026 has become a fixture in GEO audit checklists and agency service menus — usually framed as “the new robots.txt” and priced accordingly.

We review a lot of GEO recommendations in the course of our placement work, and this one comes up constantly, so let’s do what the sales decks don’t: look at the measured behavior.

What the data shows

Three independent measurements, three consistent answers.

Adoption is niche and flat. SE Ranking analyzed nearly 300,000 domains and found an llms.txt file on just 10.13% of them — after more than a year of industry conversation. More telling than the adoption number was the correlation analysis: both their statistical tests and their machine-learning model found no relationship between having an llms.txt file and how often a domain gets cited by AI systems. Removing the variable from their model actually improved its accuracy — the file was adding noise, not signal.

Crawlers barely fetch it. Otterly.AI ran the direct test: implement the file, then watch the server logs. Over a 90-day window, their test domain recorded 62,100 AI bot visits — of which 84 requested the llms.txt file. That’s 0.1% of AI crawler traffic. GPTBot, ClaudeBot, PerplexityBot and the rest overwhelmingly crawl your HTML directly, the same way they always have.

Google has answered on the record. Asked why llms.txt submissions weren’t showing up in crawl logs, Google’s John Mueller was blunt: none of the AI services have said they’re using it, and you can see in your own server logs that they don’t even check for it. He compared the file to the keywords meta tag — the canonical example of an SEO artifact that survives on hope rather than function. Google has separately confirmed that AI Overviews and AI Mode rely on regular search signals, with no special file required.

The pattern across all three is the same: as a lever for getting cited by ChatGPT, Perplexity, Gemini, or AI Overviews, llms.txt is — as of today — inert. Anyone selling it as a citation tactic is selling ahead of the evidence.

Where llms.txt actually works

Honesty cuts both ways, because there is one context where the file demonstrably earns its keep: developer tooling.

AI coding assistants — Cursor, Claude Code, GitHub Copilot and their peers — routinely fetch llms.txt when pointed at a documentation site. The agent reads the curated map, pulls only the relevant pages, and writes integration code against current docs instead of hallucinating an endpoint from stale training data. This is why Stripe, Vercel, Cloudflare, and Anthropic all ship the file: their users are literally building with AI assistants, and a clean llms.txt is the difference between generated code that works and generated code that doesn’t.

So the decision rule is simple:

• You publish API docs or technical documentation that developers consume through AI tools → ship an llms.txt. It’s a genuine developer-experience improvement, and it costs half a day.
• You’re a brand or e-commerce site hoping the file will get you mentioned in AI answers → the evidence says it won’t, and that half-day has a much better home.

One boundary worth stating because vendors blur it: llms.txt is not an access-control tool. It cannot block or restrict any crawler. That job belongs to robots.txt, which every major AI crawler actually respects.

What to spend the effort on instead

The uncomfortable thing about llms.txt as a deliverable is that it feels like GEO progress — a file was created, a checkbox was ticked — while the factors that measurably drive AI citations go untouched. Those factors are content properties, not root-directory files: facts specific enough to quote, claims attributed to identifiable people and organizations, definitions clean enough to reuse, and — the one no on-page work can fake — your claims corroborated on independent sites that AI systems already trust. We’ve laid out that full framework in our guide to the four signals that drive AI citations, along with the workflow for acting on them.

The one-line version: AI citations are earned by what your content says and who corroborates it — not configured by a file in your root directory.

Frequently asked questions

EggROI Editorial Team

The EggROI editorial team reviews publisher quality, placement context, and content citability across the guest post and press placements we process. Everything we publish follows the same standard we apply to client work: sourced data, disclosed interests, and no fabricated statistics.