Short answer: as of mid-2026, llms.txt does almost nothing to improve AI search visibility for most sites — but it isn't pointless, and the nuance is where the budget decision lives. llms.txt is a proposed standard file you place at your site root to hand large language models a clean, structured map of your content. The pitch is seductive: feed AI a tidy summary, get recommended more often. Yet independent server-log data from hundreds of thousands of sites shows AI crawlers barely request the file. This guide separates the wishful thinking from what the logs actually prove, then gives you a working template and a way to test the file on your own domain.
We track how brands surface across ChatGPT, Gemini, Perplexity, Claude, Copilot, and Google AI Overviews every day, so this is written from the measurement side: not "here's a shiny new file," but "here's what moves cited brand mentions and what doesn't."
What is llms.txt?
llms.txt is a plain-text Markdown file placed at your site root (/llms.txt) that gives language models a curated, ad-free map of your most important content. Jeremy Howard of Answer.AI proposed it on September 3, 2024. The reasoning: robots.txt tells crawlers what they may fetch, but nothing tells an LLM what a site is about or which pages matter most.
The format is simple and human-readable. The only required element is an H1 with the site or project name. Everything else is optional but recommended: a short blockquote description, free-form detail, and curated link lists with one-line annotations. A companion llms-full.txt can hold the entire site as a single Markdown document for ingestion. The full spec lives at llmstxt.org.
Early adopters skew technical — Anthropic, Stripe, Cursor, Mintlify, Zapier, and FastHTML — which is a clue about who the file genuinely serves. (Mintlify auto-generates it across every docs site it hosts, which is how thousands of pages got one overnight.) That pattern matters when we decide who should bother.
Does llms.txt actually work for AI visibility?
No — not in any measurable way for brand citations in AI search. Multiple 2026 studies converge on the same result: AI crawlers almost never fetch the file, and its presence shows no correlation with more AI mentions or rankings. The data is unusually consistent for such a hyped topic.
Three sources worth putting side by side:
- Ahrefs (May 2026): Across 137,210 domains in a bot-traffic study, 97% of llms.txt files received zero requests of any kind. Of the files that did see traffic, most was non-AI tooling (SEO auditors, tech profilers, generic web crawlers). AI bots were only about 19% of requests — and within that slice, coding agents and training crawlers dominated, while the AI search/assistant bots that actually generate citations made up roughly 2%.
- Otterly.ai (90-day single-site test): Out of 62,100+ AI bot visits, only 84 reached
/llms.txt— about 0.1% of AI traffic, and roughly 3× fewer than the site's average page (~265 visits). Otterly concluded llms.txt is not a significant driver of AI crawl behavior. - Google: John Mueller said no AI service has confirmed using the file and that "you can tell when you look at your server logs that they don't even check for it," likening it to the long-dead keywords meta tag. Google has not announced support, and AI Overviews don't consult it.
If you take one thing from this section: the file is rarely read, and "rarely read" cannot move citations. For a deeper breakdown of the citation numbers, see Does llms.txt Work? Evidence From AI Citation Data.
Why the "it works" claims and the server logs disagree
The gap is real and explainable: vendors and platform blogs describe intended behavior, while server logs record actual behavior — and the two rarely match. You'll find confident posts claiming "Perplexity retrieves llms.txt" or "Claude respects it." Much of that traces to support docs or roadmap language, not measured retrieval at inference time.
The honest reconciliation in one table:
| Platform | What's claimed publicly | What server logs show |
|---|---|---|
| Google (Search / AI Overviews) | Does not support it; "not planning to" | No Googlebot or Google-Extended fetches of the file |
| OpenAI (ChatGPT / SearchGPT) | No official commitment to read it | GPTBot (a training crawler) appears occasionally; OAI-SearchBot, the answer bot, barely does |
| Anthropic (Claude) | Publishes its own llms.txt; no inference-time guarantee | Claude-Code (a coding agent) fetches docs files far more than any search bot |
| Perplexity | Sometimes cited as "supporting" it | PerplexityBot requests to the file are negligible in logs |
The most important row is Anthropic's. In the Ahrefs data, the AI traffic that did reach llms.txt files came disproportionately from AI coding agents pulling developer documentation — Claude-Code and GPTBot were the top individual bots — not from the search-and-answer bots that decide whether your brand lands on a shortlist. That single distinction explains most of the confusion: llms.txt has a narrow, real use case, and it is not "get recommended by ChatGPT."
When is llms.txt actually worth the effort?
Worth it if you publish developer docs or API references that AI coding agents ingest, or if your platform generates the file automatically for near-zero cost. Skip it if you're a marketing site hoping it lifts brand mentions in AI answers — the evidence says it won't. Treat the decision as cost-versus-evidence, not an article of faith.
A quick framework:
Keep or add llms.txt when:
- You run technical docs, an API reference, or a dev tool. Agents like Claude-Code and Cursor genuinely fetch curated Markdown maps.
- Your docs platform (Mintlify and similar) auto-generates and auto-updates it. Free upside, no maintenance tax.
- You want a tidy, canonical content index for internal LLM/RAG use.
Deprioritize or skip when:
- You're a brand chasing AI Overviews, Perplexity, or ChatGPT citations. There's no measured payoff.
- Maintaining it by hand would steal time from fixing crawlability, entity facts, or earned mentions.
- You'd be tempted to serve LLM-only content no human sees — a pattern that edges toward cloaking, which Google has openly criticized.
The file is cheap scaffolding for the right site and a distraction for the wrong one. Most teams reading this are the wrong site for it — and that's fine, because the real levers are elsewhere.
How to write an llms.txt file (template included)
To write a valid llms.txt, create a Markdown file at /llms.txt with an H1 site name, a one-line blockquote description, and short curated link lists pointing only to pages AI can already read. Keep it small and current; a bloated or stale file is worse than none.
Follow these steps:
- Create the file at your site root so it resolves at
https://yourdomain.com/llms.txt. - Add an H1 with your brand or project name (the only required element).
- Write a one-sentence blockquote describing what you do, in plain language.
- List 5–15 high-value pages in grouped sections (Docs, Key pages, Optional), each with a one-line annotation.
- Link only to crawlable, canonical URLs — not JavaScript-rendered pages an AI bot can't read.
- Set the file to
noindexif you don't want the raw file cluttering search results.
A minimal llms.txt template
# Acme Analytics
> Acme Analytics is a product analytics platform for B2B SaaS teams,
> tracking activation, retention, and revenue in one place.
## Docs
- [Quickstart](https://acme.com/docs/quickstart): Set up in 10 minutes
- [API reference](https://acme.com/docs/api): Full REST API
- [Integrations](https://acme.com/docs/integrations): Supported data sources
## Key pages
- [Pricing](https://acme.com/pricing): Plans and usage limits
- [Security](https://acme.com/security): SOC 2, GDPR, data handling
## Optional
- [Changelog](https://acme.com/changelog): Recent releases
Common mistakes that quietly waste the effort
- Dumping every URL. The point is a curated map; a sitemap-sized dump defeats it.
- Letting it drift. A file that contradicts your live site teaches AI nothing useful.
- Linking to content crawlers can't render. If the target page is client-side JavaScript, the link is dead weight.
- Blocking the bots, then hoping llms.txt saves you. If your robots.txt blocks GPTBot or ClaudeBot, the file is moot — decide crawler access first.
- Treating it as a substitute for readable HTML. It isn't. AI reads your pages, not your wishes about your pages.
What actually gets your brand recommended by AI
If llms.txt isn't the lever, here's what is: crawlable HTML, clear entity facts, earned third-party citations, and answer-first content — the things AI systems demonstrably read and repeat. This is where the budget that almost went into a root-directory file should go instead.
In day-to-day brand-mention tracking, the pages that get cited share a profile: they're directly readable in HTML, they state facts in clean passages, and they're reinforced by mentions on sites AI already trusts. None of that depends on llms.txt. The practical priorities:
- Make sure AI can read you at all. Crawler access is the foundation — if bots can't fetch your pages, nothing else matters.
- Publish unambiguous brand facts. Answer engines repeat entities they can parse: a consistent name, category, offerings, and proof points.
- Earn off-site mentions. Reddit, G2, Wikipedia, and YouTube feed AI shortlists — see AI Recommends Competitors: Why It Happens and How to Win Back AI Shortlists.
- Write in quotable passages. Lead each section with a direct, 40–60 word answer so models can lift it cleanly.
Do these four and your answer engine optimization and generative engine optimization efforts compound. Do llms.txt instead, and you've optimized a file almost nobody fetches.
How to test whether llms.txt does anything for your site
Don't take anyone's word — including ours. Run a two-part test: check your server logs for AI-bot requests to /llms.txt, and baseline your AI citation rate before and after adding the file. That's the data-backed way to settle it for your specific domain.
A simple protocol:
- Pull 30–90 days of server or CDN logs. Filter for AI user agents (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended) and count hits to
/llms.txtversus your top content pages. Most sites will see near-zero on the file. - Baseline your citations first. Record how often your brand is mentioned and cited across AI platforms before changing anything. A structured starting point is the No-Code GEO Audit.
- Add the file, then wait and compare. Watch for any shift in citation frequency, controlling for content and PR changes so you don't credit llms.txt for unrelated wins.
- Track continuously, not once. AI answers are noisy; a single snapshot lies. Sound method matters — see AI Search Monitoring Methodology.
This is exactly the before/after an ai visibility tool is built to measure. Daily ai search monitoring turns "I think it helped" into a defensible number you can put in front of a budget owner — which is the whole point of llm brand tracking in the first place.
The bottom line on llms.txt
llms.txt is a thoughtful idea that the market got ahead of. For developer docs consumed by AI agents, it's a low-cost nicety worth keeping if it's auto-generated. For everyone chasing brand mentions in ChatGPT and AI Overviews, the honest read from server logs and controlled experiments is the same: it doesn't move the needle. Spend the effort on crawlability, entity clarity, earned citations, and measurement — then verify with your own data rather than the hype.
Frequently asked questions
Is llms.txt the same as robots.txt?
No. robots.txt controls access — which crawlers may fetch which paths. llms.txt is a content suggestion — a curated map of pages for LLMs. robots.txt is widely honored; llms.txt is widely ignored.
Does Google use llms.txt?
No. Google representatives have said Search does not support it and isn't planning to, and likened it to the deprecated keywords meta tag. AI Overviews don't consult it.
Should I add llms.txt anyway?
Only if it's auto-generated for free, or you publish developer docs that AI coding agents ingest. For a marketing site hoping to raise AI citations, the measured payoff is effectively zero.
Will llms.txt get my brand into ChatGPT answers?
Not on its own. ChatGPT's search bots rarely fetch the file. Getting recommended depends on readable content, clear entity facts, and trusted off-site mentions — covered in Why AI Search Engines Cite Competitor Pages Instead of Yours.
What's the difference between llms.txt and llms-full.txt?
llms.txt is a short, curated index of key links. llms-full.txt is an optional companion containing your full site content as one Markdown document for ingestion. Both live at the site root; neither is widely fetched today.