{"id":804,"date":"2026-06-29T03:57:01","date_gmt":"2026-06-29T03:57:01","guid":{"rendered":"https:\/\/maxaeo.ai\/blog\/llms-txt-ai-visibility\/"},"modified":"2026-06-29T03:57:01","modified_gmt":"2026-06-29T03:57:01","slug":"llms-txt-ai-visibility","status":"publish","type":"post","link":"https:\/\/maxaeo.ai\/blog\/llms-txt-ai-visibility\/","title":{"rendered":"llms.txt for AI Visibility: Does It Actually Work, and How to Write One"},"content":{"rendered":"<p><strong>Short answer: as of mid-2026, llms.txt does almost nothing to improve AI search visibility for most sites \u2014 but it isn&#39;t pointless, and the nuance is where the budget decision lives.<\/strong> llms.txt is a proposed standard file you place at your site root to hand large language models a clean, structured map of your content. The pitch is seductive: feed AI a tidy summary, get recommended more often. Yet independent server-log data from hundreds of thousands of sites shows AI crawlers barely request the file. This guide separates the wishful thinking from what the logs actually prove, then gives you a working template and a way to test the file on your own domain.<\/p>\n<p>We track how brands surface across ChatGPT, Gemini, Perplexity, Claude, Copilot, and Google AI Overviews every day, so this is written from the measurement side: not &quot;here&#39;s a shiny new file,&quot; but &quot;here&#39;s what moves cited brand mentions and what doesn&#39;t.&quot;<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" style=\"max-width:100%;height:auto\" loading=\"lazy\"  src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/06\/1782474437826-13-37839-1.jpg\" alt=\"Diagram of an llms.txt file placed at a website root, mapping key pages for AI crawlers\"><\/figure>\n<h2>What is llms.txt?<\/h2>\n<p><strong>llms.txt is a plain-text Markdown file placed at your site root (<code>\/llms.txt<\/code>) that gives language models a curated, ad-free map of your most important content.<\/strong> Jeremy Howard of <a href=\"https:\/\/www.answer.ai\/posts\/2024-09-03-llmstxt.html\" target=\"_blank\" rel=\"noopener\">Answer.AI proposed it on September 3, 2024<\/a>. The reasoning: robots.txt tells crawlers what they <em>may<\/em> fetch, but nothing tells an LLM what a site is <em>about<\/em> or which pages matter most.<\/p>\n<p>The format is simple and human-readable. The only required element is an H1 with the site or project name. Everything else is optional but recommended: a short blockquote description, free-form detail, and curated link lists with one-line annotations. A companion <code>llms-full.txt<\/code> can hold the entire site as a single Markdown document for ingestion. The full spec lives at <a href=\"https:\/\/llmstxt.org\/\" target=\"_blank\" rel=\"noopener\">llmstxt.org<\/a>.<\/p>\n<p>Early adopters skew technical \u2014 Anthropic, Stripe, Cursor, Mintlify, Zapier, and FastHTML \u2014 which is a clue about who the file genuinely serves. (Mintlify auto-generates it across every docs site it hosts, which is how thousands of pages got one overnight.) That pattern matters when we decide who should bother.<\/p>\n<h2>Does llms.txt actually work for AI visibility?<\/h2>\n<p><strong>No \u2014 not in any measurable way for brand citations in AI search. Multiple 2026 studies converge on the same result: AI crawlers almost never fetch the file, and its presence shows no correlation with more AI mentions or rankings.<\/strong> The data is unusually consistent for such a hyped topic.<\/p>\n<p>Three sources worth putting side by side:<\/p>\n<ul>\n<li><strong>Ahrefs (May 2026):<\/strong> Across <a href=\"https:\/\/ahrefs.com\/blog\/llmstxt-study\/\" target=\"_blank\" rel=\"noopener\">137,210 domains in a bot-traffic study<\/a>, 97% of llms.txt files received zero requests of any kind. Of the files that did see traffic, most was non-AI tooling (SEO auditors, tech profilers, generic web crawlers). AI bots were only about 19% of requests \u2014 and within that slice, coding agents and training crawlers dominated, while the AI search\/assistant bots that actually generate citations made up roughly 2%.<\/li>\n<li><strong>Otterly.ai (90-day single-site test):<\/strong> Out of 62,100+ AI bot visits, <a href=\"https:\/\/otterly.ai\/blog\/the-llms-txt-experiment\/\" target=\"_blank\" rel=\"noopener\">only 84 reached <code>\/llms.txt<\/code><\/a> \u2014 about 0.1% of AI traffic, and roughly 3\u00d7 fewer than the site&#39;s average page (~265 visits). Otterly concluded llms.txt is not a significant driver of AI crawl behavior.<\/li>\n<li><strong>Google:<\/strong> John Mueller said no AI service has confirmed using the file and that <a href=\"https:\/\/www.searchenginejournal.com\/google-says-llms-txt-comparable-to-keywords-meta-tag\/544804\/\" target=\"_blank\" rel=\"noopener\">&quot;you can tell when you look at your server logs that they don&#39;t even check for it,&quot;<\/a> likening it to the long-dead keywords meta tag. Google has not announced support, and AI Overviews don&#39;t consult it.<\/li>\n<\/ul>\n<p>If you take one thing from this section: the file is rarely read, and &quot;rarely read&quot; cannot move citations. For a deeper breakdown of the citation numbers, see <a href=\"https:\/\/maxaeo.ai\/blog\/does-llms-txt-work\">Does llms.txt Work? Evidence From AI Citation Data<\/a>.<\/p>\n<h2>Why the &quot;it works&quot; claims and the server logs disagree<\/h2>\n<p><strong>The gap is real and explainable: vendors and platform blogs describe <em>intended<\/em> behavior, while server logs record <em>actual<\/em> behavior \u2014 and the two rarely match.<\/strong> You&#39;ll find confident posts claiming &quot;Perplexity retrieves llms.txt&quot; or &quot;Claude respects it.&quot; Much of that traces to support docs or roadmap language, not measured retrieval at inference time.<\/p>\n<p>The honest reconciliation in one table:<\/p>\n<table>\n<thead>\n<tr>\n<th>Platform<\/th>\n<th>What&#39;s claimed publicly<\/th>\n<th>What server logs show<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Google (Search \/ AI Overviews)<\/td>\n<td>Does not support it; &quot;not planning to&quot;<\/td>\n<td>No Googlebot or Google-Extended fetches of the file<\/td>\n<\/tr>\n<tr>\n<td>OpenAI (ChatGPT \/ SearchGPT)<\/td>\n<td>No official commitment to read it<\/td>\n<td>GPTBot (a training crawler) appears occasionally; OAI-SearchBot, the answer bot, barely does<\/td>\n<\/tr>\n<tr>\n<td>Anthropic (Claude)<\/td>\n<td>Publishes its own llms.txt; no inference-time guarantee<\/td>\n<td>Claude-Code (a coding agent) fetches docs files far more than any search bot<\/td>\n<\/tr>\n<tr>\n<td>Perplexity<\/td>\n<td>Sometimes cited as &quot;supporting&quot; it<\/td>\n<td>PerplexityBot requests to the file are negligible in logs<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The most important row is Anthropic&#39;s. In the Ahrefs data, the AI traffic that <em>did<\/em> reach llms.txt files came disproportionately from <strong>AI coding agents pulling developer documentation<\/strong> \u2014 Claude-Code and GPTBot were the top individual bots \u2014 not from the search-and-answer bots that decide whether your brand lands on a shortlist. That single distinction explains most of the confusion: llms.txt has a narrow, real use case, and it is not &quot;get recommended by ChatGPT.&quot;<\/p>\n<h2>When is llms.txt actually worth the effort?<\/h2>\n<p><strong>Worth it if you publish developer docs or API references that AI coding agents ingest, or if your platform generates the file automatically for near-zero cost. Skip it if you&#39;re a marketing site hoping it lifts brand mentions in AI answers \u2014 the evidence says it won&#39;t.<\/strong> Treat the decision as cost-versus-evidence, not an article of faith.<\/p>\n<p>A quick framework:<\/p>\n<p><strong>Keep or add llms.txt when:<\/strong><\/p>\n<ul>\n<li>You run technical docs, an API reference, or a dev tool. Agents like Claude-Code and Cursor genuinely fetch curated Markdown maps.<\/li>\n<li>Your docs platform (Mintlify and similar) auto-generates and auto-updates it. Free upside, no maintenance tax.<\/li>\n<li>You want a tidy, canonical content index for internal LLM\/RAG use.<\/li>\n<\/ul>\n<p><strong>Deprioritize or skip when:<\/strong><\/p>\n<ul>\n<li>You&#39;re a brand chasing AI Overviews, Perplexity, or ChatGPT citations. There&#39;s no measured payoff.<\/li>\n<li>Maintaining it by hand would steal time from fixing crawlability, entity facts, or earned mentions.<\/li>\n<li>You&#39;d be tempted to serve LLM-only content no human sees \u2014 a pattern that edges toward cloaking, which Google has openly criticized.<\/li>\n<\/ul>\n<p>The file is cheap scaffolding for the right site and a distraction for the wrong one. Most teams reading this are the wrong site for it \u2014 and that&#39;s fine, because the real levers are elsewhere.<\/p>\n<h2>How to write an llms.txt file (template included)<\/h2>\n<p><strong>To write a valid llms.txt, create a Markdown file at <code>\/llms.txt<\/code> with an H1 site name, a one-line blockquote description, and short curated link lists pointing only to pages AI can already read.<\/strong> Keep it small and current; a bloated or stale file is worse than none.<\/p>\n<p>Follow these steps:<\/p>\n<ol>\n<li><strong>Create the file<\/strong> at your site root so it resolves at <code>https:\/\/yourdomain.com\/llms.txt<\/code>.<\/li>\n<li><strong>Add an H1<\/strong> with your brand or project name (the only required element).<\/li>\n<li><strong>Write a one-sentence blockquote<\/strong> describing what you do, in plain language.<\/li>\n<li><strong>List 5\u201315 high-value pages<\/strong> in grouped sections (Docs, Key pages, Optional), each with a one-line annotation.<\/li>\n<li><strong>Link only to crawlable, canonical URLs<\/strong> \u2014 not JavaScript-rendered pages an AI bot can&#39;t read.<\/li>\n<li><strong>Set the file to <code>noindex<\/code><\/strong> if you don&#39;t want the raw file cluttering search results.<\/li>\n<\/ol>\n<h3>A minimal llms.txt template<\/h3>\n<pre><code class=\"language-markdown\"># Acme Analytics\n\n&gt; Acme Analytics is a product analytics platform for B2B SaaS teams,\n&gt; tracking activation, retention, and revenue in one place.\n\n## Docs\n- [Quickstart](https:\/\/acme.com\/docs\/quickstart): Set up in 10 minutes\n- [API reference](https:\/\/acme.com\/docs\/api): Full REST API\n- [Integrations](https:\/\/acme.com\/docs\/integrations): Supported data sources\n\n## Key pages\n- [Pricing](https:\/\/acme.com\/pricing): Plans and usage limits\n- [Security](https:\/\/acme.com\/security): SOC 2, GDPR, data handling\n\n## Optional\n- [Changelog](https:\/\/acme.com\/changelog): Recent releases\n<\/code><\/pre>\n<h3>Common mistakes that quietly waste the effort<\/h3>\n<ul>\n<li><strong>Dumping every URL.<\/strong> The point is a <em>curated<\/em> map; a sitemap-sized dump defeats it.<\/li>\n<li><strong>Letting it drift.<\/strong> A file that contradicts your live site teaches AI nothing useful.<\/li>\n<li><strong>Linking to content crawlers can&#39;t render.<\/strong> If the target page is client-side JavaScript, the link is dead weight.<\/li>\n<li><strong>Blocking the bots, then hoping llms.txt saves you.<\/strong> If your robots.txt blocks GPTBot or ClaudeBot, the file is moot \u2014 decide crawler access first.<\/li>\n<li><strong>Treating it as a substitute for readable HTML.<\/strong> It isn&#39;t. AI reads your pages, not your wishes about your pages.<\/li>\n<\/ul>\n<h2>What actually gets your brand recommended by AI<\/h2>\n<p><strong>If llms.txt isn&#39;t the lever, here&#39;s what is: crawlable HTML, clear entity facts, earned third-party citations, and answer-first content \u2014 the things AI systems demonstrably read and repeat.<\/strong> This is where the budget that almost went into a root-directory file should go instead.<\/p>\n<p>In day-to-day brand-mention tracking, the pages that get cited share a profile: they&#39;re directly readable in HTML, they state facts in clean passages, and they&#39;re reinforced by mentions on sites AI already trusts. None of that depends on llms.txt. The practical priorities:<\/p>\n<ul>\n<li><strong>Make sure AI can read you at all.<\/strong> Crawler access is the foundation \u2014 if bots can&#39;t fetch your pages, nothing else matters.<\/li>\n<li><strong>Publish unambiguous brand facts.<\/strong> Answer engines repeat entities they can parse: a consistent name, category, offerings, and proof points.<\/li>\n<li><strong>Earn off-site mentions.<\/strong> Reddit, G2, Wikipedia, and YouTube feed AI shortlists \u2014 see <a href=\"https:\/\/maxaeo.ai\/blog\/ai-recommends-competitors\">AI Recommends Competitors: Why It Happens and How to Win Back AI Shortlists<\/a>.<\/li>\n<li><strong>Write in quotable passages.<\/strong> Lead each section with a direct, 40\u201360 word answer so models can lift it cleanly.<\/li>\n<\/ul>\n<p>Do these four and your <strong>answer engine optimization<\/strong> and <strong>generative engine optimization<\/strong> efforts compound. Do llms.txt instead, and you&#39;ve optimized a file almost nobody fetches.<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" style=\"max-width:100%;height:auto\" loading=\"lazy\"  src=\"https:\/\/maxaeo.ai\/blog\/wp-content\/uploads\/2026\/06\/1782474437826-13-37839-2.jpg\" alt=\"Server-log chart comparing AI crawler requests to llms.txt versus regular HTML pages\"><\/figure>\n<h2>How to test whether llms.txt does anything for your site<\/h2>\n<p><strong>Don&#39;t take anyone&#39;s word \u2014 including ours. Run a two-part test: check your server logs for AI-bot requests to <code>\/llms.txt<\/code>, and baseline your AI citation rate before and after adding the file.<\/strong> That&#39;s the data-backed way to settle it for your specific domain.<\/p>\n<p>A simple protocol:<\/p>\n<ol>\n<li><strong>Pull 30\u201390 days of server or CDN logs.<\/strong> Filter for AI user agents (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended) and count hits to <code>\/llms.txt<\/code> versus your top content pages. Most sites will see near-zero on the file.<\/li>\n<li><strong>Baseline your citations first.<\/strong> Record how often your brand is mentioned and cited across AI platforms before changing anything. A structured starting point is the <a href=\"https:\/\/maxaeo.ai\/blog\/geo-audit\">No-Code GEO Audit<\/a>.<\/li>\n<li><strong>Add the file, then wait and compare.<\/strong> Watch for any shift in citation frequency, controlling for content and PR changes so you don&#39;t credit llms.txt for unrelated wins.<\/li>\n<li><strong>Track continuously, not once.<\/strong> AI answers are noisy; a single snapshot lies. Sound method matters \u2014 see <a href=\"https:\/\/maxaeo.ai\/blog\/ai-search-monitoring-methodology\">AI Search Monitoring Methodology<\/a>.<\/li>\n<\/ol>\n<p>This is exactly the before\/after an <strong>ai visibility tool<\/strong> is built to measure. Daily <strong>ai search monitoring<\/strong> turns &quot;I think it helped&quot; into a defensible number you can put in front of a budget owner \u2014 which is the whole point of <strong>llm brand tracking<\/strong> in the first place.<\/p>\n<h2>The bottom line on llms.txt<\/h2>\n<p>llms.txt is a thoughtful idea that the market got ahead of. For developer docs consumed by AI agents, it&#39;s a low-cost nicety worth keeping if it&#39;s auto-generated. For everyone chasing <strong>brand mentions in ChatGPT<\/strong> and AI Overviews, the honest read from server logs and controlled experiments is the same: it doesn&#39;t move the needle. Spend the effort on crawlability, entity clarity, earned citations, and measurement \u2014 then verify with your own data rather than the hype.<\/p>\n<h2>Frequently asked questions<\/h2>\n<p><strong>Is llms.txt the same as robots.txt?<\/strong><br \/>\nNo. robots.txt controls <em>access<\/em> \u2014 which crawlers may fetch which paths. llms.txt is a <em>content suggestion<\/em> \u2014 a curated map of pages for LLMs. robots.txt is widely honored; llms.txt is widely ignored.<\/p>\n<p><strong>Does Google use llms.txt?<\/strong><br \/>\nNo. Google representatives have said Search does not support it and isn&#39;t planning to, and likened it to the deprecated keywords meta tag. AI Overviews don&#39;t consult it.<\/p>\n<p><strong>Should I add llms.txt anyway?<\/strong><br \/>\nOnly if it&#39;s auto-generated for free, or you publish developer docs that AI coding agents ingest. For a marketing site hoping to raise AI citations, the measured payoff is effectively zero.<\/p>\n<p><strong>Will llms.txt get my brand into ChatGPT answers?<\/strong><br \/>\nNot on its own. ChatGPT&#39;s search bots rarely fetch the file. Getting recommended depends on readable content, clear entity facts, and trusted off-site mentions \u2014 covered in <a href=\"https:\/\/maxaeo.ai\/blog\/why-ai-search-engines-cite-competitor-pages-instead-of-yours\">Why AI Search Engines Cite Competitor Pages Instead of Yours<\/a>.<\/p>\n<p><strong>What&#39;s the difference between llms.txt and llms-full.txt?<\/strong><br \/>\n<code>llms.txt<\/code> is a short, curated index of key links. <code>llms-full.txt<\/code> is an optional companion containing your full site content as one Markdown document for ingestion. Both live at the site root; neither is widely fetched today.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Server logs show AI crawlers rarely fetch llms.txt. Here&#8217;s what the 2026 evidence really says, when it actually helps, and a template to copy. Track your AI visibility.<\/p>\n","protected":false},"author":1,"featured_media":802,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-804","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts\/804","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/comments?post=804"}],"version-history":[{"count":0,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/posts\/804\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/media\/802"}],"wp:attachment":[{"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/media?parent=804"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/categories?post=804"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/maxaeo.ai\/blog\/wp-json\/wp\/v2\/tags?post=804"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}