llms.txt is a plain-text Markdown file placed at the root of a website (yourdomain.com/llms.txt) that gives large language models a curated map of the site's most important pages. The spec was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, on September 3, 2024, and published at llmstxt.org. As of mid-2026, roughly 5% to 10% of websites publish one, including Anthropic, Stripe, Vercel, Cloudflare, Mintlify, Cursor, Supabase, and Coinbase. The honest verdict on whether it works: major AI search engines (ChatGPT, Gemini, Perplexity's answer surfaces, Google AI Overviews) do not measurably read llms.txt in their production retrieval pipelines as of Q2 2026, and Google has publicly said it will not support the spec. Where llms.txt does work is in IDE coding agents (Cursor, Claude Code, GitHub Copilot, Continue, Cline) and MCP integrations, which fetch the file for real-time documentation retrieval. The strategic read for site owners: ship an llms.txt because the cost is near-zero and the forward-optionality is real, but do not expect a citation lift in AI search today.

The llms.txt conversation matters in 2026 because the spec sits at the intersection of two things that are moving fast: LLM context-window economics (models pay a real cost to parse messy HTML) and the emerging agentic web (IDE agents, browsing agents, and MCP-connected AI hosts increasingly need machine-readable entry points into websites). The debate about whether llms.txt is "the new robots.txt" or "the new keywords meta tag" oversimplifies the actual state of the spec. Neither framing captures where the file is producing value today and where it is not. This piece walks through what llms.txt is, what the specification requires, where the adoption data sits, whether major AI engines read the file, and what to put in yours if you decide to ship one.

What Is llms.txt

llms.txt is a plain-text Markdown file placed at the root of a website that provides AI systems with a curated, human-readable roadmap to the site's most important content. The file is intended for large language models and AI agents that are trying to make sense of a site without parsing every HTML page. Think of it as a Wikipedia-style summary written in machine-friendly Markdown, hand-curated to point AI systems at the site's canonical, authoritative pages.

The distinction between llms.txt and adjacent web standards is precise. robots.txt tells crawlers where they cannot go. sitemap.xml tells search engines the complete list of URLs on a site. schema.org markup tells parsers about specific entities within a page. llms.txt sits in a different position: it is a site-level editorial map that tells language models which pages the site owner considers most important and how the content is organized. The file does not block, restrict, or hide anything. It curates.

The spec was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, in a post published on September 3, 2024. The core insight that motivated the proposal was economic: LLM context windows remain smaller than most websites, and HTML pages are full of navigation, ads, JavaScript, and boilerplate that consumes tokens before the model reaches the content that matters. A curated Markdown entry point cuts through the noise and lets models spend context on the parts of the site that answer the query.

What the llms.txt Specification Requires

The specification is intentionally minimal and lives at llmstxt.org, which hosts the canonical llms.txt example for reference. A valid llms.txt file has four required elements and one optional element.

• H1 heading with the brand or site name. This must be the first element in the file. It gives the model an immediate identifier for what the site is

• Blockquote with a one-to-three sentence summary. Written in third person, describing what the site covers and who it serves. This blockquote is the highest-signal element in the file and is often treated by parsers as the equivalent of a system prompt or grounding statement

• H2 section headings grouping related pages by topic or purpose. Common groupings include core pages, documentation, product references, policies, and research

• Annotated links inside each H2 section, with a short one-sentence description of each linked page written for a reader who knows nothing about the site

• Optional H2 section (literally named "Optional") containing links to less critical pages that models can skip when context budget is constrained

The companion format is llms-full.txt, which contains the full Markdown content of every page referenced in llms.txt, concatenated into a single file. This is the format Anthropic, Mintlify, Vercel, and LangGraph publish alongside their llms.txt. The purpose is to let agents that want deep site ingestion fetch everything in one request rather than following 20 individual URLs. Both files together form what most sophisticated implementations ship: the index for orientation and the full-text dump for deep ingestion.

Where llms.txt Adoption Stands in 2026

The adoption data through Q2 2026 is consistent across multiple independent studies. SE Ranking's November 2025 crawl of 300,000 domains found a 10.13% adoption rate. A separate May 2026 crawl of the Tranco Top 10,000 found 5.86%. Limy's analysis of over 500 million LLM bot visits across a 90-day window in early 2026 found the file present at similar rates. The trajectory is upward but not steep: adoption is concentrated in developer-facing SaaS, documentation platforms, and technology brands, with mainstream enterprise adoption lagging.

The publisher list has a clear shape. Anthropic ships llms.txt for its documentation. Stripe, Vercel, Cloudflare, and Coinbase publish one. Mintlify, the documentation hosting platform, auto-generates llms.txt for every hosted docs site (which is what triggered the adoption inflection in November 2024). Cursor, Supabase, LangGraph, Pinecone, and most modern developer-tool APIs publish one. Yoast built an llms.txt generator into its WordPress plugin. Maryland.gov became the first US state to publish an llms.txt at a government domain.

The mainstream adoption lag is the more interesting signal. Enterprise adopters (large e-commerce brands, established SaaS with distributed content stacks, publishers, consumer brands) have been slower to ship llms.txt, primarily because the compliance and legal review cycles have not yet produced a formal position on whether the file introduces risk. The pattern matches early robots.txt adoption in the late 1990s: technical adopters lead, mainstream adoption follows after standards bodies formalize.

Do Major AI Engines Read llms.txt

This is the question the entire spec discussion turns on, and the honest answer through Q2 2026 is: mostly no, with narrow exceptions.

The negative evidence is consistent. In July 2025, Google's Gary Illyes confirmed publicly that Google does not support llms.txt and is not planning to. John Mueller, also at Google, compared the file to the discredited keywords meta tag: a signal that the evaluated party controls itself, which is by definition unreliable to the evaluator. OpenAI's documented recommendation for crawler control is robots.txt, not llms.txt. Server log analysis across companies that have implemented llms.txt consistently shows that GPTBot occasionally fetches the file but rarely, while ClaudeBot, Google-Extended, and PerplexityBot effectively do not request it at meaningful volume. Limy's analysis of over 515 million LLM bot traffic events found only 408 that targeted llms.txt directly across all major search and answer bots.

The positive evidence is narrower and more specific. Anthropic has publicly confirmed some support for llms.txt in Claude's ecosystem. Perplexity has said it retrieves llms.txt to help prioritize pages. The mature use case, where llms.txt is producing measurable value today, is developer tooling. AI coding assistants (Cursor, Claude Code, GitHub Copilot, Continue, Cline) fetch llms.txt when developers ask them to work with a documented API, and a well-curated file is the difference between the assistant generating working integration code and hallucinating an endpoint that does not exist. This is the context Jeremy Howard originally proposed the spec for, and it is the context where the file demonstrably works.

The SE Ranking analysis of AI citation outcomes across 300,000 domains found no statistical correlation between publishing llms.txt and improved performance in AI search results. That finding is worth stating clearly: as of mid-2026, publishing llms.txt does not measurably increase the site's citation rate in ChatGPT, Gemini, Perplexity's answer surfaces, or Google AI Overviews. It may in the future. It does not today.

Where llms.txt Works Today: IDE Agents and MCP

The context where llms.txt is producing measurable value in 2026 is agent-facing infrastructure rather than search-facing infrastructure. Three specific use cases are shipping value today.

• IDE coding agents. Cursor, Claude Code, GitHub Copilot, Continue, and Cline all fetch llms.txt when developers reference a product or API in their working session. The file lets the agent orient itself to the site's documentation structure, retrieve the relevant reference pages, and generate accurate integration code. For any product with a developer-tool audience, llms.txt is closer to a developer-experience requirement than an optional signal

• MCP integrations. Model Context Protocol servers increasingly use llms.txt as a discovery mechanism for the resources they can expose to AI hosts. A well-formed llms.txt gives an MCP server a clean starting point for what the site publishes and how it is organized. This category is small but growing rapidly as MCP adoption scales

• Agentic commerce and vendor research. As browsing agents start acting on behalf of users ("find me B2B SaaS agencies in Croatia that specialize in AEO"), the brands that publish llms.txt pointing agents to canonical service pages, pricing information, and case studies are the brands agents can transact with. Most enterprise sites have not yet shipped agent-readable llms.txt files, which is the clearest provider gap in the category today

The common thread across the three use cases: llms.txt works where the AI system is actively fetching from the site in real time to answer a specific query. It does not work (yet) where the AI system is retrieving from a pre-indexed corpus to answer a general search query. That distinction is the reason the spec's value is currently concentrated in developer tooling and agentic use cases rather than in search citation lift.

What to Put in Your llms.txt

If the strategic decision is to ship an llms.txt (which is the right call for most sites given the near-zero cost and the forward-optionality), the implementation choices worth naming are practical.

• Keep the file short. Well-structured llms.txt files are typically under 5 kilobytes. A file with 200 URLs defeats the purpose of curation. The point is to point AI systems at the site's most important pages, not to reproduce the sitemap

• Lead with a clear brand summary. The blockquote is the highest-signal element in the file. Write it in third person, one to three sentences, describing what the site covers and who it serves. Do not use marketing copy; use plain descriptive language that an agent can extract

• Organize sections by function, not chronology. Group pages by what they do (documentation, product reference, pricing, case studies, policies) rather than by when they were published. Function-based groupings match how agents retrieve

• Write link descriptions for standalone use. Each link should have a one-sentence description that stands on its own. Agents sometimes use the description as context without ever fetching the linked page. Include concrete facts in the description when possible ("Starter tier at $500/mo, Growth at $2,000/mo, Enterprise custom") rather than generic phrasing ("affordable pricing for every team")

• Coordinate with robots.txt. The two files serve different purposes but need to be consistent. Verify that robots.txt permits AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot) to reach the pages linked in llms.txt. Contradictions between the two files are a common implementation mistake

• Ship llms-full.txt if content is documentation-heavy. For SaaS products, developer tools, and API products, publishing the concatenated full-content version alongside the index is worth the extra work. For content-heavy publisher sites or product catalogs, the trade-off between file size and value is less favorable

The most common implementation mistake in 2026 is treating llms.txt as a sitemap. A file with 200 links defeats the purpose. The second most common is using template copy without customization; a generic llms.txt signals low investment to the AI systems that read the file. The third is contradicting robots.txt through file-level edits that were not coordinated across the site's technical stack.

How llms.txt Fits Into a Broader AI Search Strategy

llms.txt is one technical signal in a broader AI search optimization strategy. It addresses discovery (helping AI systems find and understand a site's structure) but does not address content structure, technical signals, entity authority, or citation engineering, which are the layers that measurably move AI citation rates today. Sites that ship a well-curated llms.txt while ignoring the deeper AEO work are optimizing the wrong layer. Sites that build strong AEO fundamentals and add llms.txt on top are covering the full stack.

For the argument on why LLM optimization matters as a discipline distinct from traditional SEO, see RZLT's POV on LLM optimization and why it matters. For the operational playbook on how to earn brand visibility in AI search across ChatGPT, Gemini, Perplexity, and how llm seo, chatgpt seo, and perplexity seo differ from traditional Google-first optimization, see RZLT's guide to LLM visibility and getting cited by ChatGPT, Gemini, and Perplexity. For the tactical breakdown on content optimization specifically for Google AI Overviews, see RZLT's practitioner guide on optimizing content for Google AI Overviews in 2026. For the connectivity layer that lets AI hosts and MCP servers read from site infrastructure programmatically, see RZLT's breakdown of MCP for marketing.

The strategic read for site owners in 2026: ship an llms.txt because the cost is measured in hours and the forward-optionality is real. Do not expect a citation lift in AI search this quarter. Treat the file as forward compatibility with the agent web and as a genuine developer-experience improvement for any product where IDE coding agents are part of the audience. The teams that get this right build the file into their release pipeline, keep it current, and coordinate it with the rest of their AI search stack.