Do You Need an llms.txt File? The Honest Answer for 2026

Short answer: no, not as a strategy, and not as a line item on an agency invoice. Google has stated it does not use llms.txt and does not plan to. No major AI provider has committed to reading it in production retrieval. Server-log analyses through 2026 find the crawlers that actually drive citations almost never request it.

Longer answer: we still ship one on our own sites, and we would still ship one on yours, because it costs an afternoon and there is a real second-order benefit that has nothing to do with crawlers. Both halves of that are worth explaining, because "llms.txt" has become the single most oversold deliverable in AI search.

What llms.txt actually is

A proposed convention, introduced in late 2024, for a markdown file at the root of your domain that gives a language model a clean, curated index of your site: what the site is, what the important pages are, and where the documentation lives.

The reasoning was sound. An HTML page is mostly not content. It is navigation, cookie banners, scripts, footers, and markup. A model that has to parse all that to find three useful paragraphs is wasting context on chrome. A markdown index hands it the signal directly.

Two files are usual: /llms.txt, a short curated index, and /llms-full.txt, an expanded version with the detail. It reads like a sitemap written for a reader rather than a parser.

It is a genuinely good idea. That is separate from whether anything consumes it.

What actually happened

Google said no. When Google published its guide to optimizing for generative AI features in May 2026, it stated directly that llms.txt files, special machine-readable formats, and AI-specific content are not needed to appear in its generative features. Google search advocates have separately compared it to the keywords meta tag, a file that sites maintained for years after search engines stopped reading it.

No major provider committed. As of mid-2026, none of OpenAI, Anthropic, Google, or Meta has published a commitment to consuming llms.txt in production retrieval. Some smaller tools and documentation platforms do read it. The engines that generate the answers your buyers see largely do not.

The logs are the clincher. Analyses of server logs published through 2026 consistently find requests to /llms.txt from citation-driving user agents to be a rounding error against requests for ordinary HTML. The crawlers come. They fetch pages. They mostly do not ask for the file.

This is the part the vendors selling llms.txt packages leave out, and it is checkable on your own server in about five minutes:

grep -i "llms.txt" /path/to/access.log | wc -l
grep -icE "gptbot|claudebot|perplexitybot|oai-searchbot" /path/to/access.log

If the second number is large and the first is near zero, you have your answer for your own domain.

Why we ship one anyway

We have written approvingly about llms.txt before, in building products designed to be cited by LLMs. The evidence moved, so the position moved with it. Here is where it landed.

It costs an afternoon. Ours is generated from the same content layer that builds the site, so it stays current without anyone maintaining it. A one-time build cost against a possible future payoff is a reasonable bet when the cost is that low.

Standards sometimes arrive late. Robots.txt was a convention before it was universal. If any major provider starts honouring llms.txt, sites that already have a good one are ahead, and sites that do not will need the same afternoon then instead of now.

The real benefit is the writing, not the file. To produce a good llms.txt you have to state, in one paragraph, what your company is, what it sells, who it is for, and what each important page does. Most companies cannot do this without an argument. That paragraph is the single most reusable asset in AI search: it goes on your about page, your LinkedIn, your Crunchbase, your press kit, and every other profile in your entity graph. Consistency across those sources is what makes a model confident enough to cite you.

So the file is a cheap by-product of an exercise that is genuinely valuable. What it is not is the reason anyone will cite you.

Where we draw the line: if a proposal leads with llms.txt, or prices it as a significant deliverable, that tells you what the rest of the engagement is worth. We cover the rest of that pattern in GEO mistakes and how to spot a fake GEO agency.

How to write one properly, if you ship one

Assuming you spend the afternoon, spend it well.

Format. Plain markdown at /llms.txt, served as text/plain or text/markdown, HTTP 200, not behind auth or a bot wall. An H1 with the site name, a blockquote with the one-paragraph entity description, then linked sections.

Content, in order of usefulness:

1. The one-paragraph entity description: what you are, what you sell, who for, where.
2. The pages that answer questions: pricing, process, comparisons, documentation.
3. Anything with hard facts: rates, timelines, specifications, coverage areas.
4. Contact and canonical profile links.

What to leave out. Marketing copy, taglines, everything that would read as filler if a stranger quoted it. The file exists to be quoted, and a paragraph that says nothing gets quoted as nothing.

Common errors we see. Serving it as HTML with a 200 that is actually your 404 page. Letting it drift a year out of date. Writing it in a different voice from the site, so the facts contradict the pages. Contradiction is worse than absence: a model that finds two different founding years for you hedges on both.

Test it:

curl -sI https://yoursite.com/llms.txt | head -1
curl -s https://yoursite.com/llms.txt | head -40

What to do with the rest of the week

The afternoon is fine. The mistake is spending a month on it. In rough order of what actually moves retrieval:

1. Confirm the crawlers can reach you. Bot protection and WAF rules return 403 to GPTBot and ClaudeBot far more often than anyone expects, and this single fault explains most cases of zero AI visibility.
2. Get into Bing's index. ChatGPT and Copilot retrieve from it. Verifying Bing Webmaster Tools takes ten minutes and remains the highest-return task in the discipline.
3. Restructure pages into passages. Question-shaped headings, answer in the first sentence, each answer self-contained in 60 to 120 words. Retrieval works on chunks, not documents.
4. Put the facts in the text. Prices, timelines, coverage, specifications. Not in an image, not behind a tab that never opens for a bot.
5. Make the entity consistent off-site. Same paragraph, same founding year, same description everywhere you appear. Most AI citations point at earned media rather than your own domain, so the off-site graph is where confidence comes from.

The full sequence, with the measurement to go with it, is in how to rank in ChatGPT, Gemini, Perplexity and Claude. The definitional groundwork is in SEO vs GEO.

The one-sentence version

Ship llms.txt if it costs you an afternoon and nothing more, write it as though it will be quoted verbatim, then go and fix the five things above, which are what a generative engine actually reads.

We build this into websites as part of the work rather than as a separate product. The methodology, including what we run on our own tools, is documented on our citations page. To scope a project, start here.

FAQ

Does ChatGPT read llms.txt?

There is no published commitment from OpenAI that llms.txt is used in production retrieval, and server logs show ChatGPT's crawlers requesting ordinary HTML pages rather than the file. Treat any claim that ChatGPT reads it as unverified.

Does Google use llms.txt?

No. Google's May 2026 guidance on its generative AI features states that llms.txt files, special machine-readable formats, and AI-specific content are not required, and Google representatives have compared the file to the long-obsolete keywords meta tag.

Is llms.txt the same as robots.txt?

No, and they solve opposite problems. Robots.txt controls what crawlers are permitted to fetch and is universally honoured. llms.txt proposes a curated content index for language models and is not yet honoured by the major engines. You need a correct robots.txt. llms.txt is optional.

Will llms.txt hurt my SEO?

No. It is an inert extra file. The only real risk is opportunity cost, spending time on it that belongs on crawler access, indexing, page structure, and off-site entity consistency.

Should I pay an agency to create llms.txt?

Not as a deliverable of any significance. It is roughly an afternoon of work, most of which is writing one paragraph you need anyway. If a proposal features it prominently, treat that as a signal about the depth of the rest of the engagement.

What is llms-full.txt?

The expanded companion file: the same curated index with the detail filled in, FAQs, specifications, pricing, technical documentation. Same standing as llms.txt, optional and largely unread by the major engines, same second-order benefit of forcing you to write your facts down in one place.

If nothing reads it, why do so many sites have one?

Because it is cheap, it looks diligent, and for about eighteen months it was widely recommended before the evidence caught up. That is also why it is easy to sell. A file that takes an afternoon and cannot be disproved as a long-term bet is an ideal thing to put on an invoice.

NOW Media is a Bangalore creative studio founded in 2019, a brand of Bleep Design Private Limited. We ship llms.txt on our own sites and tell clients honestly what it is worth.