Over the last year a new line appeared on many SEO checklists: "create an llms.txt file". Does it actually change anything, or is it a fashion?

Short answer: it does no harm, and its benefit is limited and uncertain. The long answer follows.

What is llms.txt?

llms.txt is a draft standard proposed by Jeremy Howard of Answer.AI in September 2024. You place a Markdown file at your site root (example.com/llms.txt) that introduces the site briefly and lists the most important pages under headings, each with a short description.

The aim is to let a language model start from one clean summary instead of crawling dozens of pages, menus and ad slots. robots.txt says where not to go, sitemap.xml says which pages exist, and llms.txt tries to say what the site is about and what matters most.

Who uses it?

Honesty matters here. The number of sites publishing the file grew quickly, especially among software companies with documentation. But at the time of writing there is no clear, official commitment from the major AI providers that they systematically read it when generating answers, and Google has said it does not use such a file for search ranking.

So llms.txt is not a ranking factor today. It is preparation and a signal of good intent.

Adding llms.txt takes five minutes. Mistaking it for a GEO strategy is an expensive error.

Why add one, then?

• It costs almost nothing. Write it once, update it when an important page is added.
• It forces clarity. Describing your site in a paragraph and choosing your ten most important pages is a good exercise in sharpening your own message.
• Some tools read it. Some coding assistants and agent tools look for it when reading a site.

Settings that matter more

• AI crawlers in robots.txt. You may have blocked GPTBot (OpenAI), ClaudeBot (Anthropic) or PerplexityBot without knowing it. Some security plugins and CDN settings do this by default. A model cannot read a site it is blocked from.
• Google-Extended. This controls whether Google's models may train on your content; it does not affect your visibility in search results. Do not confuse the two.
• Content in the HTML. If key information only loads through JavaScript, some crawlers never see it.
• Structured data and clear writing. The foundations covered in our AI Overviews and GEO pieces decide far more than any file.

What a good llms.txt looks like

An H1 with the site name, a one- or two-sentence summary, then sections such as Services, Guides and Contact, each with described links. Plain information rather than marketing copy. You can see ours at litmusuniverse.com/llms.txt.

Frequently asked questions

What is the difference between llms.txt and llms-full.txt?

Under the proposal, llms.txt holds a short summary and a described list of key pages, while llms-full.txt gathers the site's important content in a single text file. The full version mostly makes sense for software documentation; for most business sites the short one is enough.

Should I block AI crawlers?

It is a business decision. If you do not want your content used for model training, you can block the relevant crawlers. But if you want to appear in AI answers, blocking the crawlers used for search and answers reduces your visibility. Providers often use different crawler names for training and search, so decide for each separately.

Where should the file go, and how often should I update it?

At your domain root, alongside robots.txt. Update it when you add a service page or major guide, or when contact details change. If your publishing is automated, have the file update automatically too.

If you want your technical AI settings checked, see our search and AI visibility services.