Skip to content
No. 12 · llms.txt · AI crawlers / Technical

What is llms.txt?

llms.txt is a plain text file at the root of your website that gives AI crawlers a structured summary of what your site is about and which pages matter most. It is a proposed standard, not a ratified one: cheap to add, easy to maintain, and honest about its limits.

Section 01

What is llms.txt and where did it come from?

llms.txt is a plain text file placed at the root of your domain (yoursite.com/llms.txt) that provides a structured summary of your website for large language models. The standard was proposed by Jeremy Howard in late 2024 and is documented at llmstxt.org. Unlike robots.txt, which controls access, llms.txt is a guide: it tells AI crawlers what your site is about, which pages are most important, and how your content is organised.

The file uses a simple Markdown format with headings, descriptions, and categorised links. It is lightweight, human-readable, and trivial to maintain. You can see a live example on this site at brainbuddy.ai/llms.txt.

Section 02

What does llms.txt actually do for AI visibility?

It solves a signal-to-noise problem, with honest caveats. Your website might have hundreds of pages, but only a fraction represent your core expertise and services. Without guidance, a model might index your cookie policy with the same weight as your flagship service page. llms.txt acts as a curated table of contents: here are the pages that matter, here is what each covers, here is how they relate.

The caveats: adoption is uneven, Google has not confirmed that AI Overviews read the file, and nobody has published convincing evidence of a direct ranking effect. We implement it anyway, because it costs minutes, carries little risk, and gives any crawler that does read it an accurate picture of your site. It is one small piece of generative engine optimisation, never a substitute for schema markup and answer-shaped content.

Section 03

How do you create an llms.txt file?

Create a plain text file named llms.txt and place it at the root of your domain so it is accessible at yoursite.com/llms.txt. Start with a heading and a one-paragraph description of your business, then list your most important pages with short, plain-language descriptions. There is no complex syntax and no validation step. If you would rather not write it by hand, our free llms.txt generator builds one from your site.

  • Place the file at your domain root: yoursite.com/llms.txt
  • Start with a heading and one-paragraph site description.
  • List your most important pages with brief, clear descriptions.
  • Group pages by category if your site covers multiple topics or services.
  • Keep descriptions factual and specific, not promotional.
  • Update the file whenever you add or remove significant content.
Section 04

What goes in your llms.txt, and what stays out?

Include the pages that represent your core expertise: homepage, main service pages, about page, and your strongest educational content, each with a concise, factual description. Leave out privacy policies, login pages, and thin content. The goal is signal density. A focused llms.txt with twenty well-described pages will outperform a bloated one with two hundred undifferentiated entries. If you have a blog, include only the cornerstone articles that demonstrate genuine expertise, not every post you have ever published.

  • Include: homepage, core service pages, about page, key guides.
  • Include: case studies and results pages that demonstrate expertise.
  • Exclude: privacy policy, terms, cookie notices, login pages.
  • Exclude: thin content, duplicate pages, tag archives, pagination pages.
  • Keep descriptions under two sentences, factual and specific.

We build and maintain llms.txt files as one of the change types our agent ships, with a senior SEO approving every change. See how that works within our GEO services.

Q

Frequently asked.

Is llms.txt an official web standard?

It is a proposed standard, documented at llmstxt.org, not yet ratified by a standards body like the W3C. Adoption varies across AI platforms. Implementing it is low effort and low risk: the file is plain text, takes minutes to create, and does not interfere with anything else on your site.

What is the difference between robots.txt and llms.txt?

robots.txt controls access: it tells crawlers which pages they may or may not visit. llms.txt provides context: it tells AI models what your website is about, which pages matter most, and how your content is organised. They serve different purposes and work together.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a concise summary that lists your key pages with brief descriptions. llms-full.txt is an extended version that includes the full text content of your important pages, giving AI models complete context without crawling each page individually. Many sites publish both: the summary for orientation, the full version for depth.

Does llms.txt actually help with AI search?

Honestly: it is too early for anyone to prove a direct ranking effect, and Google has not confirmed that AI Overviews use the file. What it definitely does is give any crawler that reads it a clear, curated summary of your site at almost no cost. We treat it as cheap clarity, not magic, and implement it alongside schema markup and answer-shaped content.

Do I need llms.txt if I already have a sitemap?

They do different jobs. An XML sitemap lists URLs so crawlers can discover pages. llms.txt explains what those pages contain, what your business does, and which content represents you best. A sitemap says these pages exist. llms.txt says here is what they are about and why they matter.

How often should I update my llms.txt file?

Update it whenever you add or remove a significant page. For most businesses a quarterly review is enough. If you publish frequently, check it monthly so your most important new pages are included and outdated entries are removed. A stale llms.txt quietly misdescribes your site.

We build and maintain your llms.txtas part of every engagement.

Start with the free AI visibility audit. If you engage us, llms.txt implementation is included in your onboarding alongside schema markup, entity consolidation, and content restructuring.