llms.txt is a plain text file placed in the root directory of a website (at yourdomain.com/llms.txt) that provides AI search engines and large language models with a structured summary of the website's content, purpose, and key pages. It was proposed by Jeremy Howard (co-founder of fast.ai) as a standard for helping AI models understand websites more efficiently. Think of it as a 'robots.txt for AI' that tells AI models what your site is about rather than what they can or cannot crawl.

What is the difference between robots.txt and llms.txt?

robots.txt tells search engine crawlers which pages they are allowed or not allowed to crawl. It controls access. llms.txt tells AI models what your website is about, what topics it covers, and which pages are most important. It provides context. robots.txt says 'you can or cannot access this page.' llms.txt says 'here is what this website contains and why it matters.' Both files serve different purposes and should be used together for comprehensive AI search optimisation.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a concise summary file that lists your key pages with brief descriptions (typically 1-5 KB). llms-full.txt is an extended version that includes the full text content of your important pages (can be 50-500+ KB). llms.txt gives AI models a quick overview of your site structure. llms-full.txt gives AI models complete content without needing to crawl each page individually. Most websites should have both: llms.txt for overview and llms-full.txt for comprehensive coverage.

How to make an llms.txt file?

To create an llms.txt file: 1) Create a plain text file named llms.txt. 2) Start with a Markdown H1 heading with your business name. 3) Add a description of what your business does. 4) List your key pages using Markdown links with descriptions. 5) Include sections for services, topics, and FAQs. 6) Upload to your website root so it is accessible at yourdomain.com/llms.txt. Brain Buddy AI's tools can generate llms.txt files automatically from your website content.

Does llms.txt actually help with AI search?

llms.txt is an emerging standard that is not yet universally adopted by all AI platforms, but it provides a clear signal to AI crawlers about your site's content and structure. Websites that implement llms.txt alongside comprehensive schema markup, structured content, and strong entity signals create multiple pathways for AI models to understand and cite their content. As AI search adoption grows (527% YoY according to Semrush), early adoption of standards like llms.txt builds a foundation for future visibility.

Do I need llms.txt if I already have a sitemap?

Yes. An XML sitemap lists URLs for search engine crawlers to discover pages. llms.txt provides context about what those pages contain, what your business does, and how the content relates to specific topics. A sitemap says 'these pages exist.' llms.txt says 'here is what these pages are about and why they matter.' They serve complementary purposes. For comprehensive AI search optimisation, use both alongside schema markup.

Which AI platforms support llms.txt?

llms.txt is an open community standard proposed by Jeremy Howard, not an official requirement from any specific AI platform. However, AI crawlers from various platforms are increasingly checking for llms.txt as part of their content discovery process. The specification is documented at llmstxt.org. Adoption is growing as more websites and AI tools recognise the value of providing structured content summaries for AI consumption.

No. 12 / llms.txt

llms.txt / AI Crawlers / Technical

The file that tells AI crawlers where to look.

llms.txt is a proposed web standard that gives large language models a structured guide to the most important content on your website. Think of it as robots.txt for the AI era: instead of telling crawlers what to avoid, it tells them what to prioritise. Here is what it is, why it matters, and how to implement it.

Section 01

What llms.txt is and where it came from.

llms.txt is a plain text file placed at the root of your domain (yoursite.com/llms.txt) that provides a structured summary of your website for large language models. The standard was proposed by Jeremy Howard in late 2024 as a practical solution to a growing problem: AI models struggle to efficiently parse and prioritise the vast amount of content on a typical website. Unlike robots.txt, which controls access, llms.txt is a guide. It tells AI crawlers what your site is about, what your most important pages are, and how your content is organised. The file uses a simple Markdown-based format with headings, descriptions, and categorised links. It is designed to be lightweight, human-readable, and trivial to maintain.

Section 02

Why llms.txt matters for AI visibility.

When an AI model retrieves information about your business, it faces a signal-to-noise problem. Your website might have hundreds or thousands of pages, but only a fraction of them represent your core expertise, services, and value proposition. Without guidance, the model might index your cookie policy with the same weight as your flagship service page. llms.txt solves this by acting as a curated table of contents. It tells the model: here are the pages that matter, here is what each one covers, and here is how they relate to each other. Early evidence suggests that sites with a well-structured llms.txt file see improved entity resolution in AI outputs and more accurate citations. It is a small technical investment with a potentially outsized impact on how AI engines understand and represent your business.

Section 03

How to implement llms.txt on your site.

Implementation is straightforward. Create a plain text file named llms.txt and place it at the root of your domain so it is accessible at yoursite.com/llms.txt. The file uses a simple structure: a top-level heading with your site name, a brief description of what your business does, and then categorised sections with links to your most important pages. Each link includes a short, plain-language description of what the page covers. The format is intentionally minimal. There is no complex syntax, no required schema, and no validation step. You can create it in any text editor and deploy it alongside your existing files.

Place the file at your domain root: yoursite.com/llms.txt
Start with a heading and one-paragraph site description.
List your most important pages with brief, clear descriptions.
Group pages by category if your site covers multiple topics or services.
Keep descriptions factual and specific, not promotional.
Update the file whenever you add or remove significant content.

Section 04

What goes in your llms.txt and what stays out.

Include the pages that represent your core expertise and services. Your homepage, main service pages, about page, pricing page, and your highest-quality educational content should all be listed. Include a concise, factual description for each link that tells the model what the page covers without marketing language. Leave out low-value pages like privacy policies, terms of service, cookie notices, login pages, and thin content. The goal is signal density. Every entry in your llms.txt should be a page you would confidently point to as a strong representation of your business. If you have a blog, include only the cornerstone articles that demonstrate genuine expertise, not every post you have ever published. Quality over quantity is the guiding principle. A focused llms.txt with twenty well-described pages will outperform a bloated one with two hundred undifferentiated entries.

Include: homepage, core service pages, about page, pricing, key guides.
Include: case studies and results pages that demonstrate expertise.
Exclude: privacy policy, terms, cookie notices, login pages.
Exclude: thin content, duplicate pages, tag archives, pagination pages.
Keep descriptions under two sentences, factual and specific.

Common questions

Frequently asked.

Q.01Is llms.txt an official web standard?

It is a proposed standard, not yet ratified by a standards body like the W3C. However, it has gained significant traction in the AI and SEO communities, and early adoption signals are positive. Implementing it now carries no risk and positions your site ahead of competitors who wait for formal ratification.

Q.02Does Google use llms.txt for AI Overviews?

Google has not officially confirmed that its AI Overview system reads llms.txt. However, Google's crawlers do access the file when it exists, and the structured context it provides aligns with how retrieval-augmented generation systems select and prioritise sources. The indirect benefits are well established even if direct confirmation is pending.

Q.03How often should I update my llms.txt file?

Update it whenever you add or remove a significant page from your site. For most businesses, a quarterly review is sufficient. If you publish content frequently, consider a monthly check to ensure your most important new pages are included and outdated ones are removed.

Q.04Can llms.txt hurt my traditional SEO?

No. llms.txt is a separate file that AI crawlers read independently. It does not affect your robots.txt, your sitemap, or any traditional ranking signals. It is purely additive. There is no downside to implementing it.

Chapter 07 / The closing word

We build and maintain your llms.txt as part of every engagement.

Start with the free AI visibility audit. If you engage us, llms.txt implementation is included in your onboarding alongside schema markup, entity consolidation, and content restructuring.

See what you're missing (free, 60 seconds)Speak to a human