llms.txt

llms.txt is a proposed standard: a simple text file at the root of a website (for example, example.com/llms.txt) that helps large language models understand the structure and most important content of your site. The idea is similar to robots.txt or sitemap.xml, but instead of targeting search crawlers it targets AI assistants. The proposal came from Jeremy Howard (fast.ai) in 2024 and has been picked up by a growing number of sites since.

The problem it solves: modern websites are full of navigation, ads, scripts, and other clutter a model has to wade through to find the substance, and a model's context window is limited. llms.txt offers a curated, clean overview in Markdown: a short description of the project, links to key pages with a note on what each one contains, and optionally links to full plain-text versions of your documentation. The model gets the essence without the noise.

The format is deliberately simple. It starts with an H1 holding the project name, then a short blockquote summary, followed by link sections ordered by importance. There's also an llms-full.txt convention that bundles the entire documentation into one file.

It's worth being honest: this is still a proposed standard, not something major providers officially guarantee they read. Even so, it's cheap to add, does no harm, and fits into the broader work of helping AI tools understand and cite you correctly. We treat it as one part of GEO, not a standalone magic fix.