Guide to the llms.txt file: what it is, how to write it, and what to really expect
When I built caprai.dev, I decided the site should speak two languages: the language of people and the language of AI models. The llms.txt file was one of the first steps, and today caprai.dev is one of the 3,826 sites listed in the llmstxt.cloud directory, which collects adopters of the standard.
This is the handbook I wish I had back then: what the file does, what it doesn't do, and how to write it in an hour.
In brief. llms.txt is a Markdown text file, published at the root of the site, that introduces language models to who you are and which pages to read first. It doesn't control access like robots.txt and doesn't list everything like the sitemap: it's a curated presentation, designed for AI assistants that read your site on someone's behalf.
What is llms.txt
The proposal comes from Jeremy Howard, co-founder of Answer.AI, and was published in September 2024 on llmstxt.org. It's not an official standard, but a convention that spread quickly. To understand it, just compare it with the two files we already know:
- robots.txt tells crawlers where they can go.
- sitemap.xml lists all pages to be indexed.
- llms.txt selects the pages that matter and explains in a few words why.
The file lives at a fixed address, yoursite.com/llms.txt, often accompanied by llms-full.txt with the full text of the content.
What it really does, in October 2026
In 2026, the role of llms.txt has become clearer, and for the better. Google Search documentation explains that the file is not a necessary ranking factor (Search Engine Journal). However, since May, Google checks it in Lighthouse and PageSpeed Insights, within the new Agentic Browsing category (DebugBear): the one that measures how ready a site is for AI agents, alongside accessibility and tools offered to agents.
This is its main field. When an assistant or agent arrives on your site on behalf of a person and looks for a starting point, llms.txt delivers the version of the facts you've chosen. It's a marketing tool before it's an SEO or GEO tool, optimization for AI responses: it guides how an AI tells your brand story when someone asks.
How to create the file: the structure
The specification is minimal. Here's the skeleton:
# Site Name
> One or two sentences: who you are, who you work for, what you offer.
A few lines of context: languages, geographic area, how to read the site.
## Services
- [AI Training for Companies](https://yoursite.com/companies): team programs, with prices and rules
## Main Guides
- [Guide Title](https://yoursite.com/guide): the problem it solves, in one line
## Articles URLs
- [Article Archive](https://yoursite.com/blog): less central insights
The formal rules are three: a single title with #, the summary in the line starting with >, then sections with ## made of links, each followed by a brief description after the colon. The Optional section tells the model that material can be skipped when space is limited.
The handbook: eight operational rules
- Write the summary as a positioning statement. It's the line a model will repeat most often: who you are, who you work for, what sets you apart.
- Select, don't list. If the file becomes a copy of the sitemap, it has lost its purpose. Top pages you want to see cited, the rest in Optional or in llms-full.txt.
- Write descriptions that convey value. Don't copy the meta description: explain what the page is for and for whom.
- Order by commercial priority. First the pages that bring contacts or sales (services, courses, about), then the in-depth content.
- Use absolute and stable links. A link that goes elsewhere or to a page that no longer exists is a missed opportunity.
- Handle languages with care. If the site is multilingual, one section per language with links to the right version, never the same list repeated.
- Stay consistent with the rest of the site. Name, description, and services must say the same things as the about page and structured data. AIs cross-reference sources, and a contradiction costs credibility.
- Generate it, don't write it by hand. A static file ages with the first new page: if the site has a CMS or database, make it update automatically.
The caprai.dev case
Our llms.txt isn't handwritten: the site generates it by reading from the database public articles, published courses, and main pages, in Italian, English, and Spanish. At the bottom there's a section that guides usually don't mention: the tools the site makes available to AI agents, which I discussed in the article on agentic web. Alongside are llms-full.txt, with the full text of articles, and a Markdown version of each article, lighter for a model to process.
I learned two lessons in the field. The first: llms-full.txt publishes everything, and it's the most delicate point of the site. In the past, a code error caused the full text of restricted articles to end up in the file. We fixed it, and now automated tests check that no closed content ends up in the two files. The second: legal notices ended up in the list as if they were articles. Now they're excluded.
Before publishing llms-full.txt, open it and read it as a competitor would.
The most common mistakes
- Blocking in robots.txt the bots that should read it, or letting them be stopped by the hosting firewall.
- Publishing it at an address that redirects elsewhere or gives an error: opening
yoursite.com/llms.txtshould immediately show the text. - Filling it with keywords: a model doesn't get convinced, it gets confused.
- Exposing restricted content, drafts, or legal pages in llms-full.txt.
- Forgetting it: dead links and services that no longer exist tell the story of a company that isn't there.
Verify llms.txt in one minute
Run the site through PageSpeed Insights: among Google's checks there's also "The llms.txt file meets necessary requirements," which verifies that the file is in Markdown and has at least one first-level heading. It's a form check, not quality, but it catches obvious errors immediately. caprai.dev passes it, along with checks on tools for AI agents.
How to know if someone is reading it
Search Console doesn't say. The reliable source is server logs: filter requests to /llms.txt and /llms-full.txt by the name the bot presents itself with (GPTBot, ClaudeBot, PerplexityBot and similar) and watch the trend over time.
Frequently asked questions
Does Google use llms.txt to rank pages?
Officially no to date, and Google Search documentation says so. However, Lighthouse verifies it, among checks dedicated to AI agents. However, I wouldn't rule out that in the medium/short term it could acquire relevance for ranking as well.
Is it useful for an SME or e-commerce too?
Yes: just the key pages, services or product categories, described well.
In summary
llms.txt won't make you immediately rise on Google, but it allows you to decide how to present yourself to an artificial intelligence. To control all your site's visibility to AIs, from hosting firewall to sitemap, there's the guide SEO for AI and GEO. And if you want to bring this method to your team, we discuss it in training programs for companies.

The image in this article was generated using AI.



