Robots.txt tells crawlers where they may go. llms.txt tells AI models what your site is about. Small file, growing importance.
Definition
llms.txt
llms.txt is a proposed web standard (2024): a markdown file at a fixed location (/llms.txt) that gives AI language models a concise, structured summary of a website: which pages exist, what they cover and where the key information lives. Where robots.txt governs what crawlers may do, llms.txt helps AI systems understand what is there.
AI search engines and assistants draw their answers from web content. An llms.txt makes it easy for them to summarise and cite your site correctly, resulting in better representation in AI answers. It costs nothing and can only help.
What does it look like?
An llms.txt is plain markdown: a title, a short description of the site and a grouped list of links with one line of context per page. Larger sites add an llms-full.txt with the full content as plain text. This site has one itself: see fwrd.digital/llms.txt for a live example.
Status of the standard
llms.txt is a proposal, not an official standard: no AI vendor has guaranteed support. But the cost is zero, adoption is growing and the major crawlers demonstrably fetch the file. Comparable to the early days of sitemap.xml: early adopters benefit longest.
Necessary cookies are always on: the site runs on them. On top of that you can turn on analytics & ad measurement, so we can see which pages work and whether our ads bring in new customers.
Choose per category. Strictly necessary cookies can't be turned off. The site won't work without them.
Strictly necessary always on
Cloudflare bot protection and preview mode. No tracking.
Analytics & ad measurement
Google Analytics 4 in anonymised mode plus the Reddit and Google Ads measurement pixels. Those platforms receive a cookie ID, and on a sign-up or enquiry also a hashed (irreversibly encrypted) version of your email address, so they can match a conversion to the right ad click. We never share the address itself.
