llms.txt and AI crawlers explained: what to allow on your website
What llms.txt really is, how robots.txt controls GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and how to decide what to allow.
ToBrand team · 6 min read
More and more people ask ChatGPT, Claude, Perplexity or Gemini for a recommendation before they ever open Google. Those assistants learn about your business partly by sending automated programs, called crawlers, to read your website. Two small text files decide much of what they see: robots.txt, which has been around for decades, and llms.txt, a newer idea that gets talked about a lot. This guide explains both in plain language and helps you decide what to allow.
What a crawler is, and why there are now so many
A crawler is a program that visits web pages and copies their text. Google has used one (Googlebot) since the beginning, to build its search index. AI companies now run their own crawlers for two different jobs:
- Training: collecting public text that helps build future versions of a language model.
- Search and answers: fetching pages in real time, or keeping a fresh index, so the assistant can answer a question and link to sources.
The difference matters. Blocking a training crawler keeps your text out of future model training. Blocking a search crawler can keep your business out of the answers people read today, including the links that send visitors to you.
The AI crawlers worth knowing by name
Each crawler announces itself with a name, called a user agent. The ones a small business is most likely to meet are:
- GPTBot (OpenAI): collects content that may be used to train OpenAI's models. OpenAI uses a separate name, OAI-SearchBot, for the crawler behind search results in ChatGPT.
- ClaudeBot (Anthropic): collects public web content for training Anthropic's Claude models. Anthropic documents separate agents for fetching pages when a user asks.
- PerplexityBot (Perplexity): indexes pages so Perplexity can show and link to them in its answers.
- Google-Extended (Google): not a separate crawler but a control token. Googlebot still does the visiting; Google-Extended tells Google whether content it crawls may be used to train and ground its Gemini models. According to Google's Search Central documentation, it does not affect whether or how your pages appear in Google Search.
- CCBot (Common Crawl): builds a large public archive of the web that many AI projects have used.
Names and purposes change from time to time, so check each company's own documentation before you rely on a rule. Google's list of its crawlers and tokens is on Google Search Central.
robots.txt: the file crawlers actually check
robots.txt is a plain text file at the root of your site, for example yourdomain.com/robots.txt. It lists which crawlers may visit which parts of the site. Reputable crawlers read it before they fetch anything. It is a polite request, not a lock: a badly behaved bot can ignore it, so never use robots.txt to hide private information. Anything private needs a password.
A rule has two parts: the crawler it applies to, and what it may not fetch. Here is an example that keeps Google Search open, allows AI search crawlers, and opts out of model training:
# Everyone may crawl the public site, except the admin area
User-agent: *
Disallow: /admin/
# Opt out of AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# AI search crawlers may read the public site
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://www.yourdomain.com/sitemap.xmlA crawler follows the most specific group that names it, so GPTBot obeys its own group and ignores the general one. If a crawler is not named at all, it falls back to the rules under User-agent: *.
llms.txt: a proposal, not a standard
llms.txt is a suggestion first published in 2024: put a Markdown file at yourdomain.com/llms.txt that gives language models a short, clean summary of your site, with links to the pages that matter most. Think of it as a reading list written for a machine, without menus, pop-ups and footers in the way.
It is important to be honest about where it stands:
- It is a proposed convention, not an official web standard, and no standards body has adopted it.
- It does not block or allow anything. Permission still lives in robots.txt.
- The large AI companies have not publicly committed to reading it, and Google has not said it uses it for Search. Some developer tools and documentation sites do use it.
So treat llms.txt as a low-cost extra. It takes an hour to write, it does no harm, and it forces you to state clearly what your business does, which is useful in itself. A simple version looks like this:
# Sharma Dental Clinic
> Family dental clinic in Kothrud, Pune. Open Monday to Saturday.
## Services
- [Root canal treatment](https://www.example.com/root-canal/): cost, steps, recovery
- [Braces and aligners](https://www.example.com/braces/): options for teens and adults
## About
- [Our dentists](https://www.example.com/team/)
- [Contact and directions](https://www.example.com/contact/)How to decide what to allow
There is no single right answer. It depends on how you earn money from your content. Work through these questions:
- Do you want to be recommended by AI assistants? If you are a clinic in Pune, a law firm in Texas or a hotel in Da Nang, being named in an answer brings customers. Keep search crawlers such as OAI-SearchBot and PerplexityBot allowed, and never block Googlebot.
- Is your content the product? Publishers, course creators and researchers whose articles are what they sell may reasonably opt out of training crawlers, while still allowing search crawlers that link back.
- Do you mind your public text being used for training? For most service businesses, the public pages are marketing. Allowing training does little harm and may help models describe you accurately. If it makes you uncomfortable, block the training agents; it is your choice.
- Is anything sensitive public by mistake? Fix that first. Price lists for partners, staff documents or client files should sit behind a login, not just a Disallow line.
A quick checklist for your site
- Open yourdomain.com/robots.txt and read it. Many sites block crawlers by accident after a redesign or a security plugin update.
- Make sure Googlebot is not blocked from your main pages, CSS or JavaScript.
- Decide on training crawlers and search crawlers separately, and write a group for each you care about.
- Point to your sitemap at the bottom of robots.txt.
- Optionally add an llms.txt that lists your ten most useful pages with one-line descriptions.
- Check again every few months, because crawler names change.
Remember that allowing crawlers only gets you read. Being cited depends on what they find: clear pages, consistent facts and real answers. Our guide to SEO for AI search covers that side, and you can run the free website check to see whether your robots.txt is blocking anything important.
If you would rather have someone set this up and keep it current, ToBrand's AI search optimisation service includes crawler rules, llms.txt and the content work that gets a business mentioned.
Was this guide helpful?
