AI Crawlers, robots.txt and llms.txt: What to Allow So ChatGPT, Gemini and Perplexity Can Cite You (2026)
By Alexey Chasin · Updated September 25, 2026 · 8 min read
A page cannot be cited by an AI engine that is not allowed to read it. In 2026 the crawlers that matter are GPTBot and OAI-SearchBot (OpenAI, ChatGPT), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini) alongside Googlebot, and Bingbot (which feeds ChatGPT search). A business that wants customers from AI search allows all of them in robots.txt, publishes an llms.txt file listing its key pages in plain text, and keeps its sitemap submitted to Google and Bing. Many WordPress sites and some hosts block AI crawlers by default, so check yours today: the cost of blocking is invisibility in every AI answer.
The crawlers and who they feed
| User agent | Company | Feeds | Allow it if you want |
|---|---|---|---|
| GPTBot | OpenAI | Model training and ChatGPT browsing | To be known to ChatGPT |
| OAI-SearchBot | OpenAI | ChatGPT search results and citations | To be cited in ChatGPT answers |
| ChatGPT-User | OpenAI | Live fetches when a user asks about a page | Same |
| ClaudeBot | Anthropic | Claude's web search and citations | To be cited by Claude |
| PerplexityBot | Perplexity | Perplexity's index and answers | To be cited by Perplexity |
| Google-Extended | Gemini and AI features training | To be known to Gemini; does not affect Search ranking | |
| Googlebot | Search, AI Overviews, AI Mode | Always | |
| Bingbot | Microsoft | Bing, Copilot, and ChatGPT search retrieval | Always |
What to put in robots.txt
The safe default for a business site is to allow everything except admin and private paths, then name the AI crawlers explicitly so a future default cannot silently block them. A rule set that allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended with 'Allow: /' each, plus the general 'User-agent: *' rule allowing the site and disallowing your admin and API paths, and a Sitemap line at the end, covers every engine in the table. Test it in Search Console's robots tester and by fetching the file yourself at yoursite.com/robots.txt.
How sites end up blocked without knowing
- WordPress security and SEO plugins that added 'block AI bots' toggles in 2024 and 2025, on by default in some versions.
- Cloudflare's one-click AI crawler block, which some hosts enable for every site on the account.
- A robots.txt copied from a template that disallows everything but Googlebot.
- A staging site setting, 'Disallow: /', that shipped to production.
- A firewall rule that returns 403 to any user agent containing 'bot' that is not Google.
The check takes two minutes: open yoursite.com/robots.txt and read it, then use a fetch tool or Search Console's URL inspection to confirm a page returns 200 to a non-Google user agent. If you find a block, the fix is one line, and the pages start appearing in AI answers within weeks of the next crawl.
What llms.txt is
llms.txt is a plain-text file at the site root, proposed in 2024 and adopted widely through 2025 and 2026, that gives language models a curated index of a site: a one-line description of the business, then the most important pages with a title and one sentence each. It is the sitemap's editorial cousin. A sitemap says what exists; llms.txt says what matters and what each page answers. Engines that read it get the site's structure without crawling every page, and the pages you list are the ones that get read first.
How to write llms.txt
- First line: the business name as a heading, then one sentence saying what it does and where.
- Then sections: Services, Locations, Pricing, Guides. Under each, one line per page: the page title, a link, and one sentence with the answer the page gives.
- Keep it under 100 lines. It is a reading list, not a copy of the sitemap.
- Update it when you add a page that matters. A growth plan that adds pages monthly updates it monthly.
The trade-off with content protection
Publishers with paywalled content have reasons to block training crawlers, and Google-Extended exists so a site can stay in Search while opting out of Gemini training. A local business selling services has the opposite interest: the content on the site is the pitch, and every engine that reads it is a channel. Block nothing that can cite you. If a particular page must stay private, disallow that path, not the crawler.
Sitemaps and indexing requests
Keep one sitemap that lists every indexable page with a real last-modified date, submit it in Google Search Console and Bing Webmaster Tools, and request indexing for new pages instead of waiting. Bing matters more than it used to: ChatGPT's search retrieval draws on Bing's index, so a page Bing has not indexed cannot be cited by ChatGPT search however well Google ranks it.
This site allows every crawler in the table, publishes llms.txt, and keeps a 149-URL sitemap with stable last-modified dates. Fetch velourweb.com/robots.txt and velourweb.com/llms.txt to see the pattern.