AI crawlers

Also written GPTBot, ClaudeBot, PerplexityBot, Google-Extended

The user agents AI companies use to fetch web pages, either for training or to answer a question in the moment.

The main ones are GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended, which governs whether Google may use your content for AI features separately from ordinary search.

Blocking them in robots.txt is a real choice with a real cost. A publisher selling subscriptions may reasonably decide their archive should not be free training material. A company that wants to be recommended is removing itself from retrieval, which means it cannot be cited.

Worth ruling out first when you find you are absent from answers. It takes a minute to check and explains the problem entirely when it is the cause.

llms.txtA proposed plain-text file at a site's root that offers language models a curated map of its most useful pages.Retrieval-augmented generation (RAG)The pattern where a model searches for relevant documents first, then writes its answer using what it found.Citation rateThe proportion of AI answers that cite at least one page on your own domain as a source.