Should you block AI crawlers?
Blocking AI crawlers is a real trade rather than a best practice. A publisher whose archive is the product has a genuine case for it. A company that wants to be recommended is removing itself from retrieval, which means it cannot be cited no matter what it publishes. The answer differs by business model, and the file lets you split the difference.
What actually happens when you block GPTBot?
Your pages stop being fetched by that crawler, so they stop entering the retrieved set an assistant writes from. You cannot be cited from content it cannot read.
The nuance most robots files miss is that these crawlers do different jobs. Some fetch for training, some fetch live to answer a question in the moment. Blocking both with one rule is common and usually not what the site owner intended.
Who should block, and who should not
If your archive is the product and subscriptions pay for it, giving it away as training material has an obvious cost and no obvious return.
You want to be recommended. Blocking removes you from the competition entirely, which is a strange way to win it.
Blocking does not stop assistants discussing you. It stops them reading your correct, current description, which usually makes the misinformation problem worse.
Can I allow retrieval but refuse training?
Partly. Some crawlers are separated by purpose, so you can allow the search-facing agent while disallowing the training one. Coverage is inconsistent and changes, so check the current agent names rather than copying an old file.
Common questions
It stops compliant crawlers. It does not stop content reaching a model through third parties who quoted you, which is often how a brand appears in answers anyway.
Read your own robots.txt and look for the AI agent names specifically. Many sites block them through a wildcard rule added years ago for a different reason.
Covers the work that changes whether an engine names you.