Glossary

robots.txt

robots.txt is a file at a site's root that tells crawlers, including AI crawlers, which parts of the site they may access. Full definition and related terms.

Diploria

robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they are allowed to access. It is a long-standing web standard, and AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot check it to decide whether they may fetch your pages.

For brands, robots.txt is consequential in a way that some newer files are not: if it blocks the AI crawlers, your content cannot be retrieved or cited by those systems, no matter how good it is. The practical step is to confirm you are not accidentally disallowing the AI user agents you want to reach you, while still controlling access where you genuinely need to.

Learn more

Frequently asked questions