robots.txt Generator
Including the AI crawlers nobody else lets you toggle
Runs in your browser. Nothing is uploaded.
About the robots.txt Generator
Generate a WordPress robots.txt with the standard rules — and per-bot toggles for GPTBot, ClaudeBot, PerplexityBot, CCBot, and Google-Extended. Decide what trains on your writing, one checkbox at a time.
Related
Other tools you might need
FAQ
Questions
Does blocking Google-Extended hurt my search ranking?
No. Google-Extended controls whether your content is used to train Gemini. It has no effect on Googlebot, on indexing, or on your position in Google Search. This is the single most common misunderstanding about the AI crawler directives.
Will robots.txt actually stop AI companies scraping my site?
It stops the ones that respect it, which is most of the named ones. It is a request, not a fence — a crawler that ignores robots.txt is entirely unaffected. Anyone selling robots.txt as protection against scraping is overstating it.
Where does the robots.txt file go?
In your site's web root, alongside wp-config.php, so it resolves at example.com/robots.txt. WordPress serves a virtual robots.txt when no real file exists; uploading a real one takes over from it.