Runs in your browser. Nothing is uploaded.
WordPress robots.txt Generator with AI Crawler Toggles
The standard WordPress rules, plus a toggle for each AI crawler grouped by what it does
The default WordPress robots.txt
If there is no real robots.txt file at your web root, WordPress prints this one on the fly (for a site installed at the domain root, with pretty permalinks and search engines allowed):
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml Replace example.com with your domain. The Sitemap line has been added by core since WordPress 5.5, and only while the site is public. Without pretty permalinks it points at /?sitemap=index instead.
About the WordPress robots.txt Generator with AI Crawler Toggles
Build a WordPress robots.txt: block the admin and search pages, choose which AI crawlers to block by what each one does, and add your sitemap line.
Worked example: keep AI training crawlers out, stay in AI search
Settings
Disallow /wp-admin/ (but allow admin-ajax.php): on
Disallow internal search results (/?s=): on
Model training and datasets: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, CCBot
AI search and pages a user asks for: none ticked
Sitemap URL: https://example.com/wp-sitemap.xml Generated robots.txt
# Generated at odiethemes.com/tools/wordpress-robots-txt-generator
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
# AI crawlers
# Each record asks one bot not to crawl the site.
# Crawlers that ignore robots.txt are not affected.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/wp-sitemap.xml - The
User-agent: *group applies to every crawler that has no group of its own. Its first two rules are the ones WordPress prints itself;Disallow: /?s=adds internal search results, which WordPress serves at addresses starting with /?s=. - Each named bot gets its own group with
Disallow: /, which asks it not to fetch anything. A crawler follows the group that names it, so these bots skip the*rules and use their own. - GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot and CCBot are the tool's model training and datasets group: their makers say the content may be used to train AI models, or, for CCBot, goes into an open dataset. Amazon also uses Amazonbot to improve its products and services; its search crawler, Amzn-SearchBot, is a separate bot and is not blocked here. OAI-SearchBot, Claude-SearchBot and PerplexityBot are left alone, so the site can still appear in ChatGPT search, Claude's search results and Perplexity.
- The Sitemap line is needed because a real robots.txt replaces the one WordPress prints, and with it the Sitemap line core adds. /wp-sitemap.xml is core's sitemap index; use your SEO plugin's address if it replaces core's.
When you need it
Most sites do fine with the default file shown at the top of this page. Generate your own when you want rules core does not add: keeping AI crawlers out, skipping internal search pages, or pointing at a sitemap from an SEO plugin instead of wp-sitemap.xml.
What the AI crawler groups mean
- Model training and datasets: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, Applebot-Extended, Amazonbot and CCBot (Common Crawl). Google-Extended and Applebot-Extended have no crawler of their own. They are tokens that tell Google and Apple whether pages their normal crawlers fetch may be used for their AI models, and both companies say blocking them does not affect search.
- AI search: OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages for ChatGPT search, Claude and Perplexity. OpenAI says a site that blocks OAI-SearchBot is not shown in ChatGPT search answers, though it can still appear as a navigational link.
- Pages a user asks for: ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person's question needs it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, so ticking these may change nothing.
- Bytespider is ByteDance's crawler. None of the sources below describe it, so the tool makes no claim about what it is used for.
How to use it
- Tick the rules you want and enter the sitemap URL for the host you actually use (www or not).
- Save the output as a plain text file named robots.txt.
- Upload it to your web root, the same folder as wp-config.php on most installs. A real file takes over from the virtual one.
- Open yoursite.com/robots.txt in a browser to confirm the new rules are live.
Common mistakes
- Using robots.txt to hide a page from Google. Google says it is not a way to keep a page out of its index; use noindex or a password instead.
- Blocking /wp-content/ or /wp-includes/. Google asks you not to block CSS and scripts it needs to understand the page.
- Expecting "Discourage search engines" to write Disallow: / into the file. Since WordPress 5.3 that setting adds a noindex robots meta tag instead.
- Uploading a real file and losing the Sitemap line. Add it in the Sitemap field above.
The WordPress robots.txt best practice guide explains each default line, and how to block AI crawlers covers what the bot toggles do and do not stop.
Sources
- WordPress Code Reference: do_robots()
- WordPress Code Reference: WP_Sitemaps::add_robots()
- OpenAI developer docs: Overview of OpenAI crawlers
- Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google for Developers: Google's common crawlers
- Perplexity docs: Perplexity crawlers
- Apple Support: About Applebot
- Amazon Developer: Amazon Web Search crawlers (Amazonbot)
- Common Crawl: CCBot
- Google Search Central: Introduction to robots.txt
- Google for Developers: How Google interprets the robots.txt specification
- WordPress Code Reference: get_trackback_url()
Guides for this tool
- Disable WordPress AI Features: Core vs Plugin vs Host
Switch off WordPress core AI with WP_AI_SUPPORT or the wp_supports_ai filter, trace plugin and host AI panels to their source, and keep AI crawlers out.
- WordPress llms.txt: What It Is, How to Add It, Does It Help?
Publish an llms.txt file on WordPress three different ways, and see what Google and the AI companies have published about whether anyone reads it.
- WordPress robots.txt Best Practice: What to Put In It
A WordPress robots.txt needs four lines: block /wp-admin/, allow admin-ajax.php and add your sitemap. See which extra rules help and which ones hurt.
- Block GPTBot and Other AI Crawlers in WordPress robots.txt
Block GPTBot, ClaudeBot and other AI crawlers in WordPress robots.txt, see what each block costs you, and know which bots a robots.txt rule cannot stop.
Other tools you might need
Questions
Does blocking Google-Extended hurt my search ranking?
No. Google says Google-Extended does not affect whether your site is included in Google Search and is not used as a ranking signal there. It decides whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini Apps and the Vertex AI API. It has no user agent of its own: Google's usual crawlers fetch the pages, and the robots.txt token only controls how the content is used.
Will robots.txt actually stop AI companies scraping my site?
Only crawlers that choose to follow it. OpenAI, Anthropic, Google, Apple, Amazon, Perplexity and Common Crawl all publish the names of their crawlers, but OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. A scraper that ignores robots.txt is not affected at all. To refuse requests outright you need a rule on your server or firewall.
Where does the robots.txt file go?
In your site's web root, usually the folder that holds wp-config.php, so it opens at example.com/robots.txt. WordPress prints a virtual robots.txt only while no real file exists. Once you upload one, that version is no longer served, including the Sitemap line WordPress adds, so put your sitemap URL in the file yourself.