Skip to content

Runs in your browser. Nothing is uploaded.

WordPress robots.txt Generator with AI Crawler Toggles

The standard WordPress rules, plus a toggle for each AI crawler grouped by what it does

The default WordPress robots.txt

If there is no real robots.txt file at your web root, WordPress prints this one on the fly (for a site installed at the domain root, with pretty permalinks and search engines allowed):

robots.txt (WordPress default)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Replace example.com with your domain. The Sitemap line has been added by core since WordPress 5.5, and only while the site is public. Without pretty permalinks it points at /?sitemap=index instead.

WordPress basics
AI crawlers

Each ticked bot gets its own record asking it not to crawl the site. A crawler that ignores robots.txt is not affected. The bots are grouped by what their makers say they do: blocking an AI search bot can keep you out of that product's answers, and two of the user fetchers are documented as not always following robots.txt.

Model training and datasets

AI search

Pages a user asks for

Other

Sitemap
robots.txt

WordPress prints a virtual robots.txt only while no real file exists at your web root. Upload this file and it takes over, so the Sitemap line WordPress adds is gone unless you include one above.

About the WordPress robots.txt Generator with AI Crawler Toggles

Build a WordPress robots.txt: block the admin and search pages, choose which AI crawlers to block by what each one does, and add your sitemap line.

Worked example: keep AI training crawlers out, stay in AI search

Settings

Disallow /wp-admin/ (but allow admin-ajax.php): on
Disallow internal search results (/?s=): on
Model training and datasets: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, CCBot
AI search and pages a user asks for: none ticked
Sitemap URL: https://example.com/wp-sitemap.xml

Generated robots.txt

# Generated at odiethemes.com/tools/wordpress-robots-txt-generator

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=

# AI crawlers
# Each record asks one bot not to crawl the site.
# Crawlers that ignore robots.txt are not affected.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/wp-sitemap.xml
  • The User-agent: * group applies to every crawler that has no group of its own. Its first two rules are the ones WordPress prints itself; Disallow: /?s= adds internal search results, which WordPress serves at addresses starting with /?s=.
  • Each named bot gets its own group with Disallow: /, which asks it not to fetch anything. A crawler follows the group that names it, so these bots skip the * rules and use their own.
  • GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot and CCBot are the tool's model training and datasets group: their makers say the content may be used to train AI models, or, for CCBot, goes into an open dataset. Amazon also uses Amazonbot to improve its products and services; its search crawler, Amzn-SearchBot, is a separate bot and is not blocked here. OAI-SearchBot, Claude-SearchBot and PerplexityBot are left alone, so the site can still appear in ChatGPT search, Claude's search results and Perplexity.
  • The Sitemap line is needed because a real robots.txt replaces the one WordPress prints, and with it the Sitemap line core adds. /wp-sitemap.xml is core's sitemap index; use your SEO plugin's address if it replaces core's.

When you need it

Most sites do fine with the default file shown at the top of this page. Generate your own when you want rules core does not add: keeping AI crawlers out, skipping internal search pages, or pointing at a sitemap from an SEO plugin instead of wp-sitemap.xml.

What the AI crawler groups mean

  • Model training and datasets: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, Applebot-Extended, Amazonbot and CCBot (Common Crawl). Google-Extended and Applebot-Extended have no crawler of their own. They are tokens that tell Google and Apple whether pages their normal crawlers fetch may be used for their AI models, and both companies say blocking them does not affect search.
  • AI search: OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages for ChatGPT search, Claude and Perplexity. OpenAI says a site that blocks OAI-SearchBot is not shown in ChatGPT search answers, though it can still appear as a navigational link.
  • Pages a user asks for: ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person's question needs it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, so ticking these may change nothing.
  • Bytespider is ByteDance's crawler. None of the sources below describe it, so the tool makes no claim about what it is used for.

How to use it

  1. Tick the rules you want and enter the sitemap URL for the host you actually use (www or not).
  2. Save the output as a plain text file named robots.txt.
  3. Upload it to your web root, the same folder as wp-config.php on most installs. A real file takes over from the virtual one.
  4. Open yoursite.com/robots.txt in a browser to confirm the new rules are live.

Common mistakes

  • Using robots.txt to hide a page from Google. Google says it is not a way to keep a page out of its index; use noindex or a password instead.
  • Blocking /wp-content/ or /wp-includes/. Google asks you not to block CSS and scripts it needs to understand the page.
  • Expecting "Discourage search engines" to write Disallow: / into the file. Since WordPress 5.3 that setting adds a noindex robots meta tag instead.
  • Uploading a real file and losing the Sitemap line. Add it in the Sitemap field above.

The WordPress robots.txt best practice guide explains each default line, and how to block AI crawlers covers what the bot toggles do and do not stop.

Sources

Guides for this tool

Questions

Does blocking Google-Extended hurt my search ranking?

No. Google says Google-Extended does not affect whether your site is included in Google Search and is not used as a ranking signal there. It decides whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini Apps and the Vertex AI API. It has no user agent of its own: Google's usual crawlers fetch the pages, and the robots.txt token only controls how the content is used.

Will robots.txt actually stop AI companies scraping my site?

Only crawlers that choose to follow it. OpenAI, Anthropic, Google, Apple, Amazon, Perplexity and Common Crawl all publish the names of their crawlers, but OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. A scraper that ignores robots.txt is not affected at all. To refuse requests outright you need a rule on your server or firewall.

Where does the robots.txt file go?

In your site's web root, usually the folder that holds wp-config.php, so it opens at example.com/robots.txt. WordPress prints a virtual robots.txt only while no real file exists. Once you upload one, that version is no longer served, including the Sitemap line WordPress adds, so put your sitemap URL in the file yourself.