WordPress robots.txt Best Practice in 2026 (What to Actually Put In It)

A good WordPress robots.txt is about five lines long. Block /wp-admin/, allow admin-ajax.php, point crawlers at your sitemap, and stop there — the sprawling 40-line blocklists you’ll find on forums mostly break rendering or do nothing at all. The whole file:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Replace the domain, and if you use an SEO plugin that takes over sitemaps, point the Sitemap line at that plugin’s index instead. That’s it. If your file is longer than this, the extra lines are probably costing you something.

WordPress already generates one — until you create a real file

WordPress ships a virtual robots.txt. When a request comes in for /robots.txt and no actual file exists at your web root, WordPress intercepts the request and prints a response in memory. Nothing is written to disk. The default output is the wp-admin block above, plus a Sitemap: line for wp-sitemap.xml that core has emitted since WordPress 5.5.

Two things change that output. If Settings → Reading → Discourage search engines from indexing this site is checked, WordPress replaces the whole thing with Disallow: /, which blocks everything. That checkbox is the single most common cause of a site vanishing from search after a launch. Second, plugins and themes can modify the virtual output through the robots_txt filter, which is how SEO plugins inject their own sitemap lines and rules.

Now the part that wastes people’s afternoons. A physical robots.txt file in your web root silently overrides all of it. Your web server finds a real file on disk, serves it, and WordPress never loads. The virtual generator doesn’t run. The robots_txt filter doesn’t fire. Your SEO plugin’s robots.txt editor may still show you a nice interface with the right rules in it, and none of that reaches a crawler.

Where the rules live	Editable in WordPress admin	Wins on a live request
Virtual (no file on disk)	Yes, via the robots_txt filter or a plugin	Only when no physical file exists
Physical file in web root	Only if the plugin writes to disk	Always

So before you debug anything: open https://yoursite.com/robots.txt in a browser, then connect over SFTP or your host’s file manager and check whether a real robots.txt sits next to wp-config.php and .htaccess. If it does, that file is your robots.txt. Edit it there or delete it and let WordPress take over — but pick one. Running both is how sites end up with a stale sitemap URL nobody can find.

Why /wp-admin/ is blocked but admin-ajax.php is allowed

/wp-admin/ holds the dashboard. Those URLs are useless in search results and crawling them burns crawl budget on pages that redirect to a login screen. Blocking the directory is uncontroversial.

admin-ajax.php lives inside that directory but isn’t an admin page. It’s the endpoint plugins and themes use to handle front-end AJAX requests — load-more buttons, filtered product grids, calculators, forms. If a crawler can’t fetch it, it can’t see content those features load, and Google renders your page as a visitor would see it with the feature broken.

Whether Google actually needs to fetch admin-ajax.php on your specific site depends on whether your front end uses it. Honest answer: on many modern sites it doesn’t matter at all, because the theme uses the REST API at /wp-json/ instead. The Allow line is a cheap insurance policy against a real failure mode, not a ranking factor. Keep it because it costs nothing.

Do not block /wp-content/ or /wp-includes/

This is the single worst piece of robots.txt advice still in circulation, and it dates from an era when Google didn’t render pages.

/wp-content/ contains your themes, plugins, uploads — the stylesheets, scripts, and images that make your pages look like pages. /wp-includes/ contains core JavaScript that plugins depend on. Google renders your pages with a headless browser before judging them. Block those directories and the renderer gets a page with no CSS and no images. What Google evaluates is an unstyled wall of text with a broken layout, which affects how it understands your content and how it scores mobile usability.

If you inherited a file with these lines, delete them:

Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/

You can confirm the damage yourself in Google Search Console’s URL Inspection tool. Run a live test on any page, view the rendered screenshot, and check the page resources list for blocked items. If your CSS shows as blocked, that’s the cause.

The Sitemap directive

One line, absolute URL, and it can sit anywhere in the file — it isn’t tied to a User-agent group.

Sitemap: https://example.com/wp-sitemap.xml

Use the URL that actually resolves. WordPress core serves wp-sitemap.xml. Most SEO plugins disable core sitemaps and serve their own index at a different path, typically sitemap_index.xml. Load your URL in a browser before you commit it — a Sitemap line pointing at a 404 is worse than no line, because it looks correct in an audit.

Listing multiple sitemaps is fine, one per line. Submitting sitemaps in Search Console is still worth doing separately; the robots.txt directive is how crawlers that never see your Search Console account find them.

robots.txt is not noindex, and it is not security

These two misunderstandings cause most robots.txt damage.

Blocking a URL does not remove it from Google. Robots.txt governs crawling. If another site links to a blocked URL, Google can index the URL itself — you get a result with the bare address and a note that no information is available. And there’s a nastier trap: if a page already has a noindex meta tag and you then block it in robots.txt, Google can’t crawl it, can’t see the noindex, and the page can stay indexed indefinitely. To remove a page, allow crawling and serve a noindex meta robots tag in the page head or an X-Robots-Tag HTTP header. Google stopped supporting noindex directives inside robots.txt in 2019.

Blocking a path does not protect it. Your robots.txt is public by definition. Listing /private-client-files/ tells every visitor and every scanner exactly where to look, and compliance is voluntary — Google honours it, hostile bots do not. Anything that needs protecting needs authentication, an IP restriction, or server-level rules in .htaccess, not a line in a text file that advertises the target.

Rules worth adding, and rules worth skipping

Two additions are genuinely useful on many sites. Blocking a search-results path stops crawlers generating endless low-value URLs from your internal search. Blocking a cart or checkout path on WooCommerce keeps session-parameter URLs out of the crawl.

Disallow: /?s=
Disallow: /search/

Skip these:

• Crawl-delay — Google ignores it entirely. Use the crawl rate controls in Search Console if you have a real server load problem.
• Blocking /feed/ or /trackback/ — harmless URLs, and feeds are useful.
• Long lists of individual bot names — most of the bots people try to block don’t identify honestly anyway.
• Blocking /wp-json/ — it may be what your theme uses to render content.

Build the file with the WordPress robots.txt generator, which produces the minimal correct version and lets you add the search and commerce rules without hand-editing syntax.

After you change it

Load https://yoursite.com/robots.txt directly and confirm the served bytes are what you expect — not what your plugin’s editor shows you. Then run a live URL inspection in Search Console on a normal page and check that no CSS or JavaScript resources come back blocked. Google caches robots.txt for roughly a day, so a fix won’t take effect instantly, and a mistake won’t either. That delay is the reason to verify carefully rather than test in production and wait.