Crawl Control
Robots.txt tells compliant crawlers which URLs they may request. It does not make content private and it is not the right tool for removing already known pages from search.
Problem
WordPress robots.txt Checklist
Review WordPress robots.txt for accidental crawl blocks, stale staging rules, sitemap discovery, admin boundaries, and search visibility side effects.
Problem, cause, fix, verify
Problem
A robots.txt file can accidentally block important WordPress content, confuse diagnosis, or hide crawler access differences between staging and production.
Audience
WordPress administrators, SEO teams, developers, and agencies checking crawler access before launch or after index drops.
First check
Robots.txt tells compliant crawlers which URLs they may request. It does not make content private and it is not the right tool for removing already known pages from search.
Verify
WPlura Site Discovery Checker and Diagnostics can help record robots rules, sitemap discovery, public URL behavior, and owner notes during launch or index-drop review.
Editorial trust
- Publisher
- WPlura Editorial Team
- Published
- Updated
- Content type
- Problems
On this page
Symptom
A robots.txt file can accidentally block important WordPress content, confuse diagnosis, or hide crawler access differences between staging and production.
For: WordPress administrators, SEO teams, developers, and agencies checking crawler access before launch or after index drops.
Diagnosis
What to check before changing the site
Launch Leftovers
A staging block that was safe yesterday can become a production incident after launch. Always fetch robots.txt from the final host and compare it with the live sitemap index.
Asset Visibility
Blocking CSS, JavaScript, images, or render-critical paths can make pages harder to evaluate. Test assets separately from admin or internal routes.
Sitemap Discovery
A robots.txt file can point crawlers to the sitemap index. That line should use the canonical public URL and avoid stale staging, www, HTTP, or redirected sitemap addresses.
WPlura Fit
WPlura Site Discovery Checker and Diagnostics can help record robots rules, sitemap discovery, public URL behavior, and owner notes during launch or index-drop review.
WordPress Search VisibilitySeverity: HighLast reviewed: 2026-09-16
Diagnosis
Symptoms, causes, checks, and fixes
Symptoms
- Search Console reports submitted URLs blocked by robots.txt.
- Important pages are crawlable in a browser but blocked for Googlebot or other crawlers.
- The robots file still references staging, old sitemaps, blocked asset paths, or broad disallow rules.
Most Common Causes
- Robots.txt is often edited quickly during development and then forgotten during launch.
- Broad disallow rules can block CSS, JavaScript, product pages, media, feeds, or custom content paths needed for discovery.
- Robots.txt is not a privacy control and should not be used to hide sensitive URLs from search results.
How To Confirm The Cause
- Fetch robots.txt from the final public host and record status, content type, user-agent groups, disallow rules, allow rules, and sitemap lines.
- Test representative public URLs, assets, sitemap files, admin paths, internal search, and noindex pages against the actual rules.
- Compare robots rules with page-level noindex, canonical state, and sitemap membership.
Fixes
- Remove broad production blocks that prevent crawling of approved public content or required rendering assets.
- Keep private or sensitive content protected by authentication or removal, not by robots.txt alone.
- Add or correct sitemap references so crawlers can discover the canonical sitemap index.
How To Verify The Fix
- Re-fetch robots.txt after deployment and cache purge.
- Confirm approved sitemap URLs and priority pages are crawl-allowed.
- Confirm intentionally excluded areas use the right mechanism: robots for crawl control, noindex for search exclusion, authentication for private content.
When To Contact Support
Escalate when robots.txt is generated by a host, CDN, edge function, SEO plugin, multisite network rule, or deployment pipeline outside normal WordPress admin control.
References