Robots.txt and meta robots tags are the two primary mechanisms for controlling which pages search engines can crawl and index — robots.txt blocks crawlers from accessing specific URLs, while meta robots directives tell crawlers whether to index a page or follow its links after they’ve accessed it. Getting these wrong can accidentally hide your best content from Google or expose pages you intended to keep private. For Indian website owners managing WordPress sites with Rank Math, understanding these controls is essential for proper SEO in 2026.
Last updated October 2026
Robots.txt: Controlling Crawl Access
What Is robots.txt?
A plain text file at your website’s root (yoursite.com/robots.txt) that tells search engine crawlers which URLs they should or shouldn’t request. It does NOT prevent indexing — it prevents crawling.
Critical distinction: if a page is blocked in robots.txt but linked from other sites, Google may still index the URL (showing it in search results with “No information is available for this page”) — it just won’t crawl the page’s content.
Basic Syntax
User-agent: *— applies to all crawlersUser-agent: Googlebot— applies only to GoogleDisallow: /admin/— blocks the /admin/ directoryAllow: /admin/public/— allows a specific subdirectory within a blocked directorySitemap: https://yoursite.com/sitemap_index.xml— points crawlers to your sitemap
Common robots.txt for WordPress + WooCommerce
A well-configured robots.txt for an Indian business WordPress site:
Disallow: /wp-admin/(admin area)Allow: /wp-admin/admin-ajax.php(needed for some front-end features)Disallow: /cart/(WooCommerce cart)Disallow: /checkout/(WooCommerce checkout)Disallow: /my-account/(customer account pages)Disallow: /*?s=(search results pages)Disallow: /*?orderby=(WooCommerce sorting parameters)Disallow: /tag/(if tag archives are thin; optional)
Managing robots.txt in Rank Math
Rank Math Pro provides a built-in robots.txt editor: Rank Math → General Settings → Edit robots.txt. Edit directly in the WordPress dashboard without FTP access. Changes take effect immediately.
Meta Robots: Controlling Indexing
What Are Meta Robots Tags?
HTML meta tags placed in the <head> of a page that instruct search engines on indexing and link-following behaviour:
<meta name="robots" content="noindex">— don’t index this page<meta name="robots" content="nofollow">— don’t follow links on this page<meta name="robots" content="noindex, nofollow">— don’t index or follow links<meta name="robots" content="index, follow">— index and follow (default behaviour)<meta name="robots" content="noarchive">— don’t show cached version<meta name="robots" content="max-snippet:150">— limit snippet to 150 characters
When to Use noindex
Pages that should exist on your site but NOT appear in search results:
- Thank-you pages (after form submission or purchase)
- Cart, checkout, and my-account pages (WooCommerce)
- Internal search results pages
- Tag archives (if they add no value beyond category archives)
- Author archives (on single-author sites — just duplicates the blog feed)
- Pagination pages beyond page 3-5 (thin content)
- Staging/development pages accidentally left live
- Legal pages you’re legally required to have but don’t want competing for keywords (some debate here)
Setting Meta Robots in Rank Math
Rank Math provides multiple levels of control:
- Global defaults: Rank Math → Titles & Meta → [Post Type/Taxonomy] — set default robots directives per content type. Example: noindex all tag archives globally.
- Per-page override: Edit any post/page → Rank Math meta box → Advanced tab → set Robots Meta for that specific page.
- Bulk editing: Rank Math’s Posts/Pages list view shows robots meta status — bulk edit multiple pages at once.
robots.txt vs Meta Robots: Key Differences
| Aspect | robots.txt | Meta Robots |
|---|---|---|
| Controls | Crawling (access) | Indexing (display in results) |
| Scope | URL patterns (directories) | Individual pages |
| Prevents indexing? | No (page can still appear in results) | Yes (noindex removes from results) |
| Prevents crawling? | Yes | No (page must be crawled to read the tag) |
| Where applied | /robots.txt file at root | HTML head of each page |
| Management | Rank Math robots.txt editor | Rank Math per-page/global settings |
AI Crawler Management
In 2026, managing AI crawlers is a new consideration:
- GPTBot (OpenAI): Crawls content for ChatGPT training. Block with
User-agent: GPTBot / Disallow: /if you don’t want your content used for AI training. - ClaudeBot (Anthropic): Crawls for Claude’s training data.
- Google-Extended: Controls Google’s AI training crawling (separate from regular Googlebot).
- PerplexityBot: Crawls for Perplexity AI search.
- Recommendation for Indian businesses: Allow AI crawlers unless you have specific reasons to block them. Being cited in AI-generated answers (ChatGPT, Perplexity, Google AI Overviews) is a significant brand visibility opportunity. Blocking these crawlers means your content won’t be used — or cited — in AI answers.
Common Mistakes to Avoid
- Blocking CSS/JS in robots.txt: Google needs to render your pages. Blocking CSS or JavaScript files prevents proper rendering and hurts rankings.
- Using robots.txt to “noindex”: Blocking a page in robots.txt doesn’t prevent indexing. If you want to prevent indexing, use meta robots noindex tag — and DON’T block the page in robots.txt (Google must crawl the page to see the noindex tag).
- Conflicting directives: robots.txt blocks crawling AND the page has noindex tag = Google can’t see the noindex tag because it can’t crawl the page. The page might still appear in search results.
- Forgetting the staging site: After migrating from staging to production, ensure production robots.txt doesn’t carry over staging’s “Disallow: /” directive. This happens more often than you’d think.
- Overly aggressive blocking: Blocking entire directories that contain important content. Always review what’s inside a directory before blocking it.
Frequently Asked Questions
Does robots.txt prevent a page from being indexed?
No. This is the most common misconception. Robots.txt prevents crawling, not indexing. If other websites link to a page you’ve blocked in robots.txt, Google may still index the URL — it just won’t have the page’s content, showing a “No information available” snippet instead. To prevent indexing, use a meta robots noindex tag, and make sure the page is NOT blocked in robots.txt so Google can actually crawl and read that tag.
Should I block AI crawlers in robots.txt?
For most Indian businesses — no. Allowing AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) means your content can be cited in AI-generated answers, increasing brand visibility. Block AI crawlers only if your content is premium/paywalled, if you compete directly with AI tools, or if you have specific intellectual property concerns. The trend for most businesses is to welcome AI crawlers for visibility benefits.
How do I check if my robots.txt is configured correctly?
Use Google Search Console’s robots.txt Tester (under Settings → Crawl Stats → Open robots.txt). Enter URLs you want to test and verify they’re allowed or disallowed as intended. Also visit yoursite.com/robots.txt directly to see what’s being served. After any changes, use the “Request indexing” feature in Search Console URL Inspection for key pages to prompt Google to re-crawl.
Can Rank Math handle all robots.txt and meta robots needs?
Yes, for WordPress sites. Rank Math provides a robots.txt editor (General Settings → Edit robots.txt), global meta robots settings per post type/taxonomy (Titles & Meta), and per-page meta robots overrides (in the Rank Math meta box on each post/page). You shouldn’t need to manually edit robots.txt via FTP or add meta tags to your theme. One exception: if you need very complex robots.txt rules (regex-like patterns), you may need to edit the file directly.
What happens if I accidentally noindex my entire site?
Your pages will start disappearing from Google search results within days to weeks. Check immediately: visit any page → View Source → search for “noindex.” In WordPress, also check Settings → Reading → “Discourage search engines from indexing this site” — this common checkbox adds noindex globally. If accidentally enabled during staging and forgotten at launch, it blocks your entire site from indexing. Rank Math shows a warning if this setting is active. After fixing, request re-indexing via Search Console and expect 1-4 weeks for full recovery.
