What Is Robots.txt? Crawl Rules Explained
robots.txt is a plain text file at your domain root that tells cooperating crawlers which URLs they may fetch. It controls crawling, not indexing: a blocked page can still appear in results, and noindex is the tool for removal. One wrong line such as Disallow: / can stop Google crawling your whole site.
For site owners and developers editing robots.txt on WordPress or a hosting panel, who want rules that block the right paths and nothing else.
knowledge
- Read time
- 4 min
- Sections
- 8
- Updated
- 14 Nov 2026
- Written by
- WebJi® team
Last editorial update: 14 November 2026.
What is robots.txt and where does it live?
robots.txt is a plain text file at the root of a host — example.in/robots.txt — that tells cooperating crawlers which paths they may request; it manages crawling, not indexing or removal. The format is standardised as the Robots Exclusion Protocol in RFC 9309 (September 2022), and crawlers fetch it before crawling other URLs on that host.
- One per host: blog.example.in needs its own file; the one at example.in does not cover subdomains.
- Exact path only: a robots.txt inside a subfolder is ignored.
- Size limit: Google reads the first 500 KiB and ignores the rest.
- Caching: Google generally caches the file for up to 24 hours, so changes are not instant.
A missing file (404) means the site is treated as fully crawlable. A server error (5xx) is different: Google pauses crawling while the file fails, so an outage on that one URL can stall the whole site.
What does a working robots.txt look like?
A typical WordPress robots.txt closes the admin area, re-opens the AJAX endpoint the public site uses, blocks internal search and comment-reply URLs, and names the sitemap.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=
Sitemap: https://example.in/sitemap_index.xmlLines under one User-agent apply until the next User-agent group. The Sitemap line sits outside any group and must be a full URL; what the sitemap itself does is covered in what an XML sitemap does. WebJi®'s robots.txt generator builds a starting file you can edit.
How do User-agent, Allow and Disallow rules apply?
User-agent names the crawler a group applies to (* means all), Disallow lists paths not to fetch, and Allow carves an exception inside a blocked path; when rules conflict, the most specific (longest) match wins.
| Line | Effect |
|---|---|
Disallow: (empty) | Allows everything |
Disallow: / | Blocks the entire site |
Disallow: /*.pdf$ | Blocks URLs ending in .pdf (* and $ are supported by Google and Bing) |
Allow: /wp-admin/admin-ajax.php | Longer, more specific than /wp-admin/, so it wins |
Crawl-delay: 10 | Honoured by some crawlers such as Bing; ignored by Google |
Paths are case-sensitive, so /Admin/ and /admin/ are different rules.
What is the difference between robots.txt and noindex?
robots.txt stops a crawler fetching a URL, while a noindex meta tag or X-Robots-Tag header lets it fetch the page and tells it not to keep the page in results. A URL blocked in robots.txt can never be recrawled to see a noindex tag, and it can still appear in results, without a description, if other sites link to it.
| Tool | Controls | Use it to |
|---|---|---|
| robots.txt Disallow | Crawling | Save crawl effort on search results, filters, admin paths |
<meta name="robots" content="noindex"> | Indexing | Keep thank-you pages, staging or thin pages out of results |
| Password protection | Access | Keep genuinely private content private |
To remove an already-indexed page: allow the crawl, add noindex, wait for Google to recrawl, and use Search Console's Removals tool for a temporary hide in the meantime. Duplicate-URL handling is a separate tool, explained in what is a canonical URL.
How do you edit robots.txt on WordPress or a hosting panel?
On most hosting, edit the physical file in your control panel's File Manager inside the site's web root (usually public_html); on WordPress without a physical file, the CMS prints a virtual one, so edit it through your SEO plugin.
WordPress core's virtual file disallows /wp-admin/, allows /wp-admin/admin-ajax.php and, since WordPress 5.5, adds a Sitemap: line for /wp-sitemap.xml. If "Discourage search engines from indexing this site" is ticked under Settings › Reading, WordPress changes its output to discourage crawling — a common leftover from staging. SEO plugins such as Yoast or Rank Math replace the virtual output with their own rules. If a physical file exists in the web root, it overrides the virtual one entirely, so check which version the browser actually returns.
Which robots.txt mistakes cost traffic?
The costliest mistakes are shipping a staging Disallow: / to the live site and blocking CSS or JavaScript, which stops Google rendering pages properly.
- Copying
Disallow: /from a staging copy to production, which halts crawling of the whole site. - Blocking
/wp-content/or theme folders, hiding the CSS and scripts Google needs to render the page. - Blocking a page to hide it, when noindex is the tool that removes it from results.
- Treating robots.txt as private: anyone can read it and learn the paths you wanted hidden.
- Assuming every bot obeys: scrapers ignore the file, and it is a request, not access control.
For AI crawlers, separate tokens exist: for example GPTBot for OpenAI and Google-Extended, which controls use of content for Google's Gemini models without affecting Google Search crawling. WebJi®'s SEO services check these rules against Search Console before a migration goes live.
How do you test a robots.txt change?
Load the live file to confirm it returns your new lines with a 200 status, then check the robots.txt report in Google Search Console and inspect one URL you meant to block and one you meant to keep.
curl -sI https://example.in/robots.txt | head -1 # expect HTTP/2 200
curl -s https://example.in/robots.txt # read the served rulesIf a page you meant to keep shows "Blocked by robots.txt" in URL Inspection, the rule is too broad; fix it before waiting for a recrawl. The broader crawl checklist is in technical SEO.
Frequently asked questions
Does every website need a robots.txt file?
No. Without one, crawlers treat the site as fully open, which is fine for many small sites. Add one when you need to keep crawlers out of search results, filters or admin paths, or to point them to your sitemap.
Will robots.txt hide my page from Google?
Not reliably. A blocked URL can still be indexed from links elsewhere, usually shown without a description. Use noindex with crawling allowed, or password protection for private content.
Can I block a specific bot?
Yes, by giving it its own group, for example User-agent: GPTBot followed by Disallow: /. This only works for bots that follow robots.txt; abusive scrapers need firewall or rate-limit rules instead.
How long does a robots.txt change take to apply?
Google usually refreshes its cached copy within about a day. You can request a recrawl of the file from Search Console's robots.txt report if a fix is urgent.
Checked against RFC 9309 and Google Search Central documentation, 15 November 2026 · Reviewed by the WebJi® team, Jaipur.
Ready to get started with WebJi®?
Join 12,000+ businesses across India and overseas. INR pricing, 18% GST shown before payment and 24/7 email support in Hindi and English — average first reply under 60 minutes.