Robots.txt Generator
A robots.txt generator writes the plain-text file crawlers fetch at your domain root before anything else. User-agent names the crawler, Disallow blocks paths by prefix, Allow opens exceptions inside blocked folders, and Sitemap points at your XML sitemap. Deploy it at /robots.txt and verify the fetch in Search Console.
For site owners and SEOs who need a valid robots.txt file without hand-writing syntax.
needed
- Price
- Free
- Signup
- None
- Built by
- WebJi®
- Support
- 24/7 email
Last editorial update: 15 November 2026.
Set the default rule, add the paths crawlers should skip, declare your sitemap, and the generator writes a valid robots.txt you can copy or download. It runs entirely in your browser — nothing is uploaded, and the result is plain text you can read before you deploy it.
Build your robots.txt
Runs in your browserUpload the file to your site root so it answers at /robots.txt. Crawl rules are not de-indexing: to remove a page from results use noindex on the page itself, and Search Console verifies the outcome. Need the stack that runs it? See cPanel hosting.
What each line of the generated robots.txt means
A robots.txt file is a short set of instructions that crawlers fetch from your domain root before they request anything else, written with five kinds of line. The protocol they follow is standardised as the Robots Exclusion Protocol, RFC 9309, and every major search engine's crawler reads the same grammar.
| Line | What it tells a crawler |
|---|---|
User-agent: * | The rules below apply to every crawler that reads the file |
Disallow: /wp-admin/ | Do not fetch any URL whose path starts with this prefix |
Allow: /wp-admin/admin-ajax.php | An exception: this exact path inside a blocked folder may be fetched |
Crawl-delay: 5 | Wait this many seconds between requests — honoured by Bing and most engines, ignored by Google |
Sitemap: https://yourdomain.com/sitemap.xml | The absolute URL of your XML sitemap, so crawlers find the curated URL list without guessing |
Matching is prefix-based and case-sensitive: Disallow: /Cart/ does not block /cart/. An empty Disallow: line with nothing after it means "block nothing" — the opposite of what most people assume, which is why "Block everything (staging sites)" writes Disallow: / and not an empty line. The "Block Googlebot separately" option writes a second User-agent: Googlebot group ahead of the general one, because a crawler applies the most specific group that names it.
The file only counts at one address: the domain root. It must be named robots.txt in lowercase and answer at https://yourdomain.com/robots.txt; a copy in a subfolder or with different casing is never read.
Disallow is not noindex — the trap the file sets
Disallow stops the fetch, not the listing: a blocked URL can still appear in Google's results without a snippet if other pages link to it. That is the single most consequential distinction in robots.txt work, and it sends site owners down the wrong path every week.
- Disallow — the crawler skips the URL entirely. Google never reads its content, so a
noindextag on that page is never seen. You have blocked the removal order itself. - noindex — a page-level instruction that must be crawled to be obeyed. Pages you want out of search results get
noindexand stay crawlable. - Right use of Disallow — paths that waste crawl budget or should never be public-facing:
/cart/,/wp-admin/, internal search results, staging folders.
The decision therefore splits in two: the crawl rules live in robots.txt, the index rules live in each page's meta settings. Google documents this split, and Search Console's URL inspection shows which state a given URL is actually in — check there after every change rather than assuming. The long-form version is in the what robots.txt is guide.
Limits of this robots.txt generator — what it does not do
The generator writes syntactically valid rule groups for one site; it does not audit your existing file, validate your paths against a live crawl, or know which bot names are current. Three limits matter before you deploy the output.
- It is advisory, not enforcement. robots.txt is a convention polite crawlers follow; scrapers and malicious bots ignore it. Never use it to hide private content — put those behind authentication.
- Size and specificity. Google reads at most the first 500 KiB of a robots.txt file and ignores anything beyond it; with hundreds of rules, prune instead of growing. The generator also writes one general group plus an optional Googlebot group — a site needing per-bot rules for many crawlers should extend the file by hand.
- Rules can lock you out. "Block everything" is meant for staging servers. Left on a live site it drops the whole domain from crawling within days. Always verify in Search Console after deploying.
For the sitemap line, generate the URL list first with the XML sitemap generator and paste its location here — the two files work as a pair.
Deploying and verifying the generated file
Deploy in three steps: upload to the domain root, confirm the file answers at /robots.txt, then request the fetch in Search Console. On WebJi® hosting the upload is a file-manager or FTP job of one small text file — the same step as publishing any static file.
- Upload: place
robots.txtin the web root (public_html on shared hosting, the document root on a VPS). - Confirm: open
https://yourdomain.com/robots.txtin a private window; you should see the exact text you generated. - Verify: in Google Search Console, the robots.txt report shows the last fetch and flags syntax problems; re-submit after each edit so Google picks up the new version instead of its cached copy.
- Pair it: reference your sitemap, then submit that sitemap in Search Console — the XML sitemap guide covers the submission half.
If a live business site should change its crawl rules and you would rather have it done for you, the knowledgebase carries step-by-steps, and WebJi® support (support@webji.services, 24/7 in Hindi and English) can place the file for you on a managed plan.
Frequently asked questions
How do I create a robots.txt file for my website?
Fill in the generator above — default rule, disallow paths, optional sitemap URL — then copy or download the output, save it as robots.txt in lowercase, and upload it to your domain root so it answers at /robots.txt. Verify the fetch in Search Console afterwards.
What should I disallow in robots.txt?
Paths that waste crawl budget or serve no search intent: /wp-admin/, /cart/, internal search results, and staging folders. Add an Allow: line when something inside a blocked folder must stay fetchable, such as /wp-admin/admin-ajax.php, which many WordPress themes call from the front end.
Is robots.txt disallow the same as noindex?
No. Disallow stops the crawler from fetching the URL; the URL can still be listed without a snippet if other pages link to it. Noindex removes a page from results but must be crawled to be read — so never disallow a page that carries a noindex, or Google can never see the removal instruction.
Does Crawl-delay slow down Google's crawler?
No. Google ignores the Crawl-delay directive entirely, so the generator's note marks it as a Bing-and-others setting. If Googlebot's rate is the concern, the crawl-rate setting in Search Console is the lever that actually affects it.
How do I block AI bots like GPTBot in robots.txt?
Add a User-agent: line naming that bot followed by a Disallow: rule, exactly as the Googlebot option demonstrates. Bot names change as new crawlers launch, and this page does not publish a current list — check the operator's own crawler documentation for the exact token before adding a group.
Can robots.txt remove pages from Google?
Not by itself. A disallowed URL can still appear in results without content when links point at it. To remove a page, give it a noindex and keep it crawlable; use Search Console's removals tool for urgent cases, and robots.txt only for crawl-budget control.
Last editorial update: 15 November 2026.
Ready to get started with WebJi®?
Join 12,000+ businesses across India and overseas. INR pricing, 18% GST shown before payment and 24/7 email support in Hindi and English — average first reply under 60 minutes.