Robots.txt Generator and Validator (Free)

Use this robots.txt generator to build a file with a rule builder and presets, validate an existing file line by line, and test any URL against it for Googlebot, GPTBot or any other crawler. Free, and it all runs in your browser.

By Troopus · Updated: October 24, 2026

Rule builder and presets
Line-by-line validator
URL tester
Block AI bots
100% private

Robots.txt generator and validator: build and test your file

Start from a preset:

Sitemap URLs (one per line)Must be full URLs. The Sitemap line is independent of any user-agent group.

Your robots.txt

Paste a robots.txt file

—Paste a file to validate it

Test URLs against this file

User-agent to testExamples: Googlebot, Googlebot-Image, GPTBot, bingbot, or * for the default group.

URLs or paths (one per line)

100% private: your text stays in this browser. The tester follows Google's documented matching rules, but it cannot fetch your live file or know how other crawlers behave.

How do I create a robots.txt file for free?

To create a robots.txt file, write a plain text file named robots.txt with one or more groups, each starting with a User-agent line followed by Allow and Disallow rules, then upload it to the root of your domain. Use the generator above, copy the result, and place it at https://yourdomain.com/robots.txt.

Pick a preset (Allow all, Block all, WordPress, Shopify or Block AI bots), then adjust the groups. Each group has a user-agent, any number of Allow or Disallow paths and an optional Crawl-delay. Add your sitemap URLs below, and the output updates as you type. Press Copy or Download, then save the file as robots.txt in UTF-8.

Where you upload it matters. Crawlers only look at the top level of a host, so https://example.com/robots.txt works and https://example.com/blog/robots.txt does not. Each subdomain and each protocol needs its own file: shop.example.com is not covered by the file on example.com.

Robots.txt syntax: User-agent, Allow, Disallow, Sitemap

Robots.txt has four directives that Google supports: User-agent names the crawler, Disallow blocks a path, Allow re-opens a path inside a blocked one, and Sitemap points to your XML sitemap. Rules are grouped under a User-agent line, and field names are case-insensitive while paths are case-sensitive.

Robots.txt directives and how Google treats them
Directive	What it does	Example
User-agent	Starts a group and names the crawler. `*` means every crawler without its own group.	`User-agent: Googlebot`
Disallow	Blocks URLs that start with the path. An empty value blocks nothing.	`Disallow: /cart/`
Allow	Permits a path, usually an exception inside a blocked folder.	`Allow: /cart/help.html`
Sitemap	Full URL of a sitemap. Applies to the whole file, not one group.	`Sitemap: https://example.com/sitemap.xml`
Crawl-delay	Seconds between requests. Google ignores it; some other crawlers honor it.	`Crawl-delay: 10`

Two wildcards work in paths: * matches any run of characters and $ marks the end of the URL. So Disallow: /*.pdf$ blocks every URL ending in .pdf, while Disallow: /*?sessionid= blocks URLs with that parameter.

When rules conflict, the most specific one wins, meaning the longest matching path. If an Allow and a Disallow match with equal length, Google applies the less restrictive Allow. Google reads at most 500 KiB of the file and ignores the rest.

How does the robots.txt tester check if a URL is blocked?

To test whether a URL is blocked by robots.txt, paste your file into the validator tab, choose a user-agent such as Googlebot, and enter the URL or path. The tester finds the matching group, applies the longest-match rule and shows Allowed or Blocked together with the exact line that decided it.

Take this file as an example: Disallow: /private/ and Allow: /private/public-report.html, both under User-agent: *. The path /private/data.html is blocked by the first rule (9 characters). The path /private/public-report.html matches both, but the Allow is longer (28 characters), so it is allowed. Load the example in the tool to see exactly this.

Test many URLs at once, one per line, and switch the user-agent to compare crawlers. A crawler with no group of its own falls back to the * group. Googlebot-Image, Googlebot-News and Googlebot-Video fall back to Googlebot first.

This is a local tester that follows Google's documented rules. It cannot download your live file, so paste the current contents. For the final word on what Google sees, check the robots.txt report in Search Console.

Once crawling is sorted, make sure the pages you do want found are listed in your sitemap. The sitemap URL extractor pulls every URL out of a sitemap so you can test them in bulk here.

Robots.txt for WordPress, Shopify and ecommerce: examples

A typical WordPress robots.txt blocks /wp-admin/ but allows /wp-admin/admin-ajax.php, then lists the sitemap. Ecommerce sites usually also block cart, checkout, account and internal search paths, plus filter or sort parameters, so crawlers spend their time on products and categories.

WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml

The Allow line matters because some themes and plugins load content through admin-ajax.php. WordPress core generates /wp-sitemap.xml; SEO plugins often use their own sitemap URL, so check yours.

Shopify and other stores

Shopify creates a default robots.txt for you and lets you customise it with a robots.txt.liquid template. The Shopify preset in the tool lists typical store rules (admin, cart, checkout, orders, account and sorted collection URLs). Treat it as a starting point and compare it with your own store file.

Ecommerce rules worth considering

Common blocks are /cart, /checkout, /search and sort or filter parameters such as /*?sort=. Blocking stops crawling, but URLs that are already indexed can stay in the index.

Block AI crawlers robots.txt rules: GPTBot, ClaudeBot and more

To block AI crawlers, add a separate User-agent group for each bot you want to stop, such as GPTBot or ClaudeBot, followed by Disallow: /. Compliant bots honor it. Blocking training bots does not remove you from Google Search, because that is controlled by Googlebot, not Google-Extended.

Use the "Block AI bots" preset, which creates one group listing several user-agents with Disallow: / and leaves everyone else allowed. These are the tokens documented by their operators:

AI-related crawler tokens and what blocking them does
Token	Operator	What it is for
GPTBot	OpenAI	Crawls content that may be used to train its models
ChatGPT-User	OpenAI	Fetches a page when a user asks; OpenAI says robots.txt rules may not apply
ClaudeBot	Anthropic	Collects content that may be used for training
Claude-User	Anthropic	Retrieves pages when a user asks
Google-Extended	Google	Controls use of content for Gemini training and grounding; not a ranking signal

Blocking a training bot is a signal about future training. Blocking a search or user-triggered bot (OAI-SearchBot, Claude-SearchBot, Claude-User) can reduce how often AI products show or cite your pages. The preset blocks training-oriented tokens only, plus CCBot, Applebot-Extended, Bytespider and meta-externalagent. Delete any you want to allow.

Robots.txt is a request, not a lock. Well-behaved crawlers follow it; others may not. To truly keep content away from a bot, use authentication or server-level blocking.

Common robots.txt mistakes

The most damaging robots.txt mistakes are leaving Disallow: / on a live site, blocking CSS and JavaScript, and using robots.txt to keep pages out of search. A blocked page can still be indexed from links without a snippet. Use a noindex tag on a crawlable page to remove it.

Shipping a staging file to production. User-agent: * with Disallow: / blocks everything. The validator warns about it.
Blocking resources Google needs to render. Disallowing CSS, JavaScript or whole theme folders can hurt how pages are understood.
Using noindex in robots.txt. Google stopped supporting it on September 1, 2019. Use a meta robots tag or X-Robots-Tag header.
Blocking a page you want deindexed. If Google cannot crawl the page, it cannot see the noindex tag.
Wrong location or case. It must be /robots.txt at the host root, in UTF-8, and /Private/ differs from /private/.
Relative sitemap URLs. Sitemap lines need the full https:// address.

Paste your file into the validator to catch each of these, then confirm in Search Console. For pages that need a good snippet once crawled, the title and meta description checker is the next stop.

Frequently asked questions

What is a robots.txt file?

A robots.txt file is a plain text file at the root of a website that tells crawlers which URLs they may or may not request. It is part of the Robots Exclusion Protocol, standardized as RFC 9309, and compliant crawlers check it before crawling.

How do I create a robots.txt file?

Create a UTF-8 text file named robots.txt, add User-agent lines with Allow and Disallow rules, and optionally a Sitemap line. Upload it to your domain root. The generator above builds the file for you and lets you copy or download it.

Where do I put robots.txt?

Place it in the top-level directory of your host, so it loads at https://example.com/robots.txt. A copy in a subfolder is ignored. Subdomains and different protocols or ports need their own robots.txt file.

How do I use the robots.txt validator to test my file?

Paste the file into the validator tab, pick a user-agent and enter the URLs you care about. The tool shows Allowed or Blocked and the deciding rule. For what Google actually fetched, check the robots.txt report in Google Search Console.

Does robots.txt stop a page being indexed?

No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it, usually without a description. To keep a page out of search, allow crawling and use a noindex robots meta tag or header.

How do I block AI bots like GPTBot?

Add a group such as User-agent: GPTBot followed by Disallow: / and repeat for other bots like ClaudeBot. Compliant crawlers will honor it. The Block AI bots preset does this for several training-related tokens in one step.

Is there a robots txt checker for Googlebot rules?

Yes. The validator tab works as a robots txt checker: paste your file, pick Googlebot or another user-agent, and see which rule allows or blocks each URL. It also warns about syntax problems such as a missing colon, an unknown directive or a path without a leading slash.

Does Google respect Crawl-delay in robots.txt?

No. Google ignores the Crawl-delay directive. Some other crawlers, such as Bing, support it, and Anthropic says it supports it for its bots. The generator includes the field for those crawlers and the validator notes that Google ignores it.