How the Robots.txt Generator works
A robots.txt file is made of groups. Each group starts with one or more User-agent lines naming the
crawlers it applies to, followed by Allow and Disallow rules. A crawler uses the group that
names it most specifically and ignores the others; the * group is for every crawler without its own group.
Google, Bing and other major search engines choose the most specific (longest) matching rule, and Allow wins a tie.
The generator writes Allow lines before Disallow lines so that older crawlers that use the first matching rule reach
the same result.
Paths must start with / or *. In a path, * matches any sequence of characters
and a trailing $ matches the end of the URL, so /*.pdf$ covers every PDF. A group with no rules
gets an empty Disallow:, which blocks nothing. Crawl-delay asks crawlers to wait between requests;
Bing and Yandex honour it, but Google ignores it. Sitemap lines must be absolute URLs and apply to all
crawlers.
The Block AI crawlers preset keeps the site open to search engines and disallows known AI data crawlers: GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended, Applebot-Extended, CCBot (Common Crawl), PerplexityBot, Bytespider and meta-externalagent. robots.txt is a request, not access control: well-behaved crawlers follow it, but it does not stop anyone from fetching a URL.
How to use the Robots.txt Generator
- Start from a preset: Allow all, Disallow all or Block AI crawlers.
- Edit the group: list the user agents, then the Disallow and Allow paths, one per line. Add a Crawl-delay if you need one.
- Select Add group for crawlers that need different rules, and enter your sitemap URLs.
- Read the explanation table, then select Copy or Download and upload the file as robots.txt to the root of your site.
Example
A * group that disallows /admin/ and /cart but allows /admin/help/,
a second group that blocks GPTBot, and a sitemap URL produce:
User-agent: *
Allow: /admin/help/
Disallow: /admin/
Disallow: /cart
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlIf a Disallow path is written as admin, the tool reports Group 1, Disallow line 1: "admin" must start with / or *.
Common use cases
- Keeping crawlers out of admin pages, carts, internal search results and other pages with no search value.
- Asking AI training crawlers such as GPTBot and CCBot not to crawl a site while search engines still can.
- Pointing crawlers to the XML sitemap of a new website.
- Blocking a whole staging site from being crawled.
Frequently asked questions
Where do I put robots.txt?
At the root of the host, for example https://www.example.com/robots.txt. Crawlers do not look for it in subfolders, and each subdomain (such as shop.example.com) needs its own file.
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still appear in results if other pages link to it. To keep a page out of results, let it be crawled and add a noindex meta tag, which you can create with the Meta Tag Generator.
Does Google follow Crawl-delay?
No. Google ignores the Crawl-delay line and adjusts its crawl rate automatically. Bing and Yandex do honour it. The tool includes the line only if you enter a value.
How do * and $ work in paths?
* matches any sequence of characters and $ matches the end of the URL. For example, Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*?sort= blocks URLs whose query string starts with sort=. Every path must start with / or *; the tool reports any line that does not.
Does blocking AI crawlers stop all AI use of my content?
It asks the listed crawlers not to fetch your pages, and the major AI companies say they respect it. It cannot stop crawlers that ignore robots.txt or content that was collected earlier, and new crawler names appear over time.
Is anything I type sent to your server?
No. The file is generated in your browser.