About the Robots.txt Generator
Tezmy's Robots.txt Generator writes the small text file that tells search engine crawlers which parts of your site they may visit. Choose to allow crawling or block the whole site, list folders or pages to keep crawlers out of, add exceptions and your sitemap address, and set a crawl delay if you need one. One tick blocks the crawlers that collect content to train AI models, such as OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended and Common Crawl, without affecting normal search results. The file updates as you go, ready to copy or download. A built-in tester checks any page against your rules for Googlebot, Bingbot or an AI crawler, using the same matching rules Google uses.
Updated September 2026
How to use the Robots.txt Generator
- 1Choose Allow crawling (most sites) or Block the whole site (for private or test sites).
- 2List the paths to block, such as /admin/ or /cart, or use a quick preset.
- 3Add your sitemap address, and tick Block AI training crawlers if you want.
- 4Copy or download robots.txt, upload it to the top of your site, and test pages below.
What to block, and what not to
- Block: admin and login areas, internal search results, cart and checkout pages, filtered or sorted duplicates.
- Don't block: CSS and JavaScript files; Google needs them to render your pages properly.
- Don't block pages you want out of Google with robots.txt; use noindex instead.
- Don't list secret folders: robots.txt is public, so it tells everyone where they are.
Test before and after uploading
A single wrong line, such as Disallow: / under User-agent: *, can hide a whole site from search. Use the tester to check your most important pages (home page, top products, blog posts) are Allowed for Googlebot, and that private areas are Blocked. After uploading, open yoursite.com/robots.txt in a browser to confirm it is live, and check Google Search Console's robots.txt report.
More SEO basics
- Write titles and descriptions with the Meta Tag Generator.
- Keep page addresses short and clean with the Slug Generator.
- Check how your content uses its main topic with the Keyword Density Checker.
Features
- Allow or block crawling in one click
- Block paths, with * and $ patterns
- Allow exceptions
- Block AI training crawlers (GPTBot, ClaudeBot, Google-Extended…)
- Sitemap and crawl-delay lines
- Presets for WordPress, shops and private areas
- Tester using Google's longest-match rule
Frequently asked questions
What is robots.txt?
A plain text file at the top of a website (yoursite.com/robots.txt) that tells crawlers such as Googlebot which pages they may request. Well-behaved crawlers read it before crawling; it is part of the Robots Exclusion Protocol, now an internet standard (RFC 9309).
Does robots.txt stop a page appearing in Google?
Not reliably. It stops Google crawling the page, but the address can still appear in results if other sites link to it. To keep a page out of search, allow crawling and add a noindex meta tag, or protect the page with a password.
How do I block ChatGPT and other AI crawlers?
Tick “Block AI training crawlers”. It adds rules for GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Gemini), Applebot-Extended, CCBot (Common Crawl), Bytespider and Meta-ExternalAgent. These companies say their crawlers follow robots.txt. Normal Googlebot and Bingbot search crawling is unaffected.
What do * and $ mean in robots.txt?
* matches any characters, and $ marks the end of the address. So Disallow: /*.pdf$ blocks every address ending in .pdf, and Disallow: /*?sort= blocks addresses with a sort parameter.
If Allow and Disallow both match, which wins?
The most specific rule, meaning the longest path, wins. If an Allow and a Disallow rule are the same length, Allow wins. So Disallow: /admin/ with Allow: /admin/help/ lets crawlers into the help pages only. The tester applies exactly this rule.
Where do I put the robots.txt file?
In the root of your domain, so it opens at https://yoursite.com/robots.txt. Each subdomain (such as shop.yoursite.com) needs its own. On WordPress, Shopify and other platforms, use the SEO plugin or settings page to edit it.
Should I add my sitemap to robots.txt?
Yes, it's a simple way to tell every search engine where your sitemap is. Use the full address, such as https://yoursite.com/sitemap.xml. You can list several sitemaps.