About this tool

Your robots.txt tells search engines and AI crawlers which parts of your site they can access. This tool lets you start from your real file rather than a blank page: enter your domain and it fetches your live robots.txt so you can edit exactly what is there today. Add or change rules per crawler, manage the AI crawlers by name, list your sitemaps, and test whether any URL is allowed or blocked before you publish.

The AI crawler list is kept current, covering the agents that matter now: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini), PerplexityBot, CCBot, and more. One click adds rules to allow them all (if you want to be found in AI answers) or block them all (if you would rather your content was not used by AI).

Use cases

• Edit your live file safely. Import your current robots.txt, make changes, and test them before they go live.

• Manage AI crawler access. Allow the major AI crawlers to help your content surface in ChatGPT, Perplexity and Gemini, or block them to opt out, using current agent names.

• List your sitemaps. Add one or more sitemap URLs so crawlers can find your content quickly.

• Test before you publish. Check whether a specific URL is allowed or blocked for a given crawler, and test URLs in bulk.

• Launches and migrations. Generate a correct robots.txt for a new site, or clean up an old one during a rebuild.

How to generate a robots.txt file

1. Import your live file (optional). Enter your domain and select Import to pull in your current robots.txt. Leave it blank to start from scratch.

2. Edit your rules. Set the crawler, choose Allow, Disallow or Crawl-delay, and enter the path.

3. Manage AI crawlers. Use Allow AI Crawlers or Block AI Crawlers to add rules for the major AI bots in one click.

4. Add your sitemaps. Enter your sitemap URLs, one per line.

5. Test your rules. On the Test Rules tab, check whether a URL is allowed or blocked for a given crawler.

6. Copy or download your robots.txt and upload it to your site root at /robots.txt.

FAQs

Can I import my existing robots.txt?

Yes. Enter your domain and the tool fetches your live robots.txt so you can edit the exact rules you already have. Your user-agents, sitemaps and crawl rules are all imported.

Which AI crawlers can I control?

The current major AI crawlers, including GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-User (Anthropic), Google-Extended (Gemini), PerplexityBot, CCBot, Bytespider, Amazonbot, Applebot-Extended and Meta-ExternalAgent.

Should I allow or block AI crawlers?

If you want your brand to be found and cited in AI answers, allow them. Only block them if you specifically do not want your content used by AI tools. Most brands aiming for visibility should allow the AI crawlers.

Does robots.txt guarantee a page stays private?

No. Robots.txt asks well-behaved crawlers not to access a page, and most reputable ones comply, but a disallowed page can still be indexed if other sites link to it. For genuinely private content, use proper access controls, not robots.txt alone.

Can I test my rules before publishing?

Yes. The Test Rules tab lets you check whether a specific URL is allowed or blocked for a chosen crawler, and you can test multiple URLs at once.

Where does the file go?

Upload it to the root of your domain so it is available at yourdomain.com/robots.txt. That is the only location crawlers check.

Is it free?

Yes, free with no signup. If you would like help getting found across search and AI, from technical setup to content and authority, contact [email protected] or book a call with Growthack.