Generate a robots.txt file for your website to control search engine crawling.
Runs entirely in your browser.
robots.txt File Generation Tool
User Agents
All bots (*) Googlebot Bingbot GPTBot (OpenAI) ClaudeBot (Anthropic) Twitterbot Facebookbot
Default Rule for All Bots
Allow all Disallow all Custom paths
Disallow Paths
Common:
/admin/ /private/
Allow Paths
Sitemap
Sitemap URL
Crawl Delay
Delay (seconds, 0 = none)
Block AI Bots
Block all known AI crawlers
Generated robots.txt
About robots.txt
robots.txt is a file that tells search engines which pages of your site they are allowed to crawl.