Robots.txt Generator

Create robots.txt files to control search engine crawlers.

Default Access

Allow AllAll bots can crawl everything Block AllBlock all search engines Custom RulesDefine specific rules

Disallow Paths

Common paths: /admin/, /wp-admin/, /private/, /cgi-bin/

Sitemap URL

Crawl Delay (optional)

Bot-Specific Rules

Block ChatGPT/GPTBot Block Google Bard Block CCBot (Common Crawl)

Generated robots.txt