Robots.txt Generator
Construct standards-compliant robots.txt files with disallow directives, sitemap URLs, and AI bot blocking rules.
# robots.txt generated via BitSmith Tools User-agent: * Allow: / Disallow: /api/ Disallow: /admin/ # AI Crawlers & Scrapers Disallow Rules User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / Sitemap: https://bitsmith.online/sitemap.xml
How to Use Robots.txt Generator
- 1Step 1
Specify your global crawler policy and enter your canonical sitemap URL.
- 2Step 2
Add any private internal directories (/admin/, /api/) to the Disallow list.
- 3Step 3
Toggle switches for AI training crawlers you wish to block from scraping your content.
- 4Step 4
Download the completed robots.txt file and place it in your website root.
Technical Specification & Client Security
Generated files follow RFC 9309 standards. Directive precedence and path wildcards are formatted for compatibility across Googlebot, Bingbot, and AI user-agents.
Input & Output Transformation Examples
Policy: Allow all, Sitemap: https://bitsmith.online/sitemap.xml
User-agent: * Allow: / Sitemap: https://bitsmith.online/sitemap.xml
Block GPTBot and ClaudeBot
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
Frequently Asked Questions
Where should I place the robots.txt file?
The robots.txt file must be uploaded to the root directory of your website so it is accessible at https://yourdomain.com/robots.txt.
Do AI crawlers respect robots.txt?
Major reputable AI providers including OpenAI (GPTBot), Anthropic (ClaudeBot), and Common Crawl (CCBot) honor Disallow directives published in robots.txt.
Does robots.txt hide pages from appearing in Google search results?
No. To guarantee a page is excluded from Google search index, use a <meta name="robots" content="noindex" /> header tag in addition to robots.txt.