Robots.txt Generator

Web & Network

Construct standards-compliant robots.txt files with disallow directives, sitemap URLs, and AI bot blocking rules.

General Crawlers
Default Policy (User-agent: *)
Allow standard search engines by default
Sitemap URL
Disallowed Directories & Paths
/api/
/admin/
AI Bot Blocking Rules
Preview robots.txt
# robots.txt generated via BitSmith Tools

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

# AI Crawlers & Scrapers Disallow Rules
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://bitsmith.online/sitemap.xml

How to Use Robots.txt Generator

  1. 1Step 1

    Specify your global crawler policy and enter your canonical sitemap URL.

  2. 2Step 2

    Add any private internal directories (/admin/, /api/) to the Disallow list.

  3. 3Step 3

    Toggle switches for AI training crawlers you wish to block from scraping your content.

  4. 4Step 4

    Download the completed robots.txt file and place it in your website root.

Technical Specification & Client Security

Generated files follow RFC 9309 standards. Directive precedence and path wildcards are formatted for compatibility across Googlebot, Bingbot, and AI user-agents.

›Pre-configured toggles to block AI training scrapers (GPTBot, ClaudeBot, CCBot, Bytespider)
›Generates standards-compliant User-agent, Allow, Disallow, and Sitemap directives
›Dynamic path exclusion manager for private administrative and API routes
›Real-time text preview matching official RFC 9309 Robots Exclusion Protocol
›1-click direct robots.txt file download ready for your web server root
›100% private in-browser generation with zero network data transmission

Input & Output Transformation Examples

Standard Search Crawler with Sitemap
Input:
Policy: Allow all, Sitemap: https://bitsmith.online/sitemap.xml
Output:
User-agent: *
Allow: /

Sitemap: https://bitsmith.online/sitemap.xml
Block Generative AI Scrapers
Input:
Block GPTBot and ClaudeBot
Output:
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Frequently Asked Questions

Where should I place the robots.txt file?

The robots.txt file must be uploaded to the root directory of your website so it is accessible at https://yourdomain.com/robots.txt.

Do AI crawlers respect robots.txt?

Major reputable AI providers including OpenAI (GPTBot), Anthropic (ClaudeBot), and Common Crawl (CCBot) honor Disallow directives published in robots.txt.

Does robots.txt hide pages from appearing in Google search results?

No. To guarantee a page is excluded from Google search index, use a <meta name="robots" content="noindex" /> header tag in addition to robots.txt.

Related Developer Utilities