Back to Tools
SEO

Robots.txt Generator

Create a robots.txt file to control how search engines crawl your website. Block sensitive areas and guide bots to your important content.

Quick add:

Only applied to wildcard (*) user-agent. Use cautiously as it can slow crawling.

Generated robots.txt

robots.txt
# robots.txt generated by BookingBomb.com
# https://bookingbomb.com/tools/seo/robots-txt-generator

User-agent: *
Disallow: /admin/
Place this file at the root of your domain: https://yourdomain.com/robots.txt

Understanding robots.txt for SEO

The robots.txt file tells search engine crawlers which pages or files they can or can't request from your site. It's not for hiding pages from Google (use noindex for that) but for managing crawler traffic.

How robots.txt Works

When a crawler visits your site, it first checks for a robots.txt file at your domain root. The file contains rules that tell the crawler which URLs it can access. Crawlers follow these rules, but they're not enforced—malicious bots may ignore them.

Key Directives

  • User-agent: Specifies which crawler the rules apply to. Use * for all crawlers.
  • Disallow: Tells crawlers not to access certain URLs or directories.
  • Allow: Permits access to a specific URL within a disallowed directory.
  • Sitemap: Points crawlers to your XML sitemap location.
  • Crawl-delay: Requests a delay between crawler requests (not supported by all bots).

What to Block

  • Admin areas: /admin/, /wp-admin/, /dashboard/
  • Internal search: Search result pages can create duplicate content.
  • User-specific pages: Cart, checkout, account pages.
  • Development/staging: Test environments shouldn't be indexed.
  • API endpoints: Unless you want them indexed.

What NOT to Block

  • CSS and JavaScript: Google needs these to render pages properly.
  • Images you want indexed: Don't block images unless necessary.
  • Important content: If it should rank, don't block it.

robots.txt vs. Noindex

robots.txt: Prevents crawling but doesn't prevent indexing. Pages can still appear in search results if linked from elsewhere.

Noindex tag: Prevents indexing. Use this to keep pages out of search results.

Blocking AI Crawlers

Many sites now block AI training bots like GPTBot (ChatGPT), anthropic-ai (Claude), and CCBot (Common Crawl). Add specific user-agent rules to control AI access to your content.

Testing Your robots.txt

  • Use Google Search Console's robots.txt Tester
  • Check that important pages aren't accidentally blocked
  • Test with different user-agents
  • Monitor crawl stats after making changes

Need Help With Technical SEO?

Our team can audit your technical SEO setup and ensure search engines can properly crawl and index your content.

Book Your Strategy Call