utils logo UTILS
Free online tools to generate, calculate,
convert, format, encode, and play.
 

robots.txt Generator & Analyzer

Generate robots.txt files to control search engine crawlers, or analyze and validate existing ones with syntax checking and URL path testing.

Quick Templates

Custom Configuration

Use * for wildcards, e.g., *.pdf or /temp/*
Delay between requests (not supported by all crawlers)

Robots.txt Content

About robots.txt

The robots.txt file tells search engine crawlers which pages or sections of your site they can access. It's placed in the root directory of your website (e.g., https://example.com/robots.txt). This tool parses robots.txt content line by line and validates the syntax against the RFC 9309 Robots Exclusion Protocol standard.

Common Directives

  • User-agent: Specifies which crawler the rules apply to
  • Disallow: URLs that should not be crawled
  • Allow: Exceptions to disallow rules
  • Sitemap: Location of XML sitemaps
  • Crawl-delay: Time between requests (seconds)

Best Practices

  • Place robots.txt in your site's root directory
  • Always include a wildcard (*) user-agent group
  • Include sitemap references for better crawl coverage
  • Don't use robots.txt for security (use proper auth)
  • Keep the file under 500 KB (Google's limit)
  • Test rules before deploying to production

What Gets Checked

  • Syntax validation — Each line is checked for valid directive format
  • Directive ordering — User-agent must precede Allow/Disallow rules
  • Wildcard patterns — Validates * and $ pattern usage
  • Sitemap URLs — Checks for valid absolute URLs
  • Common mistakes — Detects typos, duplicate rules, conflicting directives

URL Path Testing

  • Pattern matching — Follows standard path-matching rules
  • Wildcard support — Handles * (any sequence) and $ (end anchor)
  • Specificity — Longer matching rules take precedence
  • Agent resolution — Tests against specific or wildcard user-agent
Important: robots.txt is not a security measure. Malicious crawlers can ignore it. Use proper authentication and permissions to protect sensitive content.

Related Utils

Meta Tags XML Sitemap Broken Link Checker Heading Analyzer Canonical URL Checker Keyword Density

Feedback

Help us improve this page by providing feedback, and include your name/email if you want us to reach back. Thank you in advance.


Share with