Robots.txt Generator & Tester
Control how search engines and AI bots crawl your website.
Bot Permissions
Example: /admin/, /login/, /private-files/
Robots.txt Master Guide
User-agent
The 'Who'. Identifies which crawler the rule applies to. '*' matches all bots.
Disallow
The 'Where Not'. Tells the bot which folders or files to ignore.
Allow
The 'Exception'. Overrides a Disallow rule for a specific sub-folder.
Sitemap
The 'Map'. Points bots to your XML sitemap location.
Advanced Pattern Matching
Wildcard (*)
Matches any sequence of characters.
Disallow: /*?* # Blocks URLs with paramsEnd of String ($)
Indicates the end of a URL path.
Disallow: /*.pdf$ # Blocks PDF filesGoogle Best Practices
- Don't block CSS/JS: Google needs them to render your page layout.
- Case Sensitive: Robots.txt is case-sensitive. /Admin/ is different from /admin/.
Critical Warnings
Noindex via Robots.txt
Do NOT use robots.txt to de-index pages. Use the meta robots tag instead.
Trailing Slash
Disallow: /admin blocks the folder. Disallow: /admin blocks the file. Be careful.
More Free SEO Tools
SERP Simulator
Preview how your website will look in Google search results. Optimize Title & Meta Description for higher CTR.
Meta Tag Generator
Generate standard HTML Meta Tags for your website including Title, Description, Robots, and Viewport.
Hreflang Generator
Target the right audience by language and region. Generate correct hreflang tags for International SEO.
Ultimate Schema Gen
All-in-one structured data builder. Create Article, FAQ, Local Business, Product, and Organization schemas to win Rich Snippets.
Social Preview (OG)
See exactly how your content will appear when shared on Facebook, LinkedIn, and X (Twitter).
AI Visibility Score
Will ChatGPT or Google SGE cite you? Test your content's machine readability and 'Answer Engine' readiness.
llms.txt Generator
The future of SEO. Create a context file for AI models (ChatGPT, Claude) to better understand your website.