🤖 Robots.txt Generator
Create a robots.txt file to control how search engines crawl your website.
Crawl Rules
Robots.txt Generator
Generate a robots.txt file that controls search engine access to your website. Specify which user-agents (crawlers) should be allowed or disallowed from specific paths, and include your sitemap URL. The generated file should be placed in your website's root directory.
Frequently Asked Questions
What is robots.txt?
Robots.txt is a text file placed in the root directory of a website that tells search engine crawlers which pages or sections they can or cannot crawl. It uses the Robots Exclusion Protocol to communicate with web robots.
Where should I put the robots.txt file?
The robots.txt file must be placed in the root directory of your website, accessible at yourdomain.com/robots.txt. It cannot be placed in a subdirectory — search engines only look for it at the root level.
What are common robots.txt rules?
Common rules include: User-agent: * (applies to all crawlers), Allow: / (allow everything), Disallow: /admin/ (block admin pages), Disallow: /private/ (block private content), Sitemap: URL (point to sitemap). You can also target specific bots like Googlebot.