Robots.txt Generator
SEO ToolsBuild a robots.txt file with sitemap and crawl rules.
robots.txt
AI suggestions for the robots.txt rules. Review before applying.
AI result
Suggestions
AI-generated suggestions may be inaccurate. Review before publishing. AI processing is optional and content is sent to our AI provider only when you run an action.
Optional AI assistance. Your input is only sent to our AI provider when you run an action.
What is Robots.txt Generator?
Robots.txt Generator helps you build a crawler rules file with allow and disallow directives and a sitemap reference.
How to use Robots.txt Generator
- 1 Add the user agents and paths to allow or block.
- 2 Include your sitemap URL.
- 3 Download or copy the finished robots.txt.
Supported formats and limits
- Output: robots.txt content
- Runs in your browser
Common uses
- Block a staging site from being indexed.
- Allow search engines while referencing your sitemap.
- Set crawl rules for specific user agents.
Things to know
- robots.txt is a request, not a security mechanism.
- A syntax error can unintentionally block crawlers.
Privacy and file handling
The file is generated in your browser.
Frequently asked questions
Can robots.txt hide private pages?
No. robots.txt only asks crawlers to stay away. Use authentication to protect private content.
Where does robots.txt go?
At the root of your domain, for example https://example.com/robots.txt.
Does robots.txt keep a page out of Google?
No. It only discourages crawling. To keep a page out of results, use a noindex directive.
What is the sitemap directive for?
It tells crawlers where to find your XML sitemap.
Related tools
Meta Tag Generator
Generate SEO meta tags and social sharing markup.
Schema Markup Generator
Generate JSON-LD structured data for common types.
SERP Preview
Preview how a page looks in Google search results.
Keyword Density Checker
Analyse keyword and phrase frequency in your content.