Home› SEO›Robots.txt Generator
SEO tool

Robots.txt Generator

Create a clear robots.txt file with allow, disallow and sitemap directives.

Create a robots.txt starting point

Build user-agent, allow, disallow and sitemap directives with explicit line-by-line output.

Remember: robots.txt manages crawler access; it is not an authentication or privacy mechanism.





What robots.txt controls
A robots.txt file gives compatible web crawlers instructions about paths they may or may not crawl. It normally lives at the root of a host, such as https://example.com/robots.txt.
How to use this generator
  1. Choose the crawler user-agent, or use * for a general rule group.
  2. Add allow and disallow paths one per line.
  3. Add your absolute sitemap URL if applicable.
  4. Review every path before deploying the file at the site root.
Robots.txt is not access control
A disallow rule does not make a URL private. Sensitive information must be protected with real authentication and authorization. Also remember that blocking crawling and preventing indexing are different concepts.
Test carefully before broad blocks
A rule such as Disallow: / can prevent a compliant crawler from accessing the whole site for that user-agent. On production sites, review generated rules with someone who understands the site's crawl and indexing requirements.


Frequently asked questions
Where should robots.txt be placed?
It is typically served from the root of the hostname at /robots.txt.
Does Disallow make a page private?
No. Robots.txt is a crawler directive, not an access-control system.
Can I reference a sitemap in robots.txt?
Yes. You can include an absolute Sitemap directive pointing to the sitemap URL.
What does User-agent: * mean?
It defines a rules group intended for crawlers that match the wildcard user-agent, subject to each crawler implementation.