Robots.txt Generator

Build a valid robots.txt for your site. Add allow/disallow rules per crawler and reference your sitemap.

Default policy for all crawlers Sitemap URL (optional) Crawl delay (seconds, optional) Custom Rules

Generated robots.txt

Place at the root of your site: https://yourdomain.com/robots.txt - one per (sub)domain.

* targets all crawlers. Use Googlebot, Bingbot, GPTBot etc. for specific bots.

Disallow blocks paths from crawling. Allow re-permits a sub-path inside a disallowed parent.

Helps search engines discover URLs faster. Use the full absolute URL of your sitemap.xml.