Robots.txt Generator
Create a valid robots.txt file by configuring bots, paths, sitemaps and crawl delays.
Advertisement
Bot Configuration
Select User-Agents
Paths
Disallow Paths
Allow Paths
Sitemap
Preview
Advertisement
About robots.txt
The robots.txt file is placed at the root of a website to tell web crawlers which pages or files to access. It follows the Robots Exclusion Protocol (REP).
Disallow: / blocks everything. Allow: /public/ overrides a disallow for specific paths. The Sitemap: directive helps crawlers find your sitemap quickly.