Robots.txt 生成器
配置搜索引擎爬虫规则，一键生成robots.txt文件。
Advertisement
Bot Configuration
Select User-Agents
Paths
Disallow Paths
Allow Paths
Sitemap
预览
Advertisement
About robots.txt
The robots.txt file is placed at the root of a website to tell web crawlers which pages or files to access. It follows the Robots Exclusion Protocol (REP).
Disallow: / blocks everything. Allow: /public/ overrides a disallow for specific paths. The Sitemap: directive helps crawlers find your sitemap quickly.