检测robots.txt规则
User-agent: * Disallow: /admin/ Disallow: /search? Allow: / # 禁止所有爬虫(谨慎!) # User-agent: * # Disallow: / Sitemap: https://example.com/sitemap.xml