Free Bulk robots.txt & Sitemap Checker

Find out whether search engines and AI crawlers are allowed to read your pages, which robots.txt line blocks them, and whether your sitemaps work and include each page. Check up to 20 pages at once.

Quick human check, then the tool unlocks:

Page addresses (one per line, up to 20)

What is checked

robots.txt rules for 13 crawlers: Google, Google Images, Bing, Apple, DuckDuckGo, Yandex, Baidu, and AI crawlers GPTBot (ChatGPT), ClaudeBot, Google-Extended (Gemini), PerplexityBot, Common Crawl and Bytespider. Rules are matched the way Google does it: the most specific crawler group, and the longest matching rule.
noindex in the page's meta robots tag or X-Robots-Tag header, which keeps a page out of search results even when crawling is allowed.
Sitemaps listed in robots.txt (or at /sitemap.xml): whether they load, how many addresses they hold, and whether this page is in them. Sitemap indexes and .gz sitemaps are supported.

Blocking AI crawlers

To stop AI companies training on your site, add groups like this to robots.txt. Search engines are not affected:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

🔒 The pages you check are not saved or logged.