Free SEO tool · Technical SEO
Robots.txt Tester and Validator
Read a site’s robots.txt the way search engines do: whether Googlebot and AI crawlers may crawl it, which sitemaps it declares, and whether one path you choose is allowed.
- PASS GooglebotMay crawl the home page
- WARNING /blog/Blocked for every crawler
- NOTE AI crawlersGPTBot and CCBot blocked
- NOTE Crawl-delayIgnored by Google
- PASS SitemapOne sitemap declared
What does the free Robots.txt Tester analyze?
-
1
Availability
Whether /robots.txt answers 200 or 404. Server errors make Google pause crawling the whole site.
-
2
Googlebot
Whether Google’s crawler may fetch your home page — the most expensive mistake a robots.txt can make.
-
3
Any path, any bot
Enter a path and a user-agent to see which rule applies, using the same longest-match logic as Google.
-
4
AI crawlers
Which known AI training and AI search bots are blocked, so you can decide on purpose.
-
5
Sitemaps
Sitemap lines, and whether they are absolute URLs search engines can use.
-
6
Syntax and limits
Lines Google can’t understand, rules it ignores such as crawl-delay and noindex, and the 500 KiB size limit.
How to set up robots.txt
-
1
Serve it at the root
Plain text at /robots.txt on every host, answering 200 — or 404 if you have no rules.
-
2
Block crawling, not indexing
To hide a page from results, allow crawling and add noindex instead. Meta Tag Checker →
-
3
Don’t block CSS and JavaScript
Google renders pages; blocked files can make them look broken to it.
-
4
Keep rules specific
The longest matching path wins, so a short Disallow can block more than you meant.
-
5
Declare your sitemap
Add a Sitemap line with the full https:// address. Sitemap Finder & Validator →
-
6
Decide on AI bots separately
Training crawlers and AI search crawlers have different names — block only what you mean to.
How to interpret your results
- ISSUE
- A clear fault that usually deserves a fix, e.g. a missing title or a non-200 status.
- WARNING
- May be a problem depending on the page, e.g. a missing description or canonical, or a long title.
- PASS
- Checked and looks fine.
- NOT CHECKED
- Not part of this check, or blocked by the site. The report says which — it is never counted as a fault.
Robots.txt Tester vs Website Audit vs SEO Monitor
Robots.txt Tester
One URL · free · no account
Which crawlers — search engines and AI bots — may crawl your site, and test any path.
You’re hereWebsite Audit
Many pages · free quick crawl
Crawls a limited set of pages and groups issues across your site, like duplicate titles and potential orphan pages.
Run a website audit →SEO Monitor
Connected data · account
Uses your Search Console and GA4 to track clicks, queries and changes over time.
Explore SEO Monitor →FAQ
Frequently asked questions
About the Robots.txt Tester.
01
Does blocking a page remove it from Google?
No. robots.txt stops crawling, not indexing. To keep a page out of results, allow crawling and add noindex.
02
Should I block AI crawlers?
It is your choice. Blocking training bots keeps content out of future models; blocking AI search bots also keeps you out of their answers.
03
Where must robots.txt live?
At the root of each host: https://example.com/robots.txt. Every subdomain needs its own file; one in a folder is ignored.
04
What if robots.txt returns a server error?
Google treats a 5xx answer as “don’t crawl anything” and pauses crawling the site. A 404 is fine: it means no rules.
05
Does Google support crawl-delay?
No. Google ignores crawl-delay. Some other search engines still read it.
06
Which rule wins when two match?
The most specific one — the rule with the longest matching path. If an Allow and a Disallow are equally long, Allow wins.
07
Is it free?
Yes. No account is needed. We limit how often one visitor can run checks so the tools stay fast for everyone.
