Loading…
Free SEO tool
Robots.txt and sitemap checker
Enter your domain and we check the two files search engines look for first: your robots.txt and your XML sitemap. It catches the mistake that quietly hides a whole site.
Runs entirely in your browser. Nothing you paste is sent to us or stored.
The two files every crawler looks for
When a search engine visits your site, one of the first things it does is fetch robots.txt from the root. That small text file tells crawlers where they may and may not go. Get it wrong and you can accidentally tell Google not to look at anything, which is exactly how sites sometimes vanish from search after a redesign: a developer left a blanket disallow in place from the staging site and nobody noticed.
The second file is the XML sitemap, usually at /sitemap.xml. It is a list of the pages you want found, which helps search engines discover everything, especially on larger sites or pages that are not well linked. This tool fetches both, tells you whether they exist and respond properly, shows you the actual robots rules, counts the URLs in your sitemap, and warns you loudly if robots.txt is blocking the whole site.
What good looks like
A healthy setup is simple. robots.txt exists, returns a 200, does not disallow the whole site, and points to your sitemap with a Sitemap: line. The sitemap exists, returns a 200, and lists your real pages. That is it. You do not need anything clever, and clever is usually where the mistakes creep in.
The most common problems this tool surfaces are a missing sitemap, a robots.txt that forgets to reference the sitemap, and the serious one, a stray Disallow: / that blocks everything. If you see that last warning, treat it as urgent: it means search engines are being told to ignore your site, and no amount of other SEO work will help until it is removed.
Blocking a page is not the same as hiding it
One subtlety worth knowing: robots.txt controls crawling, not indexing. A page you disallow in robots.txt can still appear in results, without a description, if other sites link to it, because Google knows the URL exists but was told not to read it. If you genuinely want a page kept out of search, use a noindex tag on the page itself rather than a robots.txt block. Our SEO score checker reports whether your important pages carry an accidental noindex.
Common questions
Where should my robots.txt and sitemap live?
robots.txt goes at the root, for example yoursite.co.uk/robots.txt. The XML sitemap usually lives at yoursite.co.uk/sitemap.xml, and robots.txt should point to it with a Sitemap: line.
What does Disallow: / do?
It tells every crawler to ignore your entire site. It is fine on a staging site and a disaster on a live one. If this tool flags it, remove it unless you truly mean to be invisible in search.
Do I need a sitemap if my site is small?
It is not mandatory, but it is cheap and helpful. Most modern platforms generate one automatically. It matters most for larger sites and for pages that are not well linked internally.
My page is in robots.txt but still shows in Google. Why?
robots.txt stops crawling, not indexing. To keep a page out of results, use a noindex meta tag on the page itself rather than blocking it in robots.txt.
More free tools and services
Further reading
- Google Search Central: intro to robots.txt ↗The official guide to what robots.txt can and cannot do.
- Sitemaps.org protocol ↗The XML sitemap format search engines expect.
Want us to just do your SEO?
These tools show you what to fix. If you would rather hand it over, we do technical SEO, schema, content and local search as a fixed-price service.