About SEOAuditBot

SEOAuditBot is the crawler behind ClckIndex's free website check. It visits a website only when someone asks this tool to check it.

How it behaves

• It identifies itself as SEOAuditBot with a link to this page.
• The page someone enters is fetched once, like a normal browser visit. Every other page is fetched only if the site's robots.txt allows it, using the rules for SEOAuditBot, or for * if there are none for it.
• It requests at most 3 pages at a time, and one at a time with a pause if robots.txt sets a crawl-delay (pauses longer than 5 seconds are shortened).
• A check covers at most 250 pages, plus a quick request to up to 100 linked pages on other websites, to see whether those links still work.
• It doesn't follow links marked nofollow, or any links on a page whose robots meta tag or X-Robots-Tag header says nofollow.
• It doesn't run JavaScript, fill in forms, sign in, or send cookies.

How to block it

Add this to your site's robots.txt file:

User-agent: SEOAuditBot
Disallow: /

After that, a check of your site only looks at the single page someone enters.