SEOToolboxBot: our crawler
Last updated
What is SEOToolboxBot?
SEOToolboxBot is the crawler behind SEO Toolbox Online. It visits a site only when someone runs an audit or checker on it. It does not crawl the web on its own, and it does not build a search index or collect training data.
User agent: SEOToolboxBot/1.0 (+https://seotoolbox.online/bot)
Does it follow robots.txt?
Yes. It obeys the SEOToolboxBot group in robots.txt, or the * group if there is none. The one exception is the exact URL someone enters for an audit, which is fetched once even if robots.txt disallows it, because a person asked for that page; links found on it are only followed where robots.txt allows.
Every request we send, including the real-browser pass used for screenshots and performance, uses this user agent. When PageSpeed Insights is enabled for an audit, Google fetches the page with its own Lighthouse user agent; those requests come from Google, not from us.
How do I block it?
Add this to your robots.txt:
User-agent: SEOToolboxBot
Disallow: /How much does it fetch?
At most 10 pages per audit, plus robots.txt, sitemaps and the specific files a check needs, such as /llms.txt or a 404 test URL.
It is causing a problem
Tell us through the contact page with the time and your domain.