SiteStyleGuide

Our crawler

SiteStyleGuideBot

If you’ve seen SiteStyleGuideBot in your logs, someone added your website to SiteStyleGuide, which makes one-page style guides of public websites. Here’s what it does and how to stop it.

How it identifies itself

SiteStyleGuideBot/1.0 (+https://sitestyleguide.com/bot)

The same name is sent with every request it makes, including the screenshots it takes with a headless browser.

What it fetches

  • Your robots.txt, first.
  • Your homepage, once.
  • The stylesheets, web manifest, favicons, logo and sharing image the homepage links to.
  • Two screenshots of the homepage, at desktop and phone widths, which load the page as a browser would.

It never follows links to other pages, submits forms or signs in. A site is read again only when its guide is refreshed, which members can do once every 90 days. Captures across all sites are limited to a few every 15 minutes, so it never visits in bursts.

How to stop it

SiteStyleGuideBot follows robots.txt. To keep it off your site, add:

User-agent: SiteStyleGuideBot
Disallow: /

It also obeys a User-agent: * group that disallows your homepage. A site that blocks it this way can’t get a new guide.

To remove a guide that already exists, use the takedown form. Removal is free, and the site can’t be added again.

Contact

Questions or problems: hello@sitestyleguide.com.