The Bias Index

BiasIndexBot

If you found BiasIndexBot in your logs: it reads the comparison pages SaaS companies publish about their rivals ("X vs Y", "X alternatives"), so The Bias Index can score how one-sided they are.

What it does

  • Runs once a week, on Sundays from 05:00 UTC, when most sites are quietest, and identifies itself as BiasIndexBot/0.1 (+https://thebiasindex.com/bot).
  • Reads your robots.txt and the sitemaps it lists, then fetches only pages whose address looks like a comparison (-vs-, alternative, compare…).
  • Makes one request at a time to your site, at least a second apart, or slower if your robots.txt sets a Crawl-delay.
  • Gives each page at most 30 seconds, and stops for the week after three blocked (403, 429, 503) or timed-out pages in a row, rather than retrying harder.
  • Doesn't run JavaScript, submit forms or log in, and never republishes your pages: it links to them.

Opting out

BiasIndexBot honors robots.txt. To keep it out entirely: 

User-agent: BiasIndexBot
Disallow: /

Your company then shows as "crawler blocked" on the index, with no score: we don't guess what we couldn't read.

Contact

Questions, or a page you think we scored wrong: david.baussart at whimsical.com. The crawler's code is open on GitHub.