AgentReadyChecker reads a handful of public pages (robots.txt, the homepage, the public product feed and up to three product pages) only when a person asks us to check a store. It respects robots.txt: disallow AgentReadyChecker to opt out.
AgentReadyChecker