# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file # User-agent: Twitterbot Disallow: User-agent: facebookexternalhit Disallow: User-agent: * Allow: /blog Allow: /$ Disallow: / Disallow: /d/random