# www.robotstxt.org/ # http://code.google.com/web/controlcrawlindex/ # Uncomment the lines below to disallow any crawling from all agents: # User-agent: * # Disallow: /